Maritime search and rescue system based on multi-sensor fusion and unmanned ship using same

Abstract

The invention discloses a search and rescue system based on multi-sensor fusion, and the system is installed on an unmanned ship, and comprises a millimeter wave radar detection module, a shipborne visual detection module, a multi-sensor fusion positioning module, an autonomous navigation module, and a data communication module. Through combined application of a millimeter wave radar and an improved visual detection algorithm, remote rapid screening and close accurate identification and positioning of people falling into water are realized, an unmanned ship is automatically controlled to approach a target to implement rescue, the method is especially suitable for water area search and rescue under complex sea conditions, the false detection rate can be effectively reduced, and the response speed can be improved.

Description

Technical Field

[0001] This invention relates to the field of marine rescue equipment technology, and in particular to a search and rescue system based on multi-sensor fusion and an unmanned vessel using the system. Background Technology

[0002] Current intelligent maritime search and rescue systems mostly employ unmanned aerial vehicles (UAVs), unmanned surface vessels (USVs), or combinations thereof. These systems use multi-source sensors to detect distressed targets in real time and automatically guide rescue equipment or vessels to the location. Common configurations in related patents and literature include: UAVs carrying infrared / camera devices patrolling the sea area and transmitting the target's location to surface rescue equipment when distress occurs; unmanned surface vessels carrying UAVs for remote deployment and range extension; or UAVs combined with lifebuoys / life rafts to achieve collaborative rescue using "unmanned + floating equipment." For example, patent CN110001888A discloses a system consisting of a UAV and a U-shaped rescue device (similar to a floating rescue raft): the UAV carries a camera and GPS to collect images of the sea surface; the rescue device integrates GPS, inertial navigation, a controller, and a thruster. When the rescue device receives images and location information from the UAV, the control module processes the signals, locates the distressed personnel, plans a route, and propels the rescue device forward using inertial navigation and propulsion.

[0003] Currently, the application of maritime search and rescue systems faces several limitations. Existing search and rescue system solutions can be typically categorized into radar-based and visual recognition-based solutions, while system mission platforms can be divided into unmanned surface vessel (USV) + unmanned aerial vehicle (UAV) combinations and single USV systems.

[0004] In terms of visual recognition, computer vision-based search and rescue solutions currently rely on shipborne cameras and deep learning detection models (such as the YOLO series) for target detection and classification. While visible light cameras offer high accuracy in shape discrimination and category differentiation, they are susceptible to strong reflections, wave obstruction, camera shake, and changes in lighting conditions at sea. Detection accuracy drops significantly, especially for small targets at medium to long distances, and is extremely difficult to identify targets in fog or at night when visibility is low. Furthermore, the rolling of the ship and vibrations during navigation can cause image blurring, leading to a substantial increase in both false positives and false negatives in complex sea conditions.

[0005] Regarding radar solutions, lidar and infrared detectors are also limited in range by rain, snow, and sea fog. Literature indicates that identification of small targets at sea typically relies on infrared or multi-source fusion, but the reliability of these identifications is far from ideal. Millimeter-wave radar possesses all-weather adaptability and resistance to fog and low-light environments in medium- to long-range detection; however, its radar echoes are susceptible to interference from wave reflections, the shape of floating objects, and humidity changes. Its radar cross-section (RCS) characteristics are similar to the human body in certain situations, especially in rough seas with dense clutter, easily leading to false alarms and misjudgments. Furthermore, the target information output by millimeter-wave radar typically only includes range, azimuth, and reflection intensity, lacking the ability to directly discriminate the target's shape and type, resulting in insufficient identification accuracy.

[0006] In terms of mission platforms, the combination of unmanned surface vessels (USVs) and unmanned aerial vehicles (UAVs) offers the advantage of "air-sea integration," improving search speed and coverage. However, it also has significant shortcomings in areas such as takeoff and landing conditions, wind and rain resistance, endurance, communication links, system complexity, and mission continuity. These problems mainly stem from the physical limitations of UAVs in harsh maritime environments and the still-developing collaborative system between UAVs and USVs. Specifically, UAV takeoffs and landings on sea decks are severely affected by strong winds, large waves, and ship swaying, often preventing safe takeoff at wind speeds exceeding a certain threshold. Even if takeoff is successful, the attitude control of UAVs in the air is easily interfered with due to the limited wind and rain resistance of rotorcraft, leading to drift or stall. Insufficient motor and battery protection can even cause failure due to rain or waves. Simultaneously, communication between UAVs and USVs is susceptible to interference from multipath effects and signal attenuation in the complex electromagnetic environment of the sea, resulting in data delays or interruptions and reducing the real-time performance and reliability of search and rescue missions. In terms of system architecture, the combination of unmanned surface vessels (USVs) and unmanned aerial vehicles (UAVs) requires additional deck landing and recovery mechanisms, and introduces complex joint control and mission management algorithms. The mechanical structure is susceptible to damage or jamming under the impact of wind and waves, and the control logic faces a higher risk of failure due to managing two heterogeneous platforms simultaneously. If a UAV malfunctions, the overall search and rescue capability will be significantly weakened. Furthermore, UAVs require frequent returns for charging or battery replacement, while USVs lack rapid automated resupply methods, leading to interruptions in rescue operations and making it difficult to guarantee continuity. In extreme weather conditions such as typhoons, heavy rains, and dense fog, UAVs are essentially unable to perform missions, forcing the system to rely solely on the USV platform, thus losing the synergistic advantages of "air-sea integration" and significantly reducing rescue efficiency and reliability. In summary, while the USV + UAV combination increases the search range, its adaptability, stability, and continuous operation capability in complex sea conditions all have significant shortcomings.

[0007] Existing maritime search and rescue scenarios often occur in extreme sea conditions such as high winds and waves, frequent heavy rain, and poor visibility. These environments not only pose a serious threat to manned maritime rescue operations, making it difficult for traditional manned vessels to quickly arrive and conduct operations, but also significantly limit the use of unmanned aerial vehicles (UAVs). Under conditions of strong winds and heavy rain, UAVs are almost impossible to take off. Even if they manage to take off, they quickly lose their operational capability due to insufficient wind resistance, rain intrusion, or a significant reduction in battery life, making it difficult to complete continuous patrol search and target identification tasks. Therefore, in the most critical sea conditions requiring search and rescue support, UAV platforms become a bottleneck in the system, failing to fulfill the intended role of the integrated air-sea system.

[0008] While existing multi-sensor fusion methods attempt to combine radar and vision, most only perform simple superposition or parallel judgment at the result level, lacking a hierarchical identification mechanism that utilizes the complementary characteristics of sensors for pre-screening and post-confirmation, making it difficult to effectively reduce the false detection rate. At the same time, data fusion on dynamic unmanned surface vessels (USVs) suffers from insufficient registration accuracy and time synchronization difficulties, leading to amplified positioning errors in windy and wavey environments, thus affecting the accuracy and safety of USVs approaching targets.

[0009] The main reasons for the above problems include the following: First, a single sensor cannot simultaneously meet the needs of long-distance detection and high-precision classification and recognition, as radar and vision each have their own physical and environmental constraints; second, existing visual detection algorithms are mostly trained on land or relatively stable scenes, lacking specific optimization for marine environments; third, the multi-sensor fusion process lacks effective false alarm filtering and accurate registration algorithms, making it difficult to fully leverage the complementary advantages between sensors; and fourth, when operating on unmanned surface vessels, the changes in hull attitude, wave interference, and real-time requirements are not fully considered, affecting the overall performance of the system.

[0010] Therefore, developing a search and rescue device that is wind and rain resistant at sea and can operate for extended periods is one of the important problems that the current maritime search and rescue system needs to solve. Summary of the Invention

[0011] To address the aforementioned problems in existing technologies, the purpose of this invention is to provide a search and rescue system based on multi-sensor fusion. This system combines millimeter-wave radar for long-range target search with a visual detection algorithm optimized for shipboard environments. By using multi-sensor fusion to improve positioning accuracy under dynamic conditions, it effectively solves the shortcomings of high false alarm rates in long-range detection, insufficient accuracy in close-range identification, and poor mission continuity of traditional UAV platforms in complex sea conditions. This enables all-weather, highly reliable detection and rescue of people who have fallen into the water.

[0012] Another objective of this invention is to provide an unmanned vessel search and rescue device based on multi-sensor fusion.

[0013] To solve the above problems, the present invention adopts the following technical solution: a maritime search and rescue system based on multi-sensor fusion, the system being applied to an unmanned vessel, including a millimeter-wave radar detection module, which is electrically connected to the shipborne control and computing unit, for scanning sea surface targets at long distances, obtaining spatial position and reflection intensity parameters, and initially screening targets requiring rescue based on radar cross-section characteristic values;

[0014] The shipborne visual inspection module includes a high-definition camera, infrared imaging equipment, and a shipborne GPU running an improved YOLOv11-USVEDT, which enables real-time detection of targets requiring rescue through optimization of the marine environment.

[0015] The multi-sensor fusion positioning module operates in the shipboard control unit. By fusing ranging data detected by millimeter-wave radar with pixel coordinates, target size and attitude data output by visual recognition, and combining the high-definition camera's intrinsic and extrinsic parameter calibration with the radar coordinate system mapping relationship, it calculates the geographical location information of the target to be rescued. The geographical location information provides data input for subsequent autonomous navigation.

[0016] The autonomous navigation module generates a trajectory for the unmanned vessel to approach the target based on the geographic location information output by the multi-sensor fusion positioning module, thus getting closer to the target in need of rescue.

[0017] The data communication module transmits search and rescue information to the shore-based command center in real time via 4G / 5G or BeiDou short message links.

[0018] Furthermore, the shipborne visual inspection module optimizes the marine environment, specifically as follows:

[0019] In the visible light channel, EfficientNetV2 is used to reconstruct the feature extraction backbone to enhance the multi-scale feature representation capability. DySample dynamic sampling is used to improve the detection accuracy of small targets. Histogram transformer module is used to enhance feature extraction in strong reflective and degraded areas. The bounding box regression performance is optimized by combining dynamic gradient gain and WIoU loss function with outlier evaluation.

[0020] In the infrared channel, thermal radiation characteristics are used to assist in the identification of targets in low visibility and at night;

[0021] By complementing and fusing visible light and infrared information, real-time human target detection can be achieved in maritime scenarios.

[0022] Furthermore, the autonomous navigation module, while the unmanned vessel is approaching the target, performs path correction and course adjustment by combining real-time obstacle avoidance sensor data.

[0023] This invention discloses a method for using a maritime search and rescue system based on multi-sensor fusion, comprising the following steps:

[0024] Step 1: Initialization. Power on the system and perform time synchronization, sensor self-test, attitude alignment, external component loading, and health status reporting. After completion, enter cruise monitoring mode. Define the world coordinate system {W}, hull coordinate system {B}, radar coordinate system {R}, and camera coordinate system {C} for the system. Obtain the rigid body external component intrinsic parameters through calibration.

[0025] The camera intrinsic parameter K and distortion parameters were obtained via a calibration board;

[0026] extrinsic parameter matrix and Obtained through joint calibration (camera observation of markers, radar ranging, and IMU attitude);

[0027] Hull attitude Obtained by tightly coupled IMU / GNSS estimation;

[0028] Step 2: Long-range search and RCS pre-screening. The millimeter-wave radar performs beam scanning at a refresh rate of 10–20 Hz and outputs target measurements. Where r is the range, θ is the azimuth, v is the radial velocity, A is the echo amplitude, and σ is the RCS estimate; the CFAR method is used for detection in the range-Doppler plane, and the human body is screened based on the RCS statistical interval and velocity prior in this frequency band; a candidate set is output. And for each candidate generated image ROI, the pointing angle and approximate depth prior are calculated;

[0029] Use a CAN analyzer to connect the radar to an embedded device or control center to obtain the attribute information of the target detected by the radar, capture and export the raw CAN data, and parse the target number, distance, speed, horizontal and vertical position, and target status information contained in the 0x60B message;

[0030] During the parsing process, the corresponding fields are extracted based on the signal start bit and length, and converted into actual physical values ​​through resolution, so that the original message is converted into structured target data;

[0031] Step 3: Close-range identification. The unmanned surface vessel's gimbal points to the candidate location, and a high-definition camera and infrared imaging device simultaneously acquire forward-view image frames. The improved YOLOv11-USVEDT model performs multimodal detection on the visible light and infrared images, outputting the bounding box of the target requiring rescue within the ROI region and in the full-frame image. Category c, confidence s, and combined through feature layer fusion and decision-level fusion;

[0032] Step 4: Radar-vision fusion positioning. The fusion module performs positioning under a unified time scale. and Perform spatial registration and state estimation;

[0033] Step 5: Autonomous navigation and dynamic obstacle avoidance. After obtaining the target's relative / absolute position, the path planner generates an approach trajectory: the far segment uses an A* path, and the mid-to-near segment uses LOS (line of sight) guidance control, with the control law as follows:

[0034]

[0035] in For the current heading, For the desired heading, These are the lateral deviation and the forward error, respectively. To increase gain; obstacle avoidance uses an artificial potential field to correct local velocity commands and limit the minimum safe distance; propulsion / rudder angle is achieved through closed-loop tracking using PID; when wind and waves are ≥3, attitude feedforward is fused for disturbance rejection compensation;

[0036] Step Six: Communication and Human-Machine Collaboration. The system reports the following to the shore-based system via the data communication module at a frequency of 1–5 Hz: target ID, latitude and longitude, fusion confidence level, latest image summary, navigation status, and remaining energy; it also receives and executes manual "confirm / cancel / change course / return" commands in real time.

[0037] Step 7: When the joint vision-radar confidence level exceeds the threshold And the uncertainty of the target position When the target falls below the threshold, it enters the "approaching state"; upon reaching the set radius, it triggers subsequent rescue actions and continues to monitor the target until it receives a recovery command.

[0038] Furthermore, the shipborne visual detection module described in step three undergoes the following enhancements in the shipborne environment: EfficientNetV2 feature extraction backbone, DySample dynamic upsampling method, histogram transformer module, WIoU loss function combining dynamic gradient gain and outlier evaluation, and anti-reflective / splash data augmentation. The EfficientNetV2 feature extraction backbone consists of a Fused-MBConv convolutional structure, an MBConv inverse residual structure, and progressive convolutional blocks. The Fused-MBConv layer uses a combination of standard convolution and inverse residual structure in the shallow layers to extract low-level features of water surface texture and reflective areas.

[0039] The MBConv layer employs depthwise separable convolutions and Squeeze-and-Excitation attention in deeper layers to extract the human contours of drowning victims and local patterns in small regions. The progressive learning structure gradually increases the input image resolution, allowing the model to first learn stable features of the water background and then gradually adapt to small target features at higher resolutions. The DySample dynamic upsampling module consists of three parts: a position offset prediction branch, a dynamic interpolation weight calculation branch, and a learnable sampling point coordinate generator. Its implementation includes: predicting the sampling position offset using a 3×3 convolution + activation function on the input low-resolution feature map; generating dynamic interpolation weights using a 1×1 convolutional layer to calculate feature values ​​during the upsampling process; determining the new sampling point coordinates based on the offset and implementing dynamic upsampling through bilinear interpolation; and outputting a feature map of the same size as traditional nearest neighbor / bilinear upsampling, but retaining more boundary and texture information. This module enhances the target boundary reconstruction capability of shipborne cameras under wave reflection and water ripple disturbances. The histogram transformer module consists of a histogram construction unit, a binning attention submodule, and a dual-scale convolutional fusion unit. The histogram construction unit performs binning statistics on the intensity values ​​of the input feature map, forming an intensity distribution reflecting local illumination changes. The binning attention submodule constructs an attention map based on the binning weights, enabling the model to focus on regions easily confused by reflections / spray. The dual-scale convolutional fusion unit uses 1×1 and 3×3 convolutions to extract small-scale details and large-scale semantics respectively, and then fuses them to achieve multi-scale feature enhancement. This module can effectively alleviate the feature blurring problem caused by sunlight refraction and water mist in shipborne images.

[0040] The WIoU loss module consists of an anchor box quality assessment unit, a gradient gain control unit, and a boundary regression loss calculation unit. First, the anchor box quality assessment unit calculates the "outlier degree" based on the center distance between the predicted and ground truth boxes, area differences, and shape differences. Then, the gradient gain control unit dynamically allocates gradients based on the outlier degree, assigning higher gradient weights to high-quality anchor boxes and automatically reducing the weights of low-quality anchor boxes to avoid training oscillations. Based on this, an improved IoU loss is calculated to achieve robust regression optimization for small targets.

[0041] Subsequently, visual measurements were established for each candidate image based on the complementary properties of the two source images. .

[0042] Furthermore, the millimeter-wave radar detection module is installed at the center of the forward part of the unmanned vessel's deck, with a height of ≥0.8 m above the water surface. The mounting base is equipped with a pitch fine-tuning mechanism. The radar operates in the 76–81 GHz frequency band, has a horizontal field of view of ≥90°, and a range of 300 m. The radar is connected to the shipborne computing unit via Ethernet or CAN-FD, and its power supply is converted from a 24 V bus to the radar's rated voltage via DC-DC converter.

[0043] 7. The application method of the maritime search and rescue system based on multi-sensor fusion according to claim 4, characterized in that the multi-sensor fusion positioning module in step 4 performs positioning under a unified time scale. and Perform spatial registration and state estimation:

[0044] (1) Geometric registration: from the center ray of the camera pixel Transform to {B}: By combining an approximate planar model of the sea surface to find the intersection point of the ray and the water surface, the prior relative position calculated solely by visual estimation is obtained. .

[0045] (2) Measurement model: radar polar coordinate measurement Visual orientation measurement Establish a joint observation model ,in, Including the target's position in {B} / {W};

[0046] (3) Filtering and association: Extended Kalman filter or unscented Kalman filter is used for state update; nearest neighbor or Hungarian algorithm is used for measurement-trajectory association in multi-target scenarios; visual high-confidence measurement is used to correct radar range scale, and radar stable range is used to suppress visual depth drift; output the latitude and longitude of the target in {W} and its relative position in {B}. And give the covariance.

[0047] The present invention also provides an unmanned vessel comprising the maritime search and rescue system based on multi-sensor fusion as described in claim 1.

[0048] The unmanned vessel adopts a catamaran hull and aluminum alloy frame, and has an integrated vibration damping gimbal at the bow. A high-definition camera and an infrared thermal imaging camera are coaxially mounted on the gimbal, and the angle between the optical axis of the camera and the water surface is 5-15°.

[0049] A millimeter-wave radar antenna is fixed in the middle position below the bow deck, and its pitch and roll angles are unified with the ship's coordinate system {B} through a special bracket;

[0050] A high-precision IMU and a dual-frequency GNSS antenna are positioned at the ship's center of gravity. The IMU and camera are rigidly connected to each other for external parameter calibration. The ship's onboard computing unit and low-power controller are installed in the sealed midship compartment. Dual electric propulsion units and electronic speed controllers (ESCs) are symmetrically arranged on both sides of the stern. Redundancy is reserved in the rudder angle and thrust channel to achieve low-speed approach control that is compatible with differential and rudder propeller.

[0051] Furthermore, the millimeter-wave radar connects to the STM32 and embedded GPU via CAN or Ethernet;

[0052] The high-definition camera and infrared camera are connected to the embedded GPU via Gigabit Ethernet or USB 3.0 / CSI-2 respectively. The IMU outputs data to the two-level control unit simultaneously via SPI and GNSS via UART.

[0053] The STM32 serves as the underlying master station, acquiring real-time closed-loop data for propulsion and servo motors, as well as high-speed I / O for obstacle avoidance ultrasonic / laser ranging. It also exchanges trajectory and attitude commands with the embedded GPU via Ethernet or UART.

[0054] Shore-based communication and remote backhaul are achieved through dual links of 4G / 5G and BeiDou short message service.

[0055] The entire ship uses the 1PPS signal of GNSS as the hardware reference, and uses PTP for clock alignment on the Ethernet side.

[0056] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0057] 1. This invention achieves high-precision detection, identification, and positioning of people who have fallen into the water under complex sea conditions through the coordinated operation of a millimeter-wave radar detection module, a shipborne visual inspection module, a multi-sensor fusion positioning module, an autonomous navigation control module, and a data communication module, and ensures that the unmanned vessel can safely and efficiently approach the target to carry out rescue operations.

[0058] 2. In the long-range detection stage, the millimeter-wave radar detection module of this invention can still maintain a target detection rate of over 95% even in environments with low visibility, insufficient lighting, or large waves. It can also perform preliminary screening through RCS feature comparison, reducing the false alarm rate to more than half that of traditional single radar solutions, thus reducing the burden of subsequent processing from the source.

[0059] 3. In the close-range identification stage, the shipborne visual detection module of this invention adopts the YOLOv11-USVEDT algorithm optimized for the dynamic marine environment. Compared with the conventional YOLOv11 model, the detection accuracy is improved by about 12% under strong reflection and wave interference conditions, and the average detection accuracy (mAP@0.5) for small targets is improved to more than 93%, effectively reducing false detections and missed detections.

[0060] 4. This invention employs a multi-sensor fusion positioning module that combines spatiotemporal data from millimeter-wave radar and visual detection, and utilizes the attitude of the unmanned vessel and camera calibration parameters for coordinate system registration. The positioning accuracy can be stably maintained within ±0.5 meters under dynamic navigation conditions, which is about 40% higher than existing multi-sensor solutions, providing a reliable basis for subsequent navigation.

[0061] 5. The autonomous navigation control module of this invention generates the optimal trajectory based on the fusion positioning results and avoids obstacles in real time, so that the heading deviation of the unmanned vessel is less than 2° under the conditions of level 3 wind and waves, and the average time to reach the target position is shortened by about 18%, which significantly improves the rescue response speed.

[0062] 6. The data communication module of this invention ensures real-time data transmission and mission command interaction between the unmanned vessel and the shore-based command center. In sea areas with weak signal, it can reliably transmit critical mission data via BeiDou short message service, ensuring the controllability and safety of the mission process.

[0063] 7. Technically, this invention significantly improves the detection accuracy, identification reliability, positioning accuracy, and autonomous navigation performance of maritime search and rescue systems. Economically, it can reduce the cost of manpower patrols and increase the completion rate of search and rescue missions. Socially, it helps to shorten the waiting time for rescue of people who have fallen into the water and improve the survival rate. It has significant practical value and promotion significance. Attached Figure Description

[0064] Figure 1 This invention relates to the YOLOv11-USVEDT visual detection model architecture;

[0065] Figure 2 This is a block diagram of the drowning detection algorithm based on RCS parameters of this invention;

[0066] Figure 3 This is a schematic diagram illustrating an example connection between an STM32 and a feasible embedded control device;

[0067] Figure 4 This is a schematic diagram of the unmanned vessel structure of the present invention;

[0068] Figure 5 This is a Cartesian coordinate system diagram of the millimeter-wave radar of this invention;

[0069] Figure 6 This is an example of CAN message parsing in this invention;

[0070] Among them, 1. High-definition camera, 2. Millimeter-wave radar, 3. GNSS antenna, and 4. Shipborne computing unit. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] Example 1

[0073] This application provides a multi-sensor fusion-based search and rescue system installed on an unmanned surface vessel (USV). Through the combined application of millimeter-wave radar and an improved visual detection algorithm, it achieves rapid long-range screening, accurate close-range identification and location of people in the water, and automatically controls the USV to approach the target for rescue. It should be noted that this invention is particularly suitable for search and rescue in complex sea conditions, effectively reducing false detection rates and improving response speed. The search and rescue system described in this application mainly includes a millimeter-wave radar detection module, a shipborne visual detection module, a multi-sensor fusion positioning module, an autonomous navigation control module, and a data communication module. Compared with existing technologies that rely on a single sensor for target identification, this invention, based on millimeter-wave radar RCS pre-screening, combines an improved YOLOv11-USVEDT model with infrared discrimination results for secondary confirmation, constructing a hierarchical, multi-modal search and identification mechanism, thereby effectively reducing the visual false detection rate under conditions of long-range detection, nighttime, and wave interference.

[0074] The system described in this application includes: a millimeter-wave radar detection module, a shipborne visual inspection module, a multi-sensor fusion positioning module, an autonomous navigation control module, a data communication module, and a power supply and distribution unit.

[0075] The millimeter-wave radar detection module is installed in the center of the forward deck, with a water clearance of ≥0.8 m. Its mounting base includes a pitch fine-tuning mechanism (±10°). The radar operates in the selectable frequency band of 76–81 GHz, has a horizontal field of view ≥90°, and a range of up to 300 m. The radar connects to the shipboard computing unit 4 via Ethernet or CAN-FD, and is powered from a 24 V bus via DC-DC converter to the radar's rated voltage.

[0076] The shipborne visual inspection module includes a forward-looking high-definition camera 1 (with a low-distortion lens), an infrared thermal imaging device, and a computing unit (embedded GPU / AI accelerator). The high-definition camera 1 and the infrared camera are fixed together on the vibration-damping unmanned surface vessel's gimbal, and the optical axes of both are kept at an adjustable angle of 5–15° with the water surface to ensure stable imaging capabilities under different weather and lighting conditions.

[0077] Both types of cameras are connected to the computing unit via Gigabit Ethernet / USB 3.0 interfaces to achieve synchronous acquisition and processing of multi-source images. The cameras are rigidly connected to the IMU for external parameter calibration, thereby ensuring the uniformity and fusion of the infrared and visible light imaging coordinate systems in a dynamic shipboard environment.

[0078] Multi-sensor fusion positioning module: Runs sensor spatiotemporal synchronization, extrinsic parameter management, and fusion algorithms on the computing unit; connects to IMU (gyroscope, accelerometer), GNSS (Global Navigation Satellite System), and ship attitude / velocity gauge. GNSS output PPS / time pulse is used for a unified time reference across the entire system (PTP optional).

[0079] The autonomous navigation control module includes a trajectory planning and motion controller, which controls the thrusters and servos (or differential propulsion). It connects to obstacle avoidance sensors (ultrasonic / laser / short-range radar) via serial port / Ethernet.

[0080] The data communication module uses dual-link redundancy of 4G / 5G cellular and BeiDou short message; it establishes an MQTT channel with the shore-based command center to transmit the latitude and longitude of the unmanned vessel, monitoring confidence level, video summary and vessel status.

[0081] Coordinate system definition and calibration

[0082] The coordinate system of the unmanned surface vessel (USV) incorporating the system described in this application is defined and calibrated. Specifically, the world coordinate system {W} (ENU or NED), the hull coordinate system {B} (x-forward, y-right, z-down), the radar coordinate system {R}, and the camera coordinate system {C} are defined. The rigid body external participation intrinsic parameters are obtained through calibration.

[0083] 1) Camera intrinsic parameters K and distortion parameters are obtained via calibration plate;

[0084] 2) Extrinsic parameter matrix and Obtained through joint calibration (camera observation of markers, radar ranging, and IMU attitude);

[0085] 3) Ship attitude Obtained by IMU / GNSS tightly coupled estimation. For example... Figure 5 As shown.

[0086] The application method of an unmanned vessel including the system described in this application specifically includes the following steps:

[0087] Step 1: Initialization

[0088] Power on the search and rescue system of the unmanned vessel. After power-on, perform time synchronization (GNSS PPS / PTP), sensor self-test, attitude alignment, external payload and health status reporting; enter cruise surveillance mode.

[0089] Step 2: Long-range search and RCS pre-screening (millimeter-wave radar detection module)

[0090] Millimeter-wave radar 2 performs beam scanning at a refresh rate of 10–20 Hz and outputs target measurement data. Where r is the range, θ is the azimuth, v is the radial velocity, A is the echo amplitude, and σ is the RCS estimate; CFAR (constant false alarm rate detection, CA-CFAR or OS-CFAR) is used for detection in the range-Doppler plane, and then screening is performed based on the RCS statistical interval of the human body in this frequency band and the velocity prior.

[0091] 1) RCS window: (e.g., 0.05–1.5 m², which can be adaptively adjusted according to sea conditions);

[0092] 2) Speed ​​constraints: Below the threshold (typically, a person falling into the water has a relatively low radial velocity);

[0093] 3) Trajectory consistency: Perform multi-frame correlation and Kalman tracking on candidate points to eliminate occasional wave peak reflections.

[0094] Output candidate set And for each candidate generated image ROI, the pointing angle and approximate depth prior are calculated.

[0095] To obtain the attribute information of radar-detected targets, a CAN analyzer can be used to connect the radar to an embedded device or control center, capture and export the raw CAN data, and parse the target number, distance, speed, horizontal and vertical position, target status, and other information contained in the 0x60B message. During the parsing process, corresponding fields need to be extracted based on the signal start bit and length, and converted into actual physical values ​​such as distance (m), relative speed (m / s), and radar cross-section (RCS) according to the resolution, thus achieving a complete conversion from raw message to structured target data. Figure 6 As shown.

[0096] Step 3: Close-range precise identification (shipborne visual inspection module)

[0097] The unmanned surface vessel's gimbal points to the candidate location, while the visible light camera and infrared thermal imaging camera simultaneously acquire front-view image frames. The improved YOLOv11-USVEDT model performs multimodal detection on the visible light and infrared images, outputting bounding boxes for human targets both within the ROI region and across the entire image. The model is evaluated using a combination of feature layer fusion and decision-level fusion to improve robustness in low-light, foggy, and highly reflective scenarios. For shipboard environments, the following enhancements were implemented: an EfficientNetV2 feature extraction backbone, a DySample dynamic upsampling method, a histogram transformer module, a WIoU loss function combining dynamic gradient gain and outlier evaluation, and anti-reflective / spray data augmentation. By leveraging the complementary characteristics of dual-source images, the model effectively reduces the probability of missed detections at night and under adverse weather conditions, establishing visual measurements for each candidate. .

[0098] Step 4: Radar-Vision Fusion Positioning (Multi-Sensor Fusion Positioning Module)

[0099] The fusion module performs the following under a unified time scale: and Perform spatial registration and state estimation:

[0100] 1) Geometric registration: from the center ray of the camera pixel Transform to {B}: By combining an approximate planar model of the sea surface (or the instantaneous water surface normal obtained from wave height gauge / attitude estimation) to find the intersection point of the ray and the water surface, the prior relative position calculated solely by visual estimation is obtained. ;

[0101] 2) Measurement model: Radar polar coordinate measurement Visual orientation measurement (From pixel to azimuth); Establish a joint observation model ( (Including the target's position in {B} / {W}).

[0102] 3) Filtering and Association: Extended Kalman Filter (EKF) or Unscented Kalman Filter (UKF) is used for state updates; Nearest Neighbor (NN) or Hungarian algorithm is used for measurement-trajectory association in multi-target scenarios; high-confidence visual measurements are used to correct the radar range scale, and radar-stabilized range is used to suppress visual depth drift. The output shows the target's latitude and longitude in {W} and its relative position in {B}. And give the covariance.

[0103] Step 5: Autonomous Navigation and Dynamic Obstacle Avoidance (Autonomous Navigation Control Module)

[0104] After obtaining the relative / absolute position of the target, the path planner generates a proximity trajectory: the far segment uses an A* path, and the mid-to-near segment uses LOS (line of sight) guidance control, with the control law as follows:

[0105]

[0106] in, For the current heading, For the desired heading, These are the lateral deviation and the forward error, respectively. To increase gain. Obstacle avoidance uses an artificial potential field to correct local velocity commands and limit the minimum safe distance. Thrust / rudder angle is achieved through closed-loop tracking using PID control. When wind and waves are ≥3, attitude feedforward is fused for disturbance rejection compensation.

[0107] Step Six: Communication and Human-Machine Collaboration (Data Communication Module)

[0108] The system reports the target ID, latitude and longitude, fusion confidence score, latest image summary, navigation status, and remaining energy to the shore base at a frequency of 1–5 Hz; it receives and executes manual commands such as "confirm / cancel / change course / return" in real time. In areas with weak coverage, it automatically switches to BeiDou short message transmission, ensuring mission controllability.

[0109] Step 7: Task Closure

[0110] When the joint confidence level of vision-radar exceeds the threshold And the uncertainty of the target position When the target falls below the threshold, it enters the "approaching state"; after reaching the set radius (e.g., 5–10 m), it triggers subsequent rescue actions (e.g., throwing rescue equipment, voice broadcasting, etc., which are not the focus of this embodiment), and continues to monitor the target until it receives a recovery command.

[0111] The abbreviations and Chinese names used in this article are explained as follows: USV (Unmanned Surface Vehicle), RCS (Radar Cross Section), ROI (Region of Interest), YOLO (You Only Look Once), YOLOv11-USVEDT (Unmanned Ship Vision Enhanced Detection and Tracking), IMU (Inertial Measurement Unit), GNSS (Global Navigation Satellite System), EKF / UKF (Extended / Unscented Kalman Filter), CFAR (Constant False Alarm Rate), LOS (Line-Of-Sight), MHT / JPDA (Multiple Hypothesis Tracking / Joint Probability Data Association), PTP (Precision Time Protocol), DWA (Dynamic Window Approach), MPC (Model Predictive Control).

[0112] Example 2

[0113] Another search method for the system employs a combination of RCS-based hierarchical triggering and visible / infrared light recognition. The overall system structure remains as shown in the attached figure. Figure 4As shown, the unmanned surface vessel (USV) adopts a sea-resistant catamaran hull and aluminum alloy frame. An integrated vibration-damping gimbal is installed at the bow, coaxially mounting a forward-looking high-definition camera 1 (low-distortion lens) and an infrared thermal imaging camera. The optical axis angle with the water surface is adjustable from 5–15° to accommodate both long-range and close-range overhead views. A 77 GHz millimeter-wave radar 2 antenna is fixed in the center below the bow deck, its pitch and roll angles unified with the ship's coordinate system {B} via a dedicated bracket. A high-precision IMU and a dual-frequency GNSS antenna 3 are arranged near the ship's center of gravity, with the IMU and camera rigidly connected for extrinsic parameter calibration. An onboard computing unit 4 (embedded GPU / AI accelerator for running vision and fusion algorithms) and a low-power controller (STM32 series MCU for real-time I / O and actuator control) are installed in the mid-seawater-resistant sealed compartment. Dual electric propulsion units and electronic speed controllers (ESCs) are symmetrically arranged at the stern, with redundancy reserved in the rudder angle and thrust channels to achieve differential and propeller-compatible low-speed approach control. The power system consists of a main lithium battery pack and an emergency power supply, with independent 5V / 12V / 24V isolated DC-DC circuits. Key sensors and controllers have independent fuses and surge protection designs. The overall protection level of the ship is no less than IP66, and exposed connectors are made of corrosion-resistant materials and coated with anti-salt spray coating.

[0114] Hardware connectivity and time synchronization are shown in the attached document. Figure 3 As shown, millimeter-wave radar 2 is connected to the STM32 and embedded GPU via CAN or Ethernet. Visible light and infrared cameras are connected to the embedded GPU via Gigabit Ethernet or USB 3.0 / CSI-2, respectively. The IMU outputs data to both control units simultaneously via SPI, and the GNSS outputs data via UART. The STM32 acts as the underlying master station, responsible for real-time closed-loop propulsion and servo motor control, as well as high-speed I / O acquisition such as obstacle avoidance ultrasonic / laser ranging. It also exchanges track and attitude commands with the embedded GPU via Ethernet or UART. Shore-based communication and remote backhaul are handled by dual-link redundancy of 4G / 5G and BeiDou short message service. The entire ship uses the 1PPS signal of GNSS as the hardware reference, and uses PTP (Precision Time Protocol) for clock alignment on the Ethernet side to ensure time consistency of radar frames, camera frames, and IMU data within a 100 ms measurement window, facilitating subsequent fusion positioning.

[0115] The overall process and module division of the software and algorithm are shown in the attached figure. Figure 1 With appendix Figure 2As shown. In the long-range search phase, the original radar points are processed by range-Doppler and then enter constant false alarm rate (CFAR) detection. Subsequently, multi-frame clustering and point association are performed to obtain a candidate set. During the candidate generation process, the system calculates the radar cross-section (RCS) features, azimuth, and range threshold of the target, and performs a first-level screening of non-human scattering bodies (such as wave crests and floating debris) based on sea state adaptive thresholds to obtain suspected human candidates (see...). Figure 2 (A block diagram of the drowning detection algorithm based on RCS parameters). When the candidate is stable for more than three frames, the gimbal is pointed at the candidate location, entering the close-range accurate recognition stage. Close-range recognition is performed by... Figure 1 The YOLOv11-USVEDT shipborne visual inspection model shown here is responsible for:

[0116] In the visible light channel, the model employs EfficientNetV2 to reconstruct the feature extraction backbone and a composite scaling strategy to enhance multi-scale feature representation. Dynamic upsampling using DySample improves the reconstruction quality of small targets, and a histogram transformer module is introduced to suppress contrast loss in areas of strong reflection, haze, and degradation. At the loss function level, WIoU bounding box regression, combining dynamic gradient gain and outlier evaluation, is used to improve robustness. In the infrared channel, thermal radiation saliency is utilized for supplementary detection in low-light, nighttime, and backlight scenes. Both channels support fusion at the feature and decision layers: firstly, cross-modal attention is applied to time-aligned infrared and visible light feature maps to enhance the separability of weakly textured targets; secondly, collaborative voting is performed on bounding boxes, categories, and confidence scores at the detection output to mitigate false detections in a single modality. The comprehensive output establishes multimodal visual measurements (pixel coordinates, box scale, category and confidence score, infrared intensity statistics, etc.) for each candidate target and provides pixel-level occlusion ratio and reflection risk indicators for adaptive weighting in the downstream localization process.

[0117] The fusion localization module runs on the embedded GPU side of the fusion process, maintaining the extrinsic parameter relationships between {W} (world / geographic coordinates), {B} (ship coordinates), {R} (radar coordinates), and {C_v} / {C_ir} (visible / infrared camera coordinates). These extrinsic parameters are obtained jointly through offline calibration and online micro-correction. The system uses an extended Kalman filter (EKF) as the core state estimator. The state vector includes the planar position and velocity of the person in the water, as well as observation confidence parameters. The measurement model intersects the radar polar coordinate measurements with the pixel rays from the dual cameras under a known sea surface height constraint. It utilizes camera intrinsic attitude (from the IMU) to project the pixel-ray into the ground intersection point in the world frame. Infrared intensity is used to increase the weight of visual measurements in low visibility conditions. To suppress the impact of ship roll on instantaneous localization, the filter introduces attitude compensation and short-term smoothing, and sets a measurement consistency check (Madara distance threshold) to avoid mistaking wave crest reflections or specular highlights as valid measurements. The fusion output is a target position and velocity estimate in geographic coordinate system, with a root mean square error of position that is better than 0.5 m (95% confidence) under measured sea conditions.

[0118] The autonomous navigation and control module uses fused positioning results as input and combines electronic charts and safety domain constraints to generate an approach track. For medium-to-long-range segments, it employs line-of-sight (LOS) track tracking and A* global path generation. For short-range segments, it switches to a speed and rudder angle constraint controller based on model predictive control (MPC), receiving obstacle avoidance ranging and radar near-range echo information uploaded by the STM32 in real time for fine-tuning the path, ensuring a safe rendezvous attitude relative to the target under crosswind and sea conditions. When the distance threshold is less than 10 m, it enters a fine-management mode, decelerating and maintaining side alignment, triggering the deployment of rescue devices or throwing of rescue ropes. The data communication module uses 4G / 5G as the primary link and BeiDou short message service as a backup link, transmitting target position, visible light and infrared keyframes, identification confidence level, and track status in real time. It also has the capability to receive shore-based manual intervention and mission replanning commands. The link employs end-to-end encryption and breakpoint resume capabilities to ensure packet loss resistance and security.

[0119] The specific execution sequence of the method is as follows: After the system is powered on, time synchronization and sensor self-check are completed, and the camera-IMU-radar external participation sea surface height model is loaded and initialized; the unmanned surface vessel cruises according to a preset route or area coverage strategy and continuously performs radar search and candidate generation; when the candidate meets the RCS and spatiotemporal consistency conditions, the gimbal is driven to point and visible light / infrared joint detection is initiated to complete target confirmation; the fusion positioning module performs weighted estimation of multi-source measurements and outputs the target's position and velocity in the {W} coordinate system; the autonomous navigation control module generates an approach track based on this and performs path correction in conjunction with real-time obstacle avoidance during navigation; after approaching to a safe distance, fine berthing and rescue actions are initiated, and the mission data is timestamped and transmitted back to the shore-based center throughout the process. In the above steps, close-range joint detection can be carried out in scanning mode before the gimbal points (especially at night with infrared wide-area scanning as a priority), or the radar trigger threshold can be lowered in bad weather to improve candidate recall; therefore, candidate screening and visual confirmation can be implemented in parallel or partially interchanged in engineering, but this is on the premise of not reducing safety and overall false alarm control.

[0120] The abbreviations and Chinese names in this article are explained as follows: RCS (Radar Cross Section), CFAR (Constant False Alarm Rate), IMU (Inertial Measurement Unit), GNSS (Global Navigation Satellite System), PTP (Precision Time Protocol), PPS (Pulse Per Second), EKF (Extended Kalman Filter), MPC (Model Predictive Control), LOS (Line-of-Sight), WIoU (Weighted Intersection over Union), ROI (Region of Interest), CAN (Controller Area Network), UART (Universal Asynchronous Receiver / Transmitter), GigE (Gigabit Ethernet), CSI-2 (Camera Serial Interface-2), ESC (Electronic... Speed ​​Controller (electronic speed controller), USV (Unmanned Surface Vehicle).

[0121] This embodiment can complete the entire process of "long-range search - close-range confirmation - fusion positioning - autonomous approach - rescue execution" in a closed loop using a single unmanned surface vessel platform. Furthermore, it improves the overall detection rate and reliability by complementing radar with visible light / infrared in harsh sea conditions, demonstrating clear engineering replicability and operability.

[0122] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

[0123] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0124] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-sensor fusion-based maritime search and rescue system, the system being applied to an unmanned ship, characterized in that, The system comprises a millimeter wave radar detection module, which is electrically connected with a shipborne control computing unit, is used for scanning a sea surface target at a long distance, and acquires a spatial position and a reflection intensity parameter, and preliminarily screens a target needing to be rescued according to a radar cross section characteristic value; The shipborne visual detection module comprises a high-definition camera (1), an infrared imaging device, and a shipborne GPU running an improved YOLOv11-USVEDT, realizes real-time detection of the target needing to be rescued through sea environment optimization, and comprises the shipborne visual detection module. The multi-sensor fusion positioning module runs in the shipborne control unit, fuses ranging data detected by the millimeter wave radar and pixel coordinates, target size and attitude data output by visual recognition, combines mapping relationship between the internal and external parameters of the high-definition camera (1) and a radar coordinate system, and calculates geographical position information of the target needing to be rescued, wherein the geographical position information provides data input for subsequent autonomous navigation. The autonomous navigation module generates a track of the unmanned ship approaching the target according to the geographical position information output by the multi-sensor fusion positioning module, and approaches the target needing to be rescued. The data communication module transmits search and rescue information to a shore-based command center in real time through a 4G / 5G or Beidou short message link.

2. The multi-sensor fusion based maritime search and rescue system as claimed in claim 1, wherein, The shipborne visual detection module optimizes a sea environment, specifically as follows: In a visible light channel, an EfficientNetV2 is used to reconstruct a feature extraction backbone, multi-scale feature representation capability is enhanced, DySample is used to improve small target detection precision, a histogram transformer module is used to strengthen feature extraction in strong light reflection and degradation areas, and a WIoU loss function combining dynamic gradient gain and outlying degree evaluation is used to optimize boundary box regression performance; In an infrared channel, a thermal radiation feature is used to assist in distinguishing low-visibility and night targets; Through complementary fusion of visible light and infrared information, real-time human target detection is realized in a sea scene.

3. The multi-sensor fusion based maritime search and rescue system as claimed in claim 1, wherein, The autonomous navigation module adjusts a path and a heading in combination with real-time obstacle avoidance sensor data during the process in which the unmanned ship approaches the target.

4. A method of using the multi-sensor fusion based maritime search and rescue system of claim 1, characterized in that, The method comprises the following steps: Step one, initialization, the system is powered on, time synchronization, sensor self-checking, attitude alignment, external parameter loading and health state reporting are performed, and after completion, a cruising monitoring state is entered; a world coordinate system {W}, a ship body coordinate system {B}, a radar coordinate system {R} and a camera coordinate system {C} are defined for the system; rigid body external and internal parameters are obtained through calibration: Camera internal parameters K and distortion parameters are obtained through a calibration board; extrinsic matrix with derived from joint calibration (camera observing markers, radar ranging, and IMU attitude) Hull attitude Obtained from IMU / GNSS tight coupling estimation; Step two, long-range search and RCS pre-screening, millimeter wave radar performs beam scanning with 10-20 Hz refresh rate, outputs target measurement where r is the distance, θ is the azimuth, v is the radial velocity, A is the echo amplitude, and σ is the RCS estimate. Adopt CFAR method to judge in distance-Doppler plane, according to the RCS statistical interval of human body in this frequency band and the speed priori to filter; output candidate set And generate the pointing angle and approximate depth priori of image ROI for each candidate; A CAN analyzer is used to connect the radar with an embedded device or a control center, attribute information of a target detected by the radar is acquired, original CAN data is captured and exported, and target number, distance, speed, transverse and longitudinal position and target state information contained in a 0x60B message are parsed; In the parsing process, corresponding fields are extracted according to a signal start bit and a length, and are converted into actual physical values through resolution conversion, so that the original message is converted into structured target data; Step three, close-range identification, the unmanned ship cloud platform points to the candidate direction, and the high-definition camera (1) and the infrared imaging device synchronously collect the front view image frame; the improved YOLOv11-USVEDT model performs multi-modal detection on the visible light and infrared images, outputs the bounding box of the target needing rescue in the ROI region and in the full image, the category c, the confidence s, and combines feature layer fusion and decision level fusion , Step four, radar-vision fusion localization, the fusion module fuses the data from the radar and vision sensors in a unified time scale to do spatial registration and state estimation. With the radar and vision sensors in a unified time scale to do spatial registration and state estimation. Step five, autonomous navigation and dynamic obstacle avoidance, after obtaining relative / absolute positions of the target, a path planner generates an approaching track: a far section adopts A* path, a middle and near section adopts LOS (line of sight) guidance control, and a control law is as follows wherein is the current heading, is the desired heading, are the lateral deviation and forward error, respectively, is the gain; obstacle avoidance uses artificial potential field to modify the local velocity command, limits the minimum safety distance; the propeller / rudder angle realizes closed-loop tracking by PID; when the wind and wave are ≥ 3, the attitude feedforward is fused to perform anti-disturbance compensation; Step six, communication and human-computer cooperation, the system reports to the shore base through the data communication module at a frequency of 1-5 Hz: target ID, latitude and longitude, fusion confidence, latest image summary, navigation state and remaining energy; receive manual "confirm / cancel / change course / return" instructions and execute them in real time; Step seven, when the visual-radar combined confidence exceeds a threshold and the target position uncertainty is below a threshold, enter the "approach state"; trigger the subsequent rescue action upon reaching the set radius and continue to maintain target surveillance until a recovery instruction is received.

5. The multi-sensor fusion based maritime search and rescue system application method according to claim 4, wherein, The shipboard visual detection module in step three makes the following enhancements in the shipboard environment: EfficientNetV2 feature extraction backbone, DySample dynamic upsampling method, histogram transformer module, WIoU loss function combined with dynamic gradient gain and outlier evaluation, and anti-glare / spray data enhancement; By complementary properties of the two source images, a visual measurement is established for each candidate .

6. The multi-sensor fusion based maritime search and rescue system application method according to claim 4, wherein, The millimeter wave radar detection module is installed in the central position of the front deck of the unmanned ship, and has a water separation height of ≥0.8 m relative to the water surface. The installation base is provided with a pitch fine adjustment mechanism. The radar operating frequency band is 76-81 GHz, the horizontal field of view is ≥90°, and the range is 300 m. The radar is connected with the shipboard computing unit (4) through Ethernet or CAN-FD, and the power supply is converted from the 24V bus to the rated voltage of the radar through DC-DC conversion.

7. The multi-sensor fusion based maritime search and rescue system application method according to claim 4, wherein, The multi-sensor fusion positioning module in step 4 performs spatial registration and state estimation under a unified time scale With spatial registration and state estimation: (1) Geometric registration: from camera pixel center ray to {B}: ; combined with the sea surface approximate plane model to find the intersection of the ray and the water surface, to get the relative position prior calculated only by vision ; (2) Measurement Model: Radar Polar Measurement , Visual Bearing Measurement ; Establish Joint Observation Model , Where, Contains the location of the target in {B} / {W}; (3) Filtering and association, extended Kalman filter or unscented Kalman filter is used for state update; nearest neighbor or Hungarian algorithm is used for measurement-track association under multi-target; high-confidence measurement of vision is used to correct radar range scale, and stable radar range is used to suppress vision depth drift; the longitude and latitude of the output target in {W} and the relative position in {B} and covariance are given.

8. An unmanned ship comprising the multi-sensor fusion-based maritime search and rescue system of claim 1, characterized in that, The unmanned ship adopts a catamaran hull and an aluminum alloy skeleton, and an integrated vibration reduction gimbal is arranged in the bow. A high-definition camera (1) and an infrared thermal imaging camera are coaxially installed on the gimbal, and the camera optical axis has an angle of 5-15° with the water surface. A millimeter wave radar (2) antenna face is fixed below the middle position of the bow deck, and the pitch angle and roll angle are unified through a special support and the ship body coordinate system {B}; A high-precision IMU and a dual-frequency GNSS antenna (3) are arranged at the center of gravity of the ship body, and the IMU and the camera are rigidly connected to implement external parameter calibration. A shipboard computing unit (4) and a low-power controller are installed in the middle sealed cabin, and double electric propulsion units and electronic speed controllers (ESCs) are symmetrically arranged at the tail. Redundancy is reserved in the rudder angle and thrust channel to realize differential and rudder-propeller compatible low-speed approaching control.

9. An unmanned ship comprising a maritime search and rescue system based on multi-sensor fusion according to claim 8, characterized in that, The millimeter wave radar (2) is connected with the STM32 and the embedded GPU through CAN or Ethernet; The high-definition camera (1) and the infrared camera are connected with the embedded GPU through gigabit Ethernet or USB3.0 / CSI-2, respectively. The IMU is connected through SPI, and the GNSS antenna (3) outputs data to the two-level control unit through UART; The STM32 is the bottom master station, which realizes real-time closed loop control of propulsion and rudder, high-speed I / O collection of obstacle avoidance ultrasonic / laser ranging, and exchanges of track and attitude instructions with the embedded GPU through Ethernet or UART; The shore-based communication and remote backhaul are realized through 4G / 5G and Beidou short message dual-link; The whole ship takes the 1PPS signal of the GNSS antenna (3) as the hardware reference, and uses PTP for clock alignment on the Ethernet side.

Images