An auxiliary search and rescue system and method for water distress persons based on visual perception

Abstract

The application provides an auxiliary search and rescue system and method for water distress personnel based on visual perception, wherein the system comprises: a data acquisition module, which is used for acquiring multi-source heterogeneous data, performing spatio-temporal synchronous fusion on the multi-source heterogeneous data, and generating a fusion data cube, wherein the multi-source heterogeneous data comprises visual data and spatial data; a distress data extraction module, which is used for performing distress target drift feature extraction processing based on the fusion data cube, and generating a drift feature vector sequence; performing environment risk field modeling processing based on the fusion data cube, and generating a dynamic risk map; an auxiliary rescue module, which is used for performing position prediction processing according to the drift feature vector sequence, generating a distress personnel position probability cloud map, and performing multi-target path planning processing based on the distress personnel position probability cloud map, the dynamic risk map and the current position of a rescue ship, and generating a rescue path. The system can improve the success rate of search and rescue operations and the safety of personnel.

Description

Technical Field

[0001] This invention belongs to the field of information technology for water rescue, and in particular relates to a visual perception-based auxiliary search and rescue system and method for people in distress at sea. Background Technology

[0002] With the development of maritime search and rescue technology, modern systems can now initially locate people in distress using satellite positioning, drone patrols, and other means. Traditional methods rely on static environmental data and pre-set path planning algorithms, such as using fixed-time weather forecasts to delineate search areas or planning rescue routes based on historical ocean current data. These technologies are effective in handling known environments and can provide basic search and rescue plans.

[0003] However, the aquatic environment is constantly changing, with wind direction, currents, and even floating obstacles all in real time. Existing systems struggle to continuously track these dynamic factors, leading to two key shortcomings: first, the actual location of a person in distress drifts rapidly with environmental changes, and the deviation between the statically predicted location and the actual location widens over time; second, rescue route planning lacks real-time risk perception capabilities, failing to promptly avoid newly emerging obstacles (such as moving floating objects or reefs). This often forces rescue teams to face a dilemma between route safety and timeliness, where the shortest route may become infeasible due to unforeseen risks, while detours may delay rescue efforts. Therefore, a search and rescue decision support mechanism that can simultaneously assess environmental risks and optimize routes in real time within a dynamic environment is urgently needed. Summary of the Invention

[0004] Therefore, it is necessary to provide a visual perception-based auxiliary search and rescue system and method for people in distress at sea, which can provide optimized path guidance for rescue decisions in complex aquatic environments, taking into account both timeliness and safety, thereby improving the success rate of search and rescue operations and the safety of personnel.

[0005] Firstly, this application provides a visual perception-based auxiliary search and rescue system for people in distress at sea, comprising a data acquisition module, a distress data extraction module, and an auxiliary rescue module:

[0006] The data acquisition module is used for:

[0007] Acquire multi-source heterogeneous data, perform spatiotemporal synchronous fusion of the multi-source heterogeneous data, and generate a fused data cube, wherein the multi-source heterogeneous data includes visual data and spatial data;

[0008] The distress data extraction module is used for:

[0009] Based on the fused data cube, drift feature extraction processing of distressed targets is performed to generate a drift feature vector sequence;

[0010] Environmental risk field modeling is performed based on fused data cubes to generate dynamic risk maps;

[0011] The auxiliary rescue module is used for:

[0012] Based on the drift feature vector sequence, a location prediction process is performed to generate a probability cloud map of the location of the distressed person.

[0013] Based on the probability cloud map of the location of the distressed persons, the dynamic risk map, and the current location of the rescue vessel, multi-objective path planning is performed to generate a rescue path, which is used to guide rescuers to conduct water search and rescue.

[0014] In one embodiment, the data acquisition module includes a multi-source data acquisition unit, a spatiotemporal synchronization processing unit, and a data fusion unit:

[0015] A multi-source data acquisition unit is used to acquire visible light video streams and infrared video streams as visual data;

[0016] Real-time ocean current vectors, wind direction and speed data, and sea surface temperature data are acquired and georeferenced to serve as spatial data.

[0017] The spatiotemporal synchronization processing unit is used to acquire the timestamps of visual data and spatial data. It uses a spatial registration algorithm to map the pixel coordinates of the visible light video stream and infrared video stream in the visual data to the geographic coordinate system of the spatial data, thereby obtaining spatiotemporally aligned multimodal data.

[0018] The data fusion unit is used to integrate and correlate spatiotemporally aligned multimodal data in terms of dimensions to generate a fused data cube, which includes time dimension, spatial dimension and data dimension.

[0019] In one embodiment, the distress data extraction module includes a drift feature extraction unit and a risk field modeling unit:

[0020] The drift feature extraction unit is used for:

[0021] A convolutional neural network based on an attention mechanism is used to detect distressed targets in visible light video streams and infrared video streams.

[0022] The target motion trajectory is generated by combining the appearance features and motion consistency of the distressed target with the spatiotemporal correlation algorithm, and the displacement vector of the distressed target between consecutive frames is calculated to obtain the drift feature vector sequence.

[0023] The risk field modeling unit is used to perform semantic segmentation on visual data to identify dynamic obstacles and static danger zones. It combines real-time ocean current vectors to predict the movement trend of dynamic obstacles and construct a dynamic risk map, which includes a basic risk field and dynamic risk increments.

[0024] In one embodiment, the drift feature extraction unit includes a target detection subunit, a trajectory generation subunit, and a displacement calculation subunit:

[0025] The target detection subunit is used to fuse features from visible light video streams and infrared video streams, identify the contours of distressed targets through an attention-based convolutional neural network, and output the target bounding box and the corresponding confidence score.

[0026] The trajectory generation subunit is used to generate target motion trajectories with consecutive timestamps by associating the IDs of the same distressed target with the target bounding boxes of adjacent frames of visible light video stream and infrared video stream through a motion consistency constraint algorithm.

[0027] The displacement calculation subunit is used to calculate the displacement vector in the image coordinate system based on the target's motion trajectory. Combined with the real-time ocean current vector in the fused data cube, it generates a sequence of drift feature vectors in the geographic coordinate system through affine transformation.

[0028] In one embodiment, the risk field modeling unit includes an obstacle recognition subunit, a risk prediction subunit, and a rasterization processing subunit:

[0029] The obstacle recognition subunit is used to perform pixel-level classification of video frames through a semantic segmentation model to obtain dynamic obstacles and static hazard areas. The attributes of dynamic obstacles include category and boundary polygon.

[0030] The risk prediction subunit is used to calculate the displacement vector of dynamic obstacles based on the real-time ocean current vector in the fused data cube, obtain the movement trend through a simplified model using the Navier-Stokes equations, and generate dynamic risk increments by combining the category of dynamic obstacles.

[0031] The rasterization processing subunit is used to rasterize the acquired electronic nautical chart into a basic risk field and overlay dynamic risk increments to generate a dynamic risk map.

[0032] In one embodiment, the auxiliary rescue module includes a location prediction unit, a route planning unit, and a route output unit:

[0033] The location prediction unit is used to input the drift feature vector sequence into the physical information neural network, and predict the probability distribution of future location by embedding fluid dynamics constraint equations, and output the location probability cloud map of the distressed personnel.

[0034] The path planning unit is used to generate the Pareto optimal path set by executing a multi-objective heuristic search algorithm, with the current position of the rescue ship as the starting point, the core area of ​​the position probability cloud map as the endpoint set, and the dynamic risk map as the constraint.

[0035] The path output unit is used to evaluate the flight time and cumulative risk value of the Pareto optimal path set and generate a rescue path, which includes a waypoint sequence and a risk heat map.

[0036] In one embodiment, the path planning unit includes a ship kinematics subunit, a cost calculation subunit, and an optimization search subunit:

[0037] The ship kinematics subunit is used to establish a ship kinematics model affected by ocean currents based on real-time ocean current vectors and wind direction and speed data, and to calculate the actual sailing speed of the rescue ship at different heading angles through the ship kinematics model;

[0038] The cost calculation subunit is used to convert the dynamic risk map into a dynamic risk field grid map, calculate the cumulative risk value of the path on the dynamic risk field grid map, and calculate the path length and corresponding estimated travel time based on the actual sailing speed.

[0039] The optimization search subunit is used to simultaneously optimize path length, cumulative risk value, and estimated travel time using the multi-objective A* algorithm, and output the Pareto optimal path set.

[0040] Secondly, this application also provides a visual perception-based method for assisting in the search and rescue of people in distress at sea, including:

[0041] Acquire multi-source heterogeneous data, perform spatiotemporal synchronous fusion of the multi-source heterogeneous data, and generate a fused data cube, wherein the multi-source heterogeneous data includes visual data and spatial data;

[0042] Based on the fused data cube, drift feature extraction processing of distressed targets is performed to generate a drift feature vector sequence;

[0043] Environmental risk field modeling is performed based on fused data cubes to generate dynamic risk maps;

[0044] Based on the drift feature vector sequence, a location prediction process is performed to generate a probability cloud map of the location of the distressed person.

[0045] Based on the probability cloud map of the location of the distressed persons, the dynamic risk map, and the current location of the rescue vessel, multi-objective path planning is performed to generate a rescue path, which is used to guide rescuers to conduct water search and rescue.

[0046] The aforementioned visual perception-based maritime search and rescue system and method acquires multi-source heterogeneous data, including visual and spatial data, through a data acquisition module. This data is then spatiotemporally fused to generate a fused data cube. A distress data extraction module, based on this fused data cube, generates a drift feature vector sequence by extracting and processing the drift features of the distressed target. This captures the dynamic motion characteristics of the distressed target, providing a dynamic basis for target location prediction. An environmental risk field modeling process generates a dynamic risk map, quantifying the aquatic environmental risk in real time, thus overcoming the shortcomings of traditional systems such as delayed location prediction and lack of risk perception. An auxiliary rescue module generates a probability cloud map of the distressed person's location based on the drift feature vector sequence, ensuring the accuracy of the distressed target's location. Combining this probability cloud map, the dynamic risk map, and the current location of the rescue vessel, multi-objective path planning is performed to generate a rescue path. This approach improves the success rate of search and rescue operations through precise positioning and balances the timeliness and safety of the rescue path through multi-objective planning, providing optimized path guidance for rescue decisions and increasing the success rate of search and rescue. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of a vision-based auxiliary search and rescue system for people in distress at sea, provided in an embodiment of the present invention.

[0049] Figure 2 This is a flowchart illustrating a visual perception-based method for assisting in the search and rescue of people in distress at sea, as provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0052] Visual perception is a technology that acquires visual information about target water areas through image sensors (such as visible light and infrared cameras) and performs preliminary processing and understanding of this information. In maritime search and rescue, it can overcome the limitations of complex environments (such as foggy days and nighttime) to capture the visual characteristics of distressed targets (such as personnel and life-saving equipment) and obstacles in the water, transforming the physical scene into analyzable image data. This technology forms the basis for subsequent target detection and risk identification, providing search and rescue systems with intuitive and real-time scene information, overcoming the limitations of traditional reliance on static data, and enhancing the ability to perceive dynamic aquatic environments.

[0053] Multi-source heterogeneous data refers to a collection of diverse data with spatiotemporal correlations collected through different sensors or platforms. In search and rescue systems, it refers to a composite of visible light / infrared visual data (from drones or shipborne cameras) and environmental spatial data (such as ocean current vectors, wind speed and direction, and other physical field data). Its heterogeneity is manifested in differences in data dimensions (image pixels, physical parameters), sampling rates (video frame rates, buoy data update frequencies), and formats (raster, vector).

[0054] Semantic segmentation is an image analysis technique in the field of computer vision. It uses algorithms to classify and label image pixels according to their actual semantic categories (such as people in distress, floating objects, and reefs), enabling a refined understanding of image content. In maritime search and rescue scenarios, this technology can accurately identify dynamic obstacles (such as floating objects) and static hazardous areas (such as reef areas) from video frames of water bodies, and output spatial extent information for various targets. It can provide accurate risk source data for the construction of dynamic risk maps, ensuring the comprehensiveness of risk assessment and supporting the avoidance of high-risk areas in rescue routes.

[0055] Based on the above definitions, the implementation environment of a visual perception-based maritime distress assistance search and rescue system 10 provided in this application embodiment will be described. Indicatively, this implementation environment includes: a terminal, sensing devices, a processor, and a memory. The terminal is connected to the processor, sensing devices, and memory via a network. Sensing devices include, but are not limited to, visible light / infrared dual-spectrum cameras mounted on drones or ships, ocean current meters, anemometers, sea surface temperature probes, high-precision GPS / BeiDou positioning modules, or ship inertial navigation units (IMUs). The processor can be a central processing unit, a graphics processing unit, or an artificial intelligence chip. The memory can be a distributed cloud storage system or a local server cluster, and is not limited here.

[0056] Based on the above definitions and implementation environment, the application scenarios of the embodiments of this application will be described. The vision-based maritime distress assistance search and rescue system 10 provided in this application embodiment can be applied to scenarios including but not limited to the following:

[0057] In ocean search and rescue missions, the location of distressed personnel continuously drifts with ocean currents and wind direction, making it difficult for traditional static prediction models to track their dynamic trajectories. This system can construct a spatiotemporally unified fusion data cube by integrating satellite remote sensing imagery, real-time data from meteorological buoys, and UAV patrol video streams, accurately capturing the coupling relationship between environmental disturbances and target displacement. Based on the drift feature vector sequence, a probability cloud map of the distressed personnel's location can be dynamically updated to search the core area. Simultaneously, a dynamic risk map quantifies the distribution of floating objects on the sea surface and the risk of sudden ocean current changes, enabling rescue vessels to plan optimal routes that balance range efficiency and obstacle avoidance safety, thereby improving the search and rescue response speed and target positioning accuracy over large sea areas.

[0058] Nearshore waters are riddled with obstacles such as reefs, fishing nets, and dock facilities, and tidal changes cause the risk field to evolve in real time. This system uses shore-based cameras and UAV infrared video streams to detect distressed targets and combines them with real-time tidal data to construct a high-resolution dynamic risk map, accurately marking dangerous areas such as reef areas and turbulent current areas. In multi-objective path planning, the system simultaneously optimizes path length and cumulative risk values. The generated rescue path can automatically avoid newly emerging shallow areas with the rise and fall of tides, while bypassing sudden obstacles such as moving fishing boats, effectively solving the challenge of balancing the safety and timeliness of rescue routes in complex nearshore terrain.

[0059] In conditions of limited visibility, such as at night or in fog, this system enhances the ability to identify distressed targets by fusing infrared thermal imaging video streams with radar monitoring data. The drift feature extraction module combines motion trajectory analysis in low-light environments with ocean current vector correction to accurately reconstruct the target drift model. The dynamic risk map integrates the distribution of reefs detected by sonar with heat source signals from ships at night, updating the channel risk level in real time. The probability cloud map generated by the auxiliary rescue module can mark the core area of ​​survival probability for those in distress in low-temperature waters, and the planned route simultaneously provides night navigation lighting requirements and collision avoidance warnings, ensuring safe and efficient search and rescue operations in low-visibility environments.

[0060] This is illustrative; the visual perception-based maritime distress assistance system 10 provided in this application embodiment can also be applied to other application scenarios. This is only an example and does not limit the specific application scenarios.

[0061] In one exemplary embodiment, such as Figure 1 As shown, a vision-based maritime distress assistance system 10 is provided. This embodiment illustrates the system's application to a terminal in the aforementioned implementation environment. It is understood that the system can also be applied to a server and implemented through interaction between the terminal and the server. In this embodiment, the system includes a data acquisition module 11, a distress data extraction module 12, and an assistance rescue module 13.

[0062] The data acquisition module 11 is used to: acquire multi-source heterogeneous data, perform spatiotemporal synchronous fusion of multi-source heterogeneous data, and generate a fused data cube, wherein the multi-source heterogeneous data includes visual data and spatial data.

[0063] Specifically, visual data can be collected through high-definition cameras installed on rescue vessels, drones, or fixed monitoring points. The captured images and videos can reflect the visual characteristics of people in distress at sea and their surrounding environment in real time. Spatial data can include location information provided by satellite positioning systems, wind speed and direction data collected by marine meteorological sensors, and water flow speed and direction measured by marine hydrological sensors. To effectively combine these data from different sources, formats, and timestamps, spatiotemporal synchronous fusion processing is performed on the multi-source heterogeneous data. Specifically, through timestamp alignment and spatial coordinate transformation algorithms, visual data and spatial data are integrated within a unified spatiotemporal framework to generate a fused data cube. This fused data cube provides a comprehensive and consistent data foundation for subsequent distress data extraction and assisted rescue, and can more accurately reflect the dynamic changes in maritime distress scenarios.

[0064] The distress data extraction module 12 is used to: extract and process the drift features of distressed targets based on the fused data cube, and generate a drift feature vector sequence;

[0065] Environmental risk field modeling is performed based on fused data cubes to generate dynamic risk maps.

[0066] Specifically, by analyzing visual data from the fused data cube, target detection algorithms are used to identify distressed individuals or objects and track their positional changes over a continuous time series. Combining this with spatial data on water flow and wind direction, the drift speed and direction of the distressed targets are calculated, generating a drift feature vector sequence to accurately reflect their movement trends in a dynamic environment. Simultaneously, this module also performs environmental risk field modeling based on the fused data cube, generating a dynamic risk map. For example, by analyzing data such as water flow speed, wind speed, and obstacle distribution, a risk assessment model is used to calculate the risk levels of different areas, which are then presented as a dynamic risk map. This dynamic risk map reflects the risk distribution of the aquatic environment in real time, providing crucial information for rescue route planning.

[0067] The auxiliary rescue module 13 is used to: perform location prediction processing based on the drift feature vector sequence to generate a probability cloud map of the location of the distressed person;

[0068] Based on the probability cloud map of the location of the distressed persons, the dynamic risk map, and the current location of the rescue vessel, multi-objective path planning is performed to generate a rescue path, which is used to guide rescuers to conduct water search and rescue.

[0069] Specifically, by utilizing the velocity and direction information in the drift feature vector sequence, combined with statistical methods or machine learning algorithms, the module predicts the possible location distribution of distressed personnel over a future period and displays it in the form of a probability cloud map. This probability cloud map visually shows the areas where distressed personnel may appear and their probabilities. Further, this module performs multi-objective path planning based on the distressed personnel location probability cloud map, dynamic risk map, and the current position of the rescue vessel to generate a rescue path. The path planning algorithm comprehensively considers multiple objectives such as the shortest time, lowest risk, and highest success rate of the rescue path. Through optimization algorithms, it weighs these objectives to generate the optimal rescue path. For example, the path planning algorithm can prioritize lower-risk paths while minimizing path length, ensuring the efficiency and safety of the rescue operation.

[0070] The aforementioned visual perception-based maritime distress search and rescue system 10 and method acquires multi-source heterogeneous data, including visual and spatial data, through a data acquisition module 11. This data is then spatiotemporally fused to generate a fused data cube. A distress data extraction module 12, based on the fused data cube, generates a drift feature vector sequence through distress target drift feature extraction, capturing the dynamic motion characteristics of the distressed target and providing dynamic basis for target location prediction. An environmental risk field modeling process generates a dynamic risk map, quantifying the aquatic environmental risk in real time, thus solving the shortcomings of traditional systems such as delayed location prediction and lack of risk perception. An auxiliary rescue module 13 generates a distressed personnel location probability cloud map based on the drift feature vector sequence, ensuring the accuracy of distress target location. Combining this location probability cloud map, the dynamic risk map, and the current location of the rescue vessel, multi-objective path planning is performed to generate a rescue path. This not only improves the success rate of search and rescue operations through precise positioning but also achieves a balance between the timeliness and safety of the rescue path through multi-objective planning, providing optimized path guidance for rescue decisions and improving the search and rescue success rate.

[0071] In one embodiment, the data acquisition module 11 includes a multi-source data acquisition unit, a spatiotemporal synchronization processing unit, and a data fusion unit:

[0072] A multi-source data acquisition unit is used to acquire visible light video streams and infrared video streams as visual data;

[0073] Real-time ocean current vectors, wind direction and speed data, and sea surface temperature data are acquired and georeferenced to serve as spatial data.

[0074] Specifically, the core function of the multi-source data acquisition unit is to acquire visual and spatial data, ensuring the initial validity and correlation of both types of data. For visual data acquisition, this unit can use a dual-mode camera device mounted on a drone platform. This device integrates a high-performance visible light imaging sensor and an infrared thermal imaging sensor. The visible light imaging sensor can capture visible light information of the target water area through a CMOS image sensing chip, or be equipped with a low-light sensor to clearly present the outline, color, and other detailed features of objects in the water scene, especially accurately identifying the appearance features of distressed persons such as clothing color and limb shape. The infrared thermal imaging sensor generates thermal images by detecting the thermal radiation intensity of the object surface. Even in low-visibility environments such as night, fog, and rain, it can accurately capture the location information of distressed persons by measuring the temperature difference between the distressed person and the surrounding water, effectively avoiding the detection blind spots of a single imaging method in complex environments. For spatial data acquisition, this unit collaborates with a distributed weather buoy and a satellite remote sensing system. The weather buoys are equipped with ocean current sensors and wind speed and direction sensors. The ocean current sensors generate real-time ocean current vector data by measuring the velocity and direction of water flow, while the wind speed and direction sensors acquire real-time wind direction and speed data by sensing airflow movement. The satellite remote sensing system, through its onboard thermal infrared radiometer, receives thermal radiation signals emitted from the sea surface and converts them into sea surface temperature data. To ensure the correlation between spatial data and geographical location, the multi-source data acquisition unit performs georegistration processing on the acquired real-time ocean current vector, wind direction and speed, and sea surface temperature data. By reading the geographic coordinate information output by the positioning modules of the weather buoys and the satellite remote sensing system, various types of spatial data are bound to their corresponding geographic coordinates, ensuring that each piece of spatial data corresponds to a specific location in the target water area, ultimately forming the spatial data required by the system.

[0075] The spatiotemporal synchronization processing unit is used to acquire timestamps of visual data and spatial data. It uses a spatial registration algorithm to map the pixel coordinates of the visible light video stream and infrared video stream in the visual data to the geographic coordinate system of the spatial data, thereby obtaining spatiotemporally aligned multimodal data.

[0076] Specifically, the unit acquires the raw timestamps for visual and spatial data. The timestamp for visual data is generated by the built-in clock of the UAV's dual-mode camera device, while the timestamp for spatial data is generated separately by the clock module of the weather buoy and the timing module of the satellite remote sensing system. Since clocks from different devices may drift, directly using the raw timestamps would lead to data time synchronization issues. Therefore, this unit can also introduce a high-precision GPS timing module. This module receives time signals transmitted by GPS satellites and generates standard UTC time, which is used as a benchmark to calibrate the raw timestamps for visual and spatial data. Specifically, the spatiotemporal synchronization processing unit receives the standard time output by the GPS timing module at fixed intervals, compares this standard time with the raw timestamps of each data point, calculates the time deviation, and then corrects the raw timestamps based on the deviation. A standardized timestamp in a uniform format is added to each frame of video data and each parameter record of spatial data, ensuring consistency between the two types of data in the time dimension. In the spatial synchronization phase, the core task of this unit is to map the pixel coordinates of the visual data to the geographic coordinate system (such as the CGCS2000 coordinate system) of the spatial data, obtain the real-time flight parameters of the UAV (including flight altitude, heading angle, pitch angle, and roll angle) and the intrinsic parameters of the dual-mode camera device (including focal length, principal point coordinates, and distortion coefficients), and construct the imaging model of the camera device based on these parameters. For each frame of the visible light video stream and infrared video stream in the visual data, feature points in the image (such as obvious water boundary points and floating object vertices) are selected, and the coordinates of the feature points in the geographic coordinate system are calculated through the imaging model. A spatial registration algorithm (such as a feature point-based registration algorithm) is used to establish the mapping relationship between the image pixel coordinates and the geographic coordinates. Through this mapping relationship, the coordinates of all pixels in each frame of video are converted into the corresponding geographic coordinates, thereby obtaining spatiotemporally aligned multimodal data. This data contains both visual information and is associated with the corresponding geographic coordinates and standard time, laying the foundation for subsequent data fusion.

[0077] The data fusion unit is used to integrate and correlate spatiotemporally aligned multimodal data in terms of dimensions to generate a fused data cube, which includes time dimension, spatial dimension and data dimension.

[0078] Specifically, this unit can also perform data preprocessing on spatiotemporally aligned multimodal data. For visual data, image enhancement algorithms (such as histogram equalization) are used to adjust the contrast of video frames, eliminating the impact of lighting changes on image quality. Simultaneously, noise reduction algorithms (such as Gaussian filtering) are used to remove noise interference from the images. For spatial data, outlier detection algorithms are used to identify and remove abnormal data caused by sensor malfunctions. Then, interpolation algorithms are used to supplement missing values ​​that occurred during data acquisition, ensuring data integrity and accuracy. After data preprocessing, this unit integrates the data according to three dimensions: time, space, and data. The time dimension uses a unified standard timestamp as an index to arrange data from different time points in chronological order. The spatial dimension uses geographic coordinates as an index to associate data from different geographic locations at the same time point. The data dimension categorizes and stores data according to data type (including visible light video frames, infrared video frames, ocean current vectors, wind direction and speed, and sea surface temperature). For example, each data unit in the fused data cube is uniquely identified by an index based on three dimensions: timestamp, geographic coordinates, and data type. This structure allows for rapid querying of various types of data at any time and location. For instance, by specifying a timestamp and geographic coordinates, visible light images, infrared images, and corresponding environmental parameter data for that spatiotemporal node can be obtained simultaneously. Furthermore, the data fusion unit establishes semantic relationships between different types of data through data association algorithms. For example, it associates floating objects identified in a video frame with corresponding ocean current vector data, providing data support for subsequent analysis of the floating object's movement trends. The generated fused data cube achieves the organic integration of multi-source data, transforming various data types from independent information fragments into an interconnected whole, providing a unified data foundation for efficient processing by subsequent modules.

[0079] In one embodiment, the distress data extraction module 12 includes a drift feature extraction unit and a risk field modeling unit:

[0080] The drift feature extraction unit is used to: perform distress target detection on visible light video streams and infrared video streams using a convolutional neural network based on an attention mechanism, and obtain distress targets;

[0081] By combining the appearance features and motion consistency of the distressed target with a spatiotemporal correlation algorithm, the target's motion trajectory is generated, and the displacement vector of the distressed target between consecutive frames is calculated to obtain a drift feature vector sequence.

[0082] Specifically, this unit utilizes a convolutional neural network (CNN) based on an attention mechanism to detect distressed targets in visible light and infrared video streams. The attention mechanism guides the neural network to focus on regions in the image relevant to the distressed target, thereby improving detection accuracy and efficiency. Through a pre-trained CNN model, the system can identify the appearance features of distressed individuals or objects, such as human shape and life jacket color, and separate them from complex backgrounds. When a distressed target is detected, the drift feature extraction unit further generates the target's trajectory by combining its appearance features and motion consistency using a spatiotemporal correlation algorithm. This algorithm analyzes the positional changes of the distressed target in consecutive frames, combining the similarity of its appearance features to ensure the continuity and accuracy of the trajectory, and calculates the displacement vector of the distressed target between consecutive frames, obtaining a drift feature vector sequence. The drift feature vector sequence not only contains information about the distressed target's direction and velocity but also reflects its drift trend in a dynamic environment. For example, when a distressed target is affected by ocean currents and wind, the drift feature vector sequence can clearly show the changes in its trajectory.

[0083] The risk field modeling unit is used to perform semantic segmentation on visual data to identify dynamic obstacles and static danger zones. It combines real-time ocean current vectors to predict the movement trend of dynamic obstacles and construct a dynamic risk map, which includes a basic risk field and dynamic risk increments.

[0084] Specifically, this unit identifies dynamic obstacles and static hazard areas by performing semantic segmentation on visual data, and constructs a dynamic risk map. For example, this unit can use semantic segmentation algorithms to process visible light and infrared video streams, classifying and labeling different objects and regions in the images. For instance, through semantic segmentation, the system can identify floating obstacles (such as logs, plastic buckets, etc.), reefs, and other static hazard areas, as well as other objects or areas that may pose a threat to rescue operations. After identifying dynamic obstacles, the risk field modeling unit combines real-time ocean current vectors to predict their movement trends. Ocean current vectors provide information on the direction and speed of water flow; by analyzing this data, the system can predict the positional changes of dynamic obstacles over a future period. Based on this information, the risk field modeling unit constructs a dynamic risk map, which includes a base risk value and dynamic risk increments. The base risk value reflects the inherent risk of static hazard areas, such as the distribution area of ​​reefs; while the dynamic risk increment represents the additional risk caused by the movement of dynamic obstacles. The dynamic risk map presents the risk distribution of the aquatic environment in a visual manner, providing important reference for rescue route planning.

[0085] In one embodiment, the drift feature extraction unit includes a target detection subunit, a trajectory generation subunit, and a displacement calculation subunit:

[0086] The target detection subunit is used to fuse features from visible light video streams and infrared video streams, identify the contours of distressed targets through an attention-based convolutional neural network, and output the target bounding box and the corresponding confidence score.

[0087] Specifically, a convolutional neural network with feature fusion and attention mechanisms is used to accurately identify distressed targets, outputting target bounding boxes and confidence scores containing target location and reliability information. In the feature fusion stage, this sub-unit first extracts features from each frame of the two types of video streams. For visible light images, multi-scale convolutional kernels are used for feature extraction. Small-scale kernels capture detailed features such as target edges and textures, while large-scale kernels capture the overall contour features of the target. For infrared images, since they focus more on the temperature difference between the target and the background, an adaptive threshold segmentation algorithm is used to initially separate the target region from the background region, and then the thermal radiation distribution features of the target are extracted through convolution operations. Furthermore, the two types of features are synergistically integrated. For example, a weighted attention fusion strategy is adopted, and the weight coefficients of visible light features and infrared features in different scenarios are learned during the training process. In well-lit scenarios, the weight of visible light features is increased to enhance detailed information; in low-visibility scenarios, the weight of infrared features is increased to highlight the target contour, generating fused features that balance detail and robustness. In the target recognition stage, a pre-trained attention-based convolutional neural network is input with fused features. This network adds a spatial attention module and a channel attention module after the traditional convolutional layers. The spatial attention module generates spatial attention weights by performing global pooling and convolution operations on the fused feature map, automatically focusing on areas where potentially distressed targets may exist. The channel attention module performs statistical analysis on feature channels, strengthening the weights of channels related to human features (such as limb proportions and contour shapes) and suppressing interference from irrelevant background channels. The network determines whether a feature region is a distressed target through a classification branch and predicts the bounding box coordinates of the target through a regression branch, while simultaneously outputting the confidence score corresponding to the bounding box. The confidence score reflects the reliability of the recognition result. When the confidence score is higher than a preset threshold, the region corresponding to the bounding box is determined to be a distressed target, completing the target recognition and outputting the target bounding box and confidence score.

[0088] The trajectory generation subunit is used to generate target motion trajectories with consecutive timestamps by associating the IDs of the same distressed target with the target bounding boxes of adjacent frames of visible light video stream and infrared video stream through a motion consistency constraint algorithm.

[0089] Specifically, before ID association, this sub-unit preprocesses the target bounding boxes of adjacent frames, removing bounding boxes with confidence scores below a threshold to avoid false detections interfering with tracking. Simultaneously, it calculates the center coordinates, aspect ratio, and other geometric parameters of each bounding box to provide foundational data for subsequent association. During the execution of the motion consistency constraint algorithm, this sub-unit constructs association criteria from three dimensions: positional continuity, velocity consistency, and appearance consistency. Regarding positional continuity, based on the center coordinates of the target bounding box in the previous frame and the video frame rate, it calculates the expected position range of the target in the current frame, including only bounding boxes within this range in the current frame in the association candidate set. Regarding velocity consistency, based on the motion velocity of the target in previous frames, it predicts the velocity range of the target in the current frame, calculates the velocity difference between the candidate bounding boxes and the target in the previous frame, and removes candidate boxes exceeding the velocity range. Regarding appearance consistency, it extracts the target appearance features (such as color histograms and HOG features) corresponding to the candidate bounding boxes, calculates the similarity with the appearance features of the target in the previous frame, and retains candidate boxes with similarity scores above a threshold. Based on the constraints of the above three dimensions, an association cost matrix is ​​constructed. The matrix elements represent the association cost between the target in the previous frame and the candidate target in the current frame. The minimum cost solution of the cost matrix can be obtained by using the Hungarian algorithm. A unique tracking ID is assigned to the same distressed target to ensure that the ID of the same target remains consistent in consecutive frames. The center coordinates of the target bounding box corresponding to each ID are arranged in the order of timestamps. Combined with the time information of the video frames, a target motion trajectory containing timestamps, target IDs and bounding box coordinates is generated to achieve continuous tracking of distressed targets.

[0090] The displacement calculation subunit is used to calculate the displacement vector in the image coordinate system based on the target's motion trajectory. Combined with the real-time ocean current vector in the fused data cube, it generates a sequence of drift feature vectors in the geographic coordinate system through affine transformation.

[0091] Specifically, this subunit first extracts the bounding box center coordinates of targets with the same ID at adjacent timestamps from the target motion trajectory output by the trajectory generation subunit. By calculating the coordinate difference, it obtains the target's displacement vector in the image coordinate system, which reflects the target's direction and distance of movement within the image plane. In the geographic coordinate system transformation stage, since the image coordinate system displacement vector cannot directly correspond to the actual geographical location changes of the water area, this subunit introduces real-time ocean current vector data from the fusion data cube and parameter data (including focal length, flight altitude, and attitude angle) from the UAV camera device to construct a transformation model. For example, a mapping relationship between the image coordinate system and the world coordinate system can be established based on the camera device parameters. Through the camera's intrinsic and extrinsic parameter matrices, the displacement vector in the image coordinate system is converted into a preliminary displacement in the world coordinate system. The preliminary displacement is then corrected using real-time ocean current vector data, as the drift of water targets is significantly affected by ocean currents. This paper analyzes the correlation between ocean current vectors and target drift direction, establishes an ocean current influence coefficient model, takes the direction and velocity of real-time ocean current vectors as input, calculates the contribution of ocean currents to target drift, and compensates for the initial displacement in the world coordinate system. Through affine transformation algorithm, the adjusted world coordinate system displacement is converted into a displacement vector in the geographic coordinate system. This vector directly corresponds to the target's drift in the actual water area. The geographic coordinate system displacement vectors under adjacent timestamps are arranged in chronological order to generate a drift feature vector sequence. This sequence provides accurate dynamic drift data support for the subsequent location prediction of the auxiliary rescue module 13.

[0092] In one embodiment, the risk field modeling unit includes an obstacle recognition subunit, a risk prediction subunit, and a rasterization processing subunit:

[0093] The obstacle recognition subunit is used to perform pixel-level classification of video frames using a semantic segmentation model to obtain dynamic obstacles and static hazard areas. The attributes of dynamic obstacles include category and boundary polygon.

[0094] Specifically, this sub-unit can adopt a deep learning model based on an encoder-decoder architecture. The encoder consists of multiple sets of convolutional layers, batch normalization layers, and activation functions. It extracts high-level semantic features of video frames through progressive downsampling operations. Shallow convolutional layers capture detailed information such as image edges and textures, while deep convolutional layers integrate detailed features to form global features related to risk sources. At the same time, residual connection structures are introduced to solve the gradient vanishing problem in deep network training, ensuring the effectiveness of feature extraction. The decoder gradually restores the feature map resolution through upsampling layers, and simultaneously fuses features from different levels of the encoder with features from corresponding levels of the decoder through skip connections to supplement high-resolution detailed information, so that the final output segmentation result can accurately correspond to each pixel of the video frame. During the model training phase, a labeled dataset containing various aquatic scenes (such as open water, nearshore areas, and complex island and reef areas) and various risk sources (such as floating objects, reefs, shoals, and out-of-control vessels) is constructed. For example, each pixel is labeled with three categories: "dynamic obstacle," "static danger zone," and "normal water area." The model parameters are jointly optimized using the cross-entropy loss function and the Dice loss function to improve the model's recognition accuracy for small target risk sources (such as small floating objects). During the classification process, video frames are input into a pre-trained semantic segmentation model, and the model outputs pixel classification results with the same size as the video frames. Based on the classification results, pixel sets of dynamic obstacles and static danger zones are extracted. The boundary polygons of the two types of risk sources are obtained through contour extraction algorithms. At the same time, the category of dynamic obstacles (such as wooden floating objects, metal ship wrecks, and aquatic plant clusters) is determined based on pixel features (such as grayscale values ​​and texture features). The resulting recognition results include dynamic obstacles (including categories and boundary polygons) and static danger zones.

[0095] The risk prediction subunit is used to calculate the displacement vector of dynamic obstacles based on the real-time ocean current vector in the fused data cube, obtain the movement trend through a simplified model using the Navier-Stokes equations, and generate dynamic risk increments by combining the category of dynamic obstacles.

[0096] For example, in the dynamic obstacle movement trend calculation stage, this subunit first extracts real-time ocean current vector data of the target water area from the fused data cube, including the direction and velocity of the ocean current. Simultaneously, it acquires the physical property parameters of the dynamic obstacle (such as volume, density, and windward area, obtained by matching against a preset parameter library according to the obstacle category). Considering the high computational complexity of the complete Navier-Stokes equations, which is difficult to meet real-time requirements, this subunit adopts a simplified model for floating objects, ignoring the complex interaction terms of viscous force and pressure gradient, retaining only the core terms of inertial force and ocean current driving force, and establishing the motion equation of the dynamic obstacle. This equation is then used to calculate the displacement vector of the dynamic obstacle within a single time step.

[0097] Specifically, the displacement vector of the dynamic obstacle is calculated using the following formula, based on the driving force provided by the real-time ocean current vector, the physical property parameters of the dynamic obstacle, and its current motion state. This yields the displacement vector of the dynamic obstacle within a single time step, which characterizes the movement trend of the dynamic obstacle:

[0098]

[0099] in, It is the displacement vector (unit: m) of a dynamic obstacle within a single time step, which directly reflects the direction and distance of movement of the obstacle within that time step and is the core parameter for judging the movement trend; Seawater density (unit: kg / m³) is obtained by querying the sea surface temperature data provided by satellite in the fusion data cube and combining it with the preset seawater density-temperature correlation table. It is a basic environmental parameter for calculating the driving force of ocean currents. The drag coefficient (dimensionless) is obtained from a preset parameter library based on the type of dynamic obstacle (such as wooden floating objects, metal ship wrecks). Different types of obstacles have different surface morphology and force characteristics, corresponding to different drag coefficients. The projected area of ​​a dynamic obstacle in the direction perpendicular to the ocean current (unit: m²) is obtained by matching from a preset parameter library according to the obstacle type. It reflects the contact area between the obstacle and the ocean current and directly affects the magnitude of the driving force. The real-time ocean current vector (unit: m / s) is extracted from the data transmitted by meteorological buoys in the fused data cube. It includes the direction of the ocean current (vector direction) and the velocity of the current (vector magnitude), and is the core external force source driving the movement of obstacles. The mass (unit: kg) of the dynamic obstacle is calculated based on the obstacle type and volume (obtained by matching from the preset parameter library) combined with the obstacle density (obtained by matching from the preset parameter library), reflecting the inertial characteristics of the obstacle; The time step (in seconds) used for calculation is kept consistent with the frame rate of the video stream to ensure that the displacement calculation is synchronized with the time dimension of the visual data; The initial velocity vector (unit: m / s) of the dynamic obstacle at the start of the current time step is calculated from the displacement vector of the previous time step and the time step, reflecting the continuity of the obstacle's motion.

[0100] The rasterization processing subunit is used to rasterize the acquired electronic nautical chart into a basic risk field and overlay dynamic risk increments to generate a dynamic risk map.

[0101] Specifically, in the basic risk field construction phase, this sub-unit acquires electronic nautical charts of the target waters. These charts contain basic information such as geographic coordinates, water depth, and seabed type. Through coordinate transformation, the coordinate system of the electronic charts is unified to a geographic coordinate system consistent with the fused data cube. Fixed-size grids are used to divide the waters covered by the electronic charts, forming a uniform grid matrix, with each grid corresponding to a fixed area of ​​the actual waters. Based on the water depth data from the electronic charts and the static hazard area information output by the obstacle identification sub-unit, a basic risk value is assigned to each grid: grids corresponding to static hazard areas such as shoals and reefs are assigned high basic risk values ​​based on the degree of hazard (e.g., the exposed height of reefs, the minimum water depth of shoals); grids with normal water depth and no static hazard are assigned low basic risk values; and transitional areas between these two (e.g., gentle slopes around reefs) are assigned medium basic risk values. This method constructs the basic risk field, reflecting the inherent static risk distribution of the waters. During the dynamic risk overlay phase, this sub-unit determines the grid range corresponding to each dynamic obstacle at different time steps based on the dynamic obstacle movement trend and dynamic risk increment output by the risk prediction sub-unit, and allocates the dynamic risk increment to these grids. For grids simultaneously affected by multiple dynamic obstacles, the total dynamic risk increment is calculated by overlay and summation, and then added to the grid's base risk value to obtain the real-time total risk value for each grid. Finally, color mapping technology is used to convert the grid's total risk value into intuitive color indicators (e.g., red for high risk, yellow for medium risk, and green for low risk), and combined with the grid's geographic coordinate information to generate a dynamic risk map. This map can be updated in real time according to the movement trend of dynamic obstacles, ensuring the timeliness and accuracy of risk distribution information.

[0102] In one embodiment, the auxiliary rescue module 13 includes a location prediction unit, a route planning unit, and a route output unit:

[0103] The location prediction unit is used to input the drift feature vector sequence into the physical information neural network, and predict the probability distribution of future location by embedding fluid dynamics constraint equations, and output the probability cloud map of the location of the distressed personnel.

[0104] Specifically, the drift feature vector sequence is preprocessed to extract key information such as timestamps and geographic coordinate displacements, constructing time-displacement sample pairs. Each sample contains displacement data for multiple consecutive historical time steps and their corresponding time intervals. The physical information neural network architecture consists of a feature extraction layer, a physical constraint layer, and a probability prediction layer. The feature extraction layer employs a bidirectional long short-term memory network, capturing long-term dependencies in the drift feature vector sequence through the interaction of forward and backward hidden states. For example, the cumulative impact of continuous changes in ocean current direction on target drift. The physical constraint layer embeds fluid dynamics constraint equations (such as simplified forms of the continuity and momentum equations) as regularization terms into the network loss function. This constrains the network output to meet physical laws, avoiding predictions that might not conform to actual drift patterns from a purely data-driven model. For instance, when the ocean current direction suddenly changes, the model must follow the principle of momentum conservation to predict a gradual change in the target drift direction rather than a sudden change. During model training, historical drift data and corresponding actual locations are used as training samples. The network parameters are optimized using a backpropagation algorithm to minimize the loss function (which includes prediction position error and physical constraint violation). During the prediction process, the drift feature vector sequence before the time to be predicted is input into the trained network. The network analyzes the historical drift pattern through the feature extraction layer, and after correction by the physical constraint layer, the probability prediction layer outputs the location probability distribution of multiple future time nodes. This distribution is represented by a Gaussian mixture model, with each mixture component corresponding to a possible location and its corresponding probability. Furthermore, after mapping the probability distribution to a geographic coordinate system, the probability values ​​of different locations can be visualized through color gradients to generate a probability cloud map of the location of the distressed person. The darker the color, the higher the probability of the distressed person being there, providing clear target area guidance for subsequent route planning.

[0105] The path planning unit is used to generate the Pareto optimal path set by executing a multi-objective heuristic search algorithm, with the current position of the rescue ship as the starting point, the core area of ​​the position probability cloud map as the endpoint set, and the dynamic risk map as the constraint condition.

[0106] Specifically, the start and end point sets are preprocessed by converting the rescue vessel's current location (e.g., geographic coordinates obtained via onboard GPS) into a starting grid in a dynamic risk map grid coordinate system. Boundary coordinates of the core area are extracted from the location probability cloud map and converted into multiple end point grids in the grid coordinate system, forming the end point set. The objective function of the multi-objective heuristic search algorithm includes three dimensions: shortest path length, lowest cumulative risk value, and shortest travel time. Path length is calculated by accumulating the Euclidean distances between grids, the cumulative risk value is obtained by integrating the dynamic risk values ​​of the grids traversed by the path, and the travel time is determined by combining the path length with the rescue vessel's speed under different ocean current conditions. In the initial stage of the algorithm, a batch of candidate paths from the start to the end point set is randomly generated, each path consisting of a series of continuous grid coordinates. Subsequently, a heuristic evaluation function is used to filter the candidate paths, retaining those with better objective function values ​​for the iterative process. During the iteration process, crossover and mutation operations are used to generate new paths. The crossover operation generates offspring paths by exchanging the intermediate grid sequences of two paths, while the mutation operation randomly adjusts some grids in the path to introduce diversity. Simultaneously, all paths are stratified based on non-dominated ranking rules, ensuring that paths at the same level are not mutually dominant (i.e., no one path is superior to another in all objective functions). Only the optimal paths in each level are retained to form a new candidate set. After multiple iterations, when the path set no longer significantly optimizes, the iteration stops and the Pareto optimal path set is output. This set contains multiple paths that perform well in different objective functions; for example, some paths are short but risky, while others are low-risk but have long travel times, providing diverse choices for subsequent evaluations.

[0107] The path output unit is used to evaluate the flight time and cumulative risk value of the Pareto optimal path set and generate a rescue path, which includes a waypoint sequence and a risk heat map.

[0108] Specifically, each path in the Pareto optimal path set is evaluated for both travel time and cumulative risk value. The travel time evaluation considers factors such as path length, rescue vessel speed, and potential ocean current influences. The cumulative risk value evaluation combines the risk level of each point on the path, calculating the total risk value of the path through integration or weighted summation. Based on the evaluation results, one or more paths are selected as the rescue path. The rescue path includes a waypoint sequence and a risk heatmap. The waypoint sequence consists of points that the rescue vessel must pass sequentially during the rescue mission, guiding it along the predetermined path. The risk heatmap visually displays the risk distribution along the rescue path, helping rescue personnel understand risk changes in real time. For example, darker areas in the risk heatmap represent higher risks, while lighter areas represent lower risks.

[0109] In one embodiment, the path planning unit includes a ship kinematics subunit, a cost calculation subunit, and an optimization search subunit:

[0110] The ship kinematics subunit is used to establish a ship kinematics model affected by ocean currents based on real-time ocean current vectors and wind direction and speed data, and to calculate the actual sailing speed of the rescue ship at different heading angles using the ship kinematics model.

[0111] Specifically, real-time environmental parameters of the target waters are extracted from the fused data cube. The real-time ocean current vector includes the direction and speed of the current, and the wind direction and speed data includes the direction and speed of the wind. Together, these constitute the main external forces affecting the navigation of the rescue vessel. When constructing the ship kinematics model, this sub-unit is based on the classic ship maneuvering equations and introduces environmental disturbance terms to quantify the impact of ocean currents and wind on navigation. The core variables of the model include the rescue vessel's heading angle, speed, and angular velocity. The heading angle is the angle between the rescue vessel's bow and geographic north, the speed is the speed of the rescue vessel relative to the water, and the angular velocity is the rate of change of the heading angle over time. During the model construction process, the inertial and water resistance characteristics of the rescue vessel can be characterized by introducing additional mass coefficients and damping coefficients. These two coefficients are obtained from a pre-set ship parameter library based on the design parameters of the rescue vessel (such as length, beam, and draft). At the same time, real-time ocean current vectors and wind direction and speed data are converted into external force vectors acting on the hull. Ocean currents affect the lateral and longitudinal motion of the hull through hydrodynamic action, while wind changes the direction of the bow through aerodynamic action. These two types of external force vectors serve as input disturbance terms for the model and participate in the calculation of the equation of motion together with the propulsion force of the rescue vessel (determined by the power of the ship's main engine). When calculating the actual speed at different heading angles, this sub-unit can set a series of continuous heading angle values, covering all possible navigation directions of the rescue vessel. For each heading angle, it is substituted into the ship's kinematics model, and combined with real-time environmental parameters, the theoretical speed of the rescue vessel relative to the water is calculated. Then, through vector synthesis, the theoretical speed is superimposed with the ocean current speed. Using the geographic coordinate system as a reference, the components of the theoretical speed and ocean current speed in the east-west and north-south directions are decomposed separately. The components in the same direction are algebraically summed to obtain the actual speed components of the rescue vessel relative to the ground. Finally, the magnitude and direction of the actual speed are obtained through vector synthesis. Through this process, the actual speed of the rescue vessel at any heading angle can be obtained, providing data support that conforms to the actual navigation scenario for subsequent cost calculations.

[0112] The cost calculation subunit is used to convert the dynamic risk map into a dynamic risk field grid map, calculate the cumulative risk value of the path on the dynamic risk field grid map, and calculate the path length and corresponding estimated travel time based on the actual sailing speed.

[0113] Specifically, the grid size is determined, balancing computational accuracy and efficiency. Too small a size increases computational load and reduces real-time performance, while too large a size reduces the accuracy of risk assessment. During conversion, the geographic coordinate range of the dynamic risk map is mapped to the row and column range of the grid matrix. Each grid corresponds to a fixed geographic area in the dynamic risk map. The total risk value (including the base risk value and dynamic risk increment) for each geographic area is then extracted and assigned to the corresponding grid, forming a dynamic risk field grid map. Each grid in this map contains unique geographic coordinates and a corresponding risk value, providing an intuitive quantitative carrier for subsequent path risk calculation. When calculating the cumulative risk value of a path, this sub-unit acquires all grids the path passes through in the grid map, extracts the risk value of each grid, and sums them to obtain the total risk value of all grids. The higher the cumulative risk value, the lower the path's safety. When calculating the path length and estimated travel time, this sub-unit first calculates the distance between the centers of adjacent grids based on the grid coordinates traversed by the path, and then sums up the distances of all adjacent grids to obtain the total path length. Based on the actual travel speeds at different heading angles output by the ship kinematics sub-unit, it determines the heading angles corresponding to each segment of the path (the line segments between adjacent grids), extracts the actual travel speed at that heading angle, and obtains the travel time for that segment by dividing the segment distance by the actual travel speed. Finally, it sums up the travel times of all segments to obtain the estimated travel time for the path. Through this process, a quantitative assessment of the path cost can be completed from three dimensions: risk, distance, and time.

[0114] The optimization search subunit is used to simultaneously optimize path length, cumulative risk value and estimated travel time using the multi-objective A* algorithm, and output the Pareto optimal path set.

[0115] Specifically, the multi-objective A* algorithm comprehensively evaluates paths by simultaneously considering three optimization objectives: path length, cumulative risk value, and estimated travel time. During the search process, the algorithm uses heuristic functions to guide the search direction, prioritizing paths with lower costs. For example, when two paths have similar path lengths and cumulative risk values, the algorithm will prioritize the path with the shorter estimated travel time, optimizing the search sub-unit's output to form a Pareto optimal path set. This set contains multiple paths that achieve different balances between path length, cumulative risk value, and estimated travel time. The Pareto optimal path set provides rescue personnel with a variety of path options, enabling them to choose the most suitable path based on the actual situation and rescue needs.

[0116] In summary, the visual perception-based maritime distress assistance search and rescue system 10 provided in this application acquires visible light / infrared video streams (visual data) and real-time ocean current vectors, wind direction, and wind speed through the multi-source data acquisition unit of the data acquisition module 11. The spatiotemporal synchronization processing unit performs data timestamp calibration and coordinate mapping, and the data fusion unit generates a fused data cube containing time, space, and data dimensions, laying a unified data foundation for subsequent processing. In the distress data extraction module 12, the drift feature extraction unit identifies distressed targets through the attention mechanism convolutional neural network of the target detection subunit, and generates the target's motion trajectory using the motion consistency constraint algorithm of the trajectory generation subunit. The trajectory is then calculated by the displacement calculation subunit. The unit converts the drift feature vector sequence into a geographic coordinate system. Simultaneously, the risk field modeling unit identifies dynamic obstacles and static danger zones through semantic segmentation, simplifies the model using Navier-Stokes equations to predict obstacle movement trends, and constructs a dynamic risk map. The location prediction unit of the auxiliary rescue module 13 inputs the drift feature vector sequence into a physical information neural network embedded with hydrodynamic constraints, outputting a probability cloud map of the distressed personnel's location. The path planning unit establishes a speed model influenced by ocean currents through a ship kinematics subunit. The cost calculation subunit quantifies the cumulative risk, length, and time of the path, and the multi-objective A* algorithm of the optimization search subunit generates a Pareto optimal path set. The path output unit then evaluates and selects the rescue path. This technical solution resolves the contradiction between the timeliness and safety of rescue paths in dynamic aquatic environments, improves the accuracy and reliability of search and rescue operations in complex water conditions, and provides rescue personnel with scientific and intuitive decision support.

[0117] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0118] Based on the same inventive concept, this application also provides a vision-based method for assisting in the search and rescue of people in distress at sea, which is used to implement the vision-based system 10 for assisting in the search and rescue of people in distress at sea as described above. The solution provided by this method is similar to the solution described in the above-described method. Therefore, the specific limitations in one or more vision-based methods for assisting in the search and rescue of people in distress at sea provided below can be found in the limitations of the vision-based system 10 for assisting in the search and rescue of people in distress at sea described above, and will not be repeated here.

[0119] In one exemplary embodiment, such as Figure 2 As shown, a visual perception-based method for assisting in the search and rescue of people in distress at sea is provided, including the following steps 101 to 103:

[0120] Step 101: Acquire multi-source heterogeneous data, perform spatiotemporal synchronous fusion of the multi-source heterogeneous data, and generate a fused data cube, wherein the multi-source heterogeneous data includes visual data and spatial data;

[0121] Step 102: Extract drift features of distressed targets based on the fused data cube to generate a drift feature vector sequence;

[0122] Environmental risk field modeling is performed based on fused data cubes to generate dynamic risk maps;

[0123] Step 103: Perform location prediction processing based on the drift feature vector sequence to generate a probability cloud map of the location of the distressed person;

[0124] Based on the probability cloud map of the location of the distressed persons, the dynamic risk map, and the current location of the rescue vessel, multi-objective path planning is performed to generate a rescue path, which is used to guide rescuers to conduct water search and rescue.

[0125] In one embodiment, multi-source heterogeneous data is acquired, and spatiotemporal synchronous fusion of the multi-source heterogeneous data is performed to generate a fused data cube, including:

[0126] Acquire visible light video streams and infrared video streams as visual data;

[0127] Real-time ocean current vectors, wind direction and speed data, and sea surface temperature data are acquired and georeferenced to serve as spatial data.

[0128] The timestamps of visual and spatial data are obtained, and the pixel coordinates of the visible light video stream and infrared video stream in the visual data are mapped to the geographic coordinate system of the spatial data through a spatial registration algorithm to obtain spatiotemporally aligned multimodal data.

[0129] The spatiotemporally aligned multimodal data is integrated and correlated dimensionally to generate a fused data cube, which includes time dimension, spatial dimension and data dimension.

[0130] In one embodiment, drift feature extraction of distressed targets is performed based on a fused data cube to generate a drift feature vector sequence; environmental risk field modeling is performed based on the fused data cube to generate a dynamic risk map, including:

[0131] A convolutional neural network based on an attention mechanism is used to detect distressed targets in visible light video streams and infrared video streams.

[0132] The target motion trajectory is generated by combining the appearance features and motion consistency of the distressed target with the spatiotemporal correlation algorithm, and the displacement vector of the distressed target between consecutive frames is calculated to obtain the drift feature vector sequence.

[0133] Visual data is semantically segmented to identify dynamic obstacles and static danger zones. The movement trend of dynamic obstacles is predicted by combining real-time ocean current vectors, and a dynamic risk map is constructed. The dynamic risk map includes a basic risk field and dynamic risk increments.

[0134] In one embodiment, a convolutional neural network based on an attention mechanism performs distress target detection on visible light video streams and infrared video streams to obtain distress targets; a spatiotemporal correlation algorithm is used to combine the appearance features and motion consistency of the distress targets to generate target motion trajectories, and the displacement vectors of the distress targets between consecutive frames are calculated to obtain a drift feature vector sequence including:

[0135] Feature fusion is performed on visible light video stream and infrared video stream. The contour of the distressed target is identified through a convolutional neural network with an attention mechanism, and the target bounding box and corresponding confidence score are output.

[0136] Based on the target bounding boxes of adjacent frames of visible light video stream and infrared video stream, the IDs of the same distressed target are associated through a motion consistency constraint algorithm to generate target motion trajectories with continuous timestamps.

[0137] The displacement vector in the image coordinate system is calculated based on the target's motion trajectory. Combined with the real-time ocean current vector in the fused data cube, a sequence of drift feature vectors in the geographic coordinate system is generated through affine transformation.

[0138] In one embodiment, visual data is semantically segmented to identify dynamic obstacles and static hazard areas. A dynamic risk map is constructed by combining real-time ocean current vectors to predict the movement trend of dynamic obstacles, including:

[0139] The video frames are classified at the pixel level using a semantic segmentation model to obtain dynamic obstacles and static danger zones. The attributes of dynamic obstacles include category and boundary polygon.

[0140] Based on the real-time ocean current vector in the fused data cube, the displacement vector of the dynamic obstacle is calculated by simplifying the model through the Navier-Stokes equations to obtain the movement trend, and the dynamic risk increment is generated by combining the category of the dynamic obstacle.

[0141] The acquired electronic nautical charts are rasterized into a basic risk field, and dynamic risk increments are overlaid to generate a dynamic risk map.

[0142] In one embodiment, position prediction processing is performed based on the drift feature vector sequence to generate a probability cloud map of the distressed personnel's location; multi-objective path planning processing is then performed based on the probability cloud map of the distressed personnel's location, the dynamic risk map, and the current position of the rescue vessel to generate a rescue path, including:

[0143] The drift feature vector sequence is input into the physical information neural network, and the probability distribution of future location is predicted by embedding fluid dynamics constraint equations, outputting a probability cloud map of the location of the distressed person.

[0144] Using the current location of the rescue ship as the starting point, the core area of ​​the location probability cloud map as the endpoint set, and the dynamic risk map as the constraint, a multi-objective heuristic search algorithm is executed to generate the Pareto optimal path set;

[0145] The Pareto optimal path set is evaluated for flight time and cumulative risk value to generate a rescue path, which includes a waypoint sequence and a risk heat map.

[0146] In one embodiment, using the current position of the rescue vessel as the starting point, the core area of ​​the position probability cloud map as the endpoint set, and the dynamic risk map as constraints, a multi-objective heuristic search algorithm is executed to generate the Pareto optimal path set, including:

[0147] A ship kinematics model affected by ocean currents is established based on real-time ocean current vector and wind direction and speed data, and the actual sailing speed of the rescue ship at different heading angles is calculated using the ship kinematics model.

[0148] The dynamic risk map is converted into a dynamic risk field raster map, and the cumulative risk value of the path is calculated on the dynamic risk field raster map; and the path length and the corresponding estimated travel time are calculated based on the actual sailing speed.

[0149] The Pareto optimal path set is output by simultaneously optimizing path length, cumulative risk value, and estimated travel time using the multi-objective A* algorithm.

[0150] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A visual perception-based auxiliary search and rescue system for people in distress at sea, characterized in that, The system includes a data acquisition module, a distress data extraction module, and an auxiliary rescue module. The data acquisition module is used for: Acquire multi-source heterogeneous data, perform spatiotemporal synchronous fusion of the multi-source heterogeneous data, and generate a fused data cube, wherein the multi-source heterogeneous data includes visual data and spatial data; The distress data extraction module is used for: Based on the fused data cube, drift feature extraction processing of distressed targets is performed to generate a drift feature vector sequence; Based on the fused data cube, environmental risk field modeling is performed to generate a dynamic risk map; The auxiliary rescue module is used for: Based on the drift feature vector sequence, a location prediction process is performed to generate a probability cloud map of the location of the distressed person. Based on the probability cloud map of the location of the distressed persons, the dynamic risk map, and the current location of the rescue vessel, multi-objective path planning is performed to generate a rescue path, which is used to guide rescue personnel to conduct water search and rescue.

2. The system according to claim 1, characterized in that, The data acquisition module includes a multi-source data acquisition unit, a spatiotemporal synchronization processing unit, and a data fusion unit. The multi-source data acquisition unit is used to acquire visible light video streams and infrared video streams as the visual data; Real-time ocean current vector, wind direction and speed data, and sea surface temperature data are acquired and georeferenced to serve as the spatial data. The spatiotemporal synchronization processing unit is used to obtain the timestamps of the visual data and the spatial data, and to map the pixel coordinates of the visible light video stream and the infrared video stream in the visual data to the geographic coordinate system of the spatial data through a spatial registration algorithm to obtain spatiotemporally aligned multimodal data. The data fusion unit is used to perform dimensional integration and correlation on the spatiotemporally aligned multimodal data to generate the fused data cube, wherein the fused data cube includes a time dimension, a spatial dimension, and a data dimension.

3. The system according to claim 2, characterized in that, The distress data extraction module includes a drift feature extraction unit and a risk field modeling unit: The drift feature extraction unit is used for: A convolutional neural network based on an attention mechanism is used to detect distressed targets in the visible light video stream and the infrared video stream to obtain distressed targets. The target motion trajectory is generated by combining the appearance features and motion consistency of the distressed target with the spatiotemporal correlation algorithm, and the displacement vector of the distressed target between consecutive frames is calculated to obtain the drift feature vector sequence. The risk field modeling unit is used to perform semantic segmentation on the visual data to identify dynamic obstacles and static danger zones, and to combine the real-time ocean current vector to predict the movement trend of the dynamic obstacles to construct the dynamic risk map, wherein the dynamic risk map includes a basic risk field and dynamic risk increments.

4. The system according to claim 3, characterized in that, The drift feature extraction unit includes a target detection subunit, a trajectory generation subunit, and a displacement calculation subunit: The target detection subunit is used to perform feature fusion on the visible light video stream and the infrared video stream, identify the outline of the distressed target through an attention mechanism convolutional neural network, and output the target bounding box and the corresponding confidence score. The trajectory generation subunit is used to generate a target motion trajectory with consecutive timestamps based on the target bounding boxes of adjacent frames of the visible light video stream and the infrared video stream, and to associate the ID of the same distressed target with the motion consistency constraint algorithm. The displacement calculation subunit is used to calculate the image coordinate system displacement vector based on the target motion trajectory, and combine it with the real-time ocean current vector in the fused data cube to generate the drift feature vector sequence in the geographic coordinate system through affine transformation.

5. The system according to claim 3, characterized in that, The system is characterized in that the risk field modeling unit includes an obstacle recognition subunit, a risk prediction subunit, and a rasterization processing subunit: The obstacle recognition subunit is used to perform pixel-level classification of video frames through a semantic segmentation model to obtain the dynamic obstacles and the static danger areas, wherein the attributes of the dynamic obstacles include category and boundary polygon; The risk prediction subunit is used to calculate the displacement vector of the dynamic obstacle based on the real-time ocean current vector in the fused data cube, and obtain the movement trend by using a simplified model of the Navier-Stokes equations to obtain the movement trend, and generate the dynamic risk increment by combining the category of the dynamic obstacle. The rasterization processing subunit is used to rasterize the acquired electronic nautical chart into the basic risk field, and superimpose the dynamic risk increment to generate the dynamic risk map.

6. The system according to claim 2, characterized in that, The auxiliary rescue module includes a location prediction unit, a route planning unit, and a route output unit: The location prediction unit is used to input the drift feature vector sequence into the physical information neural network, predict the future location probability distribution by embedding fluid dynamics constraint equations, and output the location probability cloud map of the distressed person. The path planning unit is used to generate a Pareto optimal path set by executing a multi-objective heuristic search algorithm, with the current position of the rescue ship as the starting point, the core area of ​​the position probability cloud map as the endpoint set, and the dynamic risk map as the constraint condition. The path output unit is used to evaluate the flight time and cumulative risk value of the Pareto optimal path set and generate the rescue path, wherein the rescue path includes a waypoint sequence and a risk heat map.

7. The system according to claim 6, characterized in that, The path planning unit includes a ship kinematics subunit, a cost calculation subunit, and an optimization search subunit: The ship kinematics subunit is used to establish a ship kinematics model affected by ocean currents based on the real-time ocean current vector and the wind direction and speed data, and to calculate the actual sailing speed of the rescue ship at different heading angles through the ship kinematics model; The cost calculation subunit is used to convert the dynamic risk map into a dynamic risk field grid map and calculate the cumulative risk value of the path on the dynamic risk field grid map. The path length and corresponding estimated travel time are calculated based on the actual sailing speed. The optimization search subunit is used to simultaneously optimize the path length, the cumulative risk value, and the estimated travel time using a multi-objective A* algorithm, and output the Pareto optimal path set.

8. A visual perception-based method for assisting in the search and rescue of people in distress at sea, characterized in that, The method includes: Acquire multi-source heterogeneous data, perform spatiotemporal synchronous fusion of the multi-source heterogeneous data, and generate a fused data cube, wherein the multi-source heterogeneous data includes visual data and spatial data; Based on the fused data cube, drift feature extraction processing of distressed targets is performed to generate a drift feature vector sequence; Based on the fused data cube, environmental risk field modeling is performed to generate a dynamic risk map; Based on the drift feature vector sequence, a location prediction process is performed to generate a probability cloud map of the location of the distressed person. Based on the probability cloud map of the location of the distressed persons, the dynamic risk map, and the current location of the rescue vessel, multi-objective path planning is performed to generate a rescue path, which is used to guide rescue personnel to conduct water search and rescue.

Images