Unmanned aerial vehicle closed-loop intelligent cooperative search and rescue method and device

Abstract

In order to solve the problems of information fragmentation, low efficiency, and poor adaptability of shallow analysis of the traditional segmented search and rescue process, the present application provides a kind of unmanned aerial vehicle closed loop intelligent cooperative search and rescue method and device, the method realizes efficient data acquisition through adaptive multispectral acquisition driven by task, uses lightweight AI to analyze high-altitude multispectral data in real time, autonomously discovers potential targets and triggers close investigation;Based on the spectral-space feature deep fusion network, the multi-source features are fused through the multi-level cross-channel attention fusion module, and the weak targets that are difficult to detect by a single sensor are accurately identified;Combined with multi-dimensional dynamic risk perception and three-dimensional terrain modeling, the environmental risk is quantified in real time and superimposed on the digital surface model, and a multi-mode rescue path is generated that takes into account safety and efficiency.The present application designs an unmanned aerial vehicle intelligent search and rescue mode based on the "perception-decision-action-learning" closed loop, which significantly improves the target recognition accuracy, response speed and dynamic risk response capability in complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aircraft control, in particular to a method and device for closed-loop intelligent cooperative search and rescue of unmanned aerial vehicles. BACKGROUND

[0002] With the rapid development and popularization of unmanned aerial vehicle technology, its application in emergency search and rescue tasks such as natural disasters and field disappearances is increasingly widespread. Unmanned aerial vehicles have become an indispensable air force in modern emergency rescue systems due to their advantages such as mobility, quick response, and ability to penetrate dangerous areas that humans cannot reach.

[0003] The current mainstream unmanned aerial vehicle search and rescue mode still has significant technical bottlenecks. First, in terms of data acquisition, a blanket extensive scanning mode is commonly used, which is inefficient. For example, Chinese patent application CN110109480A (publication date: August 9, 2019) discloses a multi-sensor fusion-based unmanned aerial vehicle patrol search and rescue system and method, which includes an unmanned aerial vehicle subsystem, a bracelet subsystem, and a rescue command center. The unmanned aerial vehicle subsystem has functions such as image acquisition, signal searching, and flight control. The bracelet subsystem is used to send distress signals and location information. The rescue command center is responsible for receiving information and dispatching unmanned aerial vehicles for rescue. The method realizes autonomous search and rescue operations through the coordinated work of the unmanned aerial vehicle, bracelet, and rescue command center. However, the above technical solution relies on a low-efficiency blanket search along a preset path, generating a large amount of redundant data (visible light / thermal imaging video streams) in a wide area, causing key clues to be submerged, and the image processing unit only analyzes offline high-definition camera data, which cannot dynamically focus on suspicious areas, resulting in resource waste and time delay.

[0004] Secondly, in terms of information analysis, most rely on the perception data of a single sensor (such as visible light or thermal imaging), or only perform simple post-processing and superposition on multi-source data, which has serious shortcomings in identifying weak targets that are partially obscured, camouflaged, or have weak vital signs in unstructured environments. For example, Chinese patent application CN116625376A (publication date: August 22, 2023) discloses a method for planning a search and rescue path for an unmanned aerial vehicle at sea based on a tabu bee colony algorithm, which optimizes and plans the search and rescue path for the unmanned aerial vehicle by constructing a search and rescue target model and a constraint description model, and combining tabu search and bee colony algorithm. Although multiple sensors (GPS, laser radar, ultrasonic wave) are used, the obstacle avoidance unit establishes an environmental model through phase / distance information fusion, which is only a shallow multi-source data fusion. The simple superposition of data lacks deep feature interaction, especially in vegetation-obstructed (such as jungle search) or adverse weather conditions, the identification of weak vital signs and partially obscured targets has poor robustness and high miss rate.

[0005] Again, at the decision support level, the existing path planning is mostly based on the shortest path calculation of static two-dimensional map (such as CN116625376A adopts tabu bee colony algorithm), which seriously ignores the real three-dimensional terrain, dynamic risks such as landslides and dark fires, and the accessibility of rescue resources such as helicopter landing points, making the planning scheme seriously out of touch with the actual demand.

[0006] There are common problems such as fragmentation of each link (insufficient cooperation), one-way information flow, and low degree of intelligence in the prior art, which leads to the fact that the unmanned aerial vehicle still mainly plays the role of a flying camera in search and rescue applications, and its potential as an intelligent decision platform has not been explored. Under this background, it is urgent to develop an integrated decision-making method and support system that can realize intelligent perception, dynamic decision-making, and closed-loop control, and to upgrade the search and rescue unmanned aerial vehicle from a tool to an intelligent partner, and to expand the application prospect of unmanned aerial vehicle search and rescue technology. SUMMARY

[0007] Therefore, in order to solve the problems of information fragmentation, low efficiency, and poor adaptability of shallow analysis in the traditional segmented search and rescue process in the prior art, the present application provides a closed-loop intelligent cooperative search and rescue method and device for unmanned aerial vehicles, which realizes efficient data acquisition through task-driven adaptive multi-spectral acquisition, and designs an unmanned aerial vehicle intelligent search and rescue mode based on a "perception-decision-action-learning" closed loop, significantly improving the target recognition accuracy, response speed, and dynamic risk response capability in complex environments.

[0008] The present application provides a closed-loop intelligent cooperative search and rescue method for unmanned aerial vehicles, which is based on a search and rescue unmanned aerial vehicle equipped with a lightweight integrated multi-spectral imager, an edge computing unit, and an autonomous flight control system, and realizes intelligent cooperative search and rescue; the lightweight integrated multi-spectral imager is provided with a synchronous trigger controller for ensuring accurate alignment of data frames when different key spectral channel data are synchronously triggered;

[0009] The closed-loop intelligent cooperative search and rescue method for unmanned aerial vehicles comprises:

[0010] Step 110, high-altitude overview and abnormal rapid screening, including: synchronously collecting multi-spectral data by using a lightweight integrated multi-spectral imager; recording and processing multi-spectral data in real time by using an edge computing unit, marking abnormal areas, and generating potential points of interest;

[0011] Step 120, using an autonomous flight control system, conducting a close-in detailed investigation based on potential points of interest and task-driven, and obtaining detailed target data;

[0012] Step 130, input the detailed target data into the spectral-spatial feature deep fusion network, fuse the multispectral features through a multi-level cross-channel attention fusion module, and output the search and rescue target position with high confidence; the spectral-spatial feature deep fusion network is designed to realize intelligent identification of the search and rescue target;

[0013] Step 140, multi-modal path planning is performed, including: based on the search and rescue target position and the real-time three-dimensional environment model, a multi-dimensional dynamic search and rescue decision planning model is constructed by fusing multispectral dynamic risk data; a search and rescue path planning algorithm is designed to solve the multi-dimensional dynamic search and rescue decision planning model to obtain an optimal search and rescue strategy as a multi-modal search and rescue path scheme;

[0014] Step 150, the multi-modal search and rescue path scheme is visualized through a human-computer interaction interface, and a commander's decision instruction is received, and the process of intelligent identification and the multi-dimensional dynamic search and rescue decision planning model are iteratively optimized based on the action results after the task is completed.

[0015] On the other hand, the present application also provides a UAV closed-loop intelligent cooperative search and rescue device, which is based on a search and rescue UAV equipped with a lightweight integrated multispectral imager, an edge computing unit and an autonomous flight control system, and realizes intelligent cooperative search and rescue; the lightweight integrated multispectral imager is provided with a synchronous trigger controller for ensuring accurate alignment of data frames when different key spectral channel data are synchronously triggered; the device comprises:

[0016] The first module is used for high-altitude overview and abnormal rapid screening, including: synchronously collecting multispectral data by using the lightweight integrated multispectral imager; recording and processing the multispectral data in real time by using the edge computing unit, marking abnormal areas, and generating potential interest points;

[0017] The second module is used for using the autonomous flight control system to perform near-fine detailed investigation based on the potential interest points and task driving, and obtain detailed target data;

[0018] The third module is used for inputting the detailed target data into the spectral-spatial feature deep fusion network, fusing the multispectral features through a multi-level cross-channel attention fusion module, and outputting the search and rescue target position with high confidence; the spectral-spatial feature deep fusion network is designed to realize intelligent identification of the search and rescue target;

[0019] The fourth module is used for multi-modal path planning, including: based on the search and rescue target position and the real-time three-dimensional environment model, a multi-dimensional dynamic search and rescue decision planning model is constructed by fusing multispectral dynamic risk data; a search and rescue path planning algorithm is designed to solve the multi-dimensional dynamic search and rescue decision planning model to obtain an optimal search and rescue strategy as a multi-modal search and rescue path scheme;

[0020] A fifth module is configured to visualize the multi-modal search and rescue path scheme through a human-computer interaction interface and receive commander decision instructions, and iteratively optimize the intelligent identification process and multi-dimensional dynamic search and rescue decision planning model based on the action results after the task is completed.

[0021] In summary, the present application provides a kind of unmanned aerial vehicle closed loop intelligent cooperative search and rescue method, compared with prior art, the technical scheme of the present application has the following beneficial effects:

[0022] The present application provides a kind of unmanned aerial vehicle closed loop intelligent cooperative search and rescue method, which is different from the existing linear and segmented workflow. The method is not a single technology, but a complete and novel operation mode. Specifically:

[0023] 1. Through the two-stage intelligent acquisition mechanism combining high-altitude overview preliminary screening and close-in detailed investigation, an adaptive multi-spectral acquisition process based on task-driven is designed, and a lightweight AI model on board is used to analyze multi-spectral data in real time to autonomously discover potential points of interest (POI), and the flight mode is automatically adjusted according to the POI coordinates for detailed investigation, which ensures efficient and intelligent acquisition of high-value data and solves the problems of information isolation and efficiency bottleneck in the search and rescue process.

[0024] 2. By designing the architecture and fusion mechanism of spectral-spatial feature deep fusion network (SS-DFN), such as cross-channel attention mechanism, deep interaction and verification of multi-spectral data at the feature level are realized, so as to identify weak targets with unobvious features under a single sensor, ensuring the interpretation of complex data and solving the problem of weak target recognition.

[0025] 3. By fusing multi-dimensional dynamic risk search and rescue path real-time planning, a dynamic search and rescue path planning algorithm combining multi-spectral risk perception and three-dimensional terrain is designed, which specifically uses multi-spectral data to quantize environmental risks (such as fire points and slippery areas) in real time, and adds them as dynamic costs to the three-dimensional environment model to generate a multi-modal search and rescue path scheme that takes into account safety, efficiency and resource accessibility, to ensure safe and smart search and rescue decisions.

[0026] 4. Through human-computer collaboration and self-optimizing closed loop, the search and rescue method and system have the ability to interact with people, learn from actual combat and continuously evolve, ensuring the iterability and long-term effectiveness of the designed search and rescue method, and overcoming the static and closed nature of existing search and rescue systems. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 It is a step flowchart of a kind of unmanned aerial vehicle closed loop intelligent cooperative search and rescue method in an embodiment of the present application;

[0028] Figure 2This is a schematic diagram of the structure and workflow of a deep spectral-spatial feature fusion network in one embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0030] In one embodiment, the present invention provides a closed-loop intelligent collaborative search and rescue method for unmanned aerial vehicles (UAVs). The method is based on a search and rescue UAV equipped with a lightweight integrated multispectral imager, an edge computing unit, and an autonomous flight control system to achieve intelligent collaborative search and rescue. The lightweight integrated multispectral imager is equipped with a synchronous trigger controller to ensure accurate alignment of data frames when synchronously triggering data from different key spectral channels.

[0031] The aforementioned closed-loop intelligent collaborative search and rescue method for unmanned aerial vehicles (UAVs) is as follows: Figure 1 As shown, it includes the following steps:

[0032] Step 110 involves conducting an aerial overview and rapid anomaly screening, including: simultaneously acquiring multispectral data using a lightweight integrated multispectral imager; recording and processing multispectral data in real time using an edge computing unit, marking abnormal areas, and generating potential points of interest.

[0033] Step 120: Using the autonomous flight control system, conduct a detailed close-range survey based on potential points of interest and mission-driven approach to obtain detailed target data;

[0034] Step 130: Input detailed target data into the spectral-spatial feature deep fusion network, fuse multispectral features through a multi-level cross-channel attention fusion module, and output the location of the search and rescue target with high confidence; the spectral-spatial feature deep fusion network is designed to achieve intelligent identification of search and rescue targets;

[0035] Step 140 involves multimodal path planning, including: constructing a multidimensional dynamic search and rescue decision planning model based on the search and rescue target location and real-time 3D environment model by fusing multispectral dynamic risk data; designing a search and rescue path planning algorithm; solving the multidimensional dynamic search and rescue decision planning model to obtain the optimal search and rescue strategy, which serves as the multimodal search and rescue path scheme.

[0036] Step 150: Visualize the multimodal search and rescue path scheme through the human-computer interaction interface and receive the commander's decision instructions. After the mission is completed, iteratively optimize the intelligent recognition process and the multidimensional dynamic search and rescue decision planning model based on the action results.

[0037] In traditional drone search and rescue operations, data collection usually follows a carpet-scanning mode with a preset route. Although it can ensure full coverage of the area, its inherent defects are also obvious: low efficiency, serious waste of resources, and treating all areas the same, resulting in insufficient data quality in key areas.

[0038] To overcome these drawbacks, this invention designs a novel paradigm: an adaptive multispectral data fusion and task-driven intelligent acquisition method. Its operating logic is not a simple flight-recording, but an intelligent and dynamic perception-decision-action closed loop.

[0039] To better achieve the search and rescue mission of the drone, this invention is based on a search and rescue drone equipped with an autonomous flight control system at the hardware level. The search and rescue drone also needs to be equipped with a highly optimized lightweight integrated multispectral imager and edge computing unit.

[0040] In one embodiment, the lightweight integrated multispectral imager, serving as the "eyes" of search and rescue missions, does not simply bundle multiple sensors together. Instead, it utilizes multi-band sensors and a synchronous trigger controller to deeply integrate and synchronously trigger four key spectral channels: visible light (RGB), near-infrared (NIR), short-wave infrared (SWIR), and thermal infrared (TIR).

[0041] The intelligent data acquisition process of this invention includes two core stages:

[0042] The first phase is step 110, which involves high-altitude overview and rapid anomaly screening. The drone first conducts an overview flight of the entire target search and rescue area at a relatively high, safe altitude and speed. During this process, the following specific tasks are performed:

[0043] Step 111 utilizes a lightweight integrated multispectral imager to deeply integrate and synchronously trigger acquisition of four key spectral channels: visible light (RGB), near-infrared (NIR), short-wave infrared (SWIR), and thermal infrared (TIR), obtaining multispectral data. The selection of these four key spectral channels has a clear search and rescue orientation: visible light provides basic environmental texture and morphological information; thermal infrared excels at penetrating smoke and darkness, directly capturing thermal signals of vital signs; near-infrared has a unique advantage in analyzing anomalies under vegetation cover (such as camouflage and non-vegetation objects); and short-wave infrared is extremely sensitive to the spectral reflectance characteristics of man-made materials (such as clothing and equipment). Optical path design ensures that the sensors corresponding to the four key spectral channels generate the same instantaneous field of view, and pixel-level precise alignment is achieved through a synchronous trigger controller, finally outputting a unified data packet after deep integration. This integration of information data from multiple key spectral channels is not repetitive or superimposed, but rather enhances detection and recognition capabilities through complementarity and fusion, facilitating the acquisition of multi-dimensional fused information sources and providing an unprecedentedly rich data foundation for subsequent intelligent analysis.

[0044] Step 112 involves using the edge computing unit onboard the drone to record multispectral data. This process is not passive data recording, but rather the real-time processing of the synchronously acquired multispectral data through a lightweight, real-time AI analysis model. The model's task is not to precisely identify targets, but to quickly detect anomalous regions within the target search area and mark the pixels in these anomalous regions as Points of Interest (POIs). For example, in a vast forest, a cluster of pixels with a significant temperature difference from the background environment (thermal infrared anomaly), a small area where vegetation reflectance suddenly stops (near-infrared anomaly), or a reflectance point exhibiting typical synthetic fiber spectral characteristics (short-wave infrared anomaly) would all be marked as a POI. This process significantly reduces the scope of irrelevant information processing, much like using a coarse net to quickly filter out the vast majority of irrelevant areas.

[0045] A typical example of such a lightweight real-time AI analytics model is the YOLO (You Only Look Once) series of models, especially miniature versions optimized for edge devices, such as YOLOv8n (nanometer version) or YOLOv5s (small version).

[0046] The second stage, step 120, involves a detailed, close-up survey based on potential points of interest (POIs) and task-driven approaches to obtain high-quality, high-value, and detailed target data. Once the system identifies one or more POIs, it automatically switches from overview mode to detailed survey mode, embodying the core concept of task-driven operation: the initial analysis results from AI become the mission instructions for the UAV's next action. The UAV's autonomous flight control system plans an optimal approach route based on the POI's coordinates, automatically lowering its altitude, adjusting the gimbal angle, and slowing its speed to perform multi-angle, high-resolution hovering or circling photography of the POI. This close-up detailed survey acquires far clearer and richer details than high-altitude scanning, such as the target's outline, color, attitude, and more precise multispectral feature data. The entire process forms an intelligent feedback loop of discovery-approach-detailed survey, where every maneuver of the UAV is driven by data analysis results, ensuring that valuable endurance time and computing resources are used effectively.

[0047] The above steps completely change the traditional workflow of blindly collecting data first and then analyzing it offline. They inject intelligent analysis capabilities into the forefront of data collection, transforming the drone from a simple flying camera into an intelligent perception platform with preliminary thinking and judgment capabilities. This narrows the focus of search and rescue from vast unknown areas to high-value potential points of interest (POIs). The output is no longer a massive amount of raw images at the TB level, but a highly targeted and valuable dataset that has undergone preliminary screening and refined collection.

[0048] After the intelligent data acquisition processes in steps 110 and 120, the challenge facing drone search and rescue technology is no longer "where to search," but rather "how to understand high-value datasets based on points of interest (POIs)." Traditional AI target recognition technologies typically rely on visible light images, judging targets by learning spatial features such as shape, color, and texture. While the introduction of other sensors, such as thermal imaging, has been helpful, most existing technologies still remain at the level of simply overlaying or independently analyzing data from different sources. This approach often fails when dealing with weak targets in complex scenes. Weak targets refer to those that, due to environmental obstruction, camouflage, distance, or inconspicuous features, cannot form clear and complete characteristics under any single spectral channel.

[0049] Furthermore, to overcome the challenge of weak target failure in complex scenarios, this invention constructs a spectral-spatial feature deep fusion network (SS-DFN) to achieve robust, rapid, and intelligent identification of weak targets. The operational logic of this deep fusion network can be understood as simulating the comprehensive judgment process of an experienced search and rescue expert. When making search and rescue judgments, an expert doesn't just look at the shape of an object; they combine all available clues: unnatural flashes of sunlight, an unnatural color in the grass, and subtle anomalies in the surrounding environment. The SS-DFN proposed in this invention is designed based on this idea. It no longer treats images from visible light, near-infrared, short-wave infrared, and thermal infrared as four independent inputs, but rather as a multi-dimensional information body describing the target with inherent connections.

[0050] like Figure 2 As shown, the SS-DFN includes at least multiple parallel feature extraction branches and a multi-level cross-channel attention fusion module. Each of the parallel feature extraction branches is specifically responsible for extracting preliminary features from its respective key spectral channels. For example, the visible light branch focuses on learning the spatial geometric information of the target, such as its contour and texture; the thermal infrared branch focuses on the intensity, distribution, and shape of the heat source; and the near-infrared and short-wave infrared branches extract unique spectral reflectance features related to vegetation, soil, and artificial fabrics, respectively. The cross-channel attention fusion module achieves mutual communication and fusion of features from different key spectral channels based on cross-channel attention and cross-channel fusion mechanisms. The fusion of features from different key spectral channels is a multi-level feature fusion process from shallow to deep layers.

[0051] The cross-channel attention mechanism acts as a "decision-making advisor," with its core task being learning and decision-making to generate weighted mappings representing the importance of features from different branch channels. Specifically, it receives preliminary features extracted from each parallel feature extraction branch, analyzes them, and calculates a set of weights to identify the more important, relevant, and valuable features of those branches, and labels their corresponding spatial locations. The cross-channel fusion mechanism acts as an "executor," with its core task being to perform multi-level fusion operations. It is configured to use the weighted mappings to perform multiple weighted fusion operations on the output features of the multiple parallel branches. Each weighted fusion operation specifically receives the output features of the parallel branches and the weights calculated by the cross-channel attention mechanism, and designs mathematical operations based on these weights to merge the features into a unified fused feature representation. These mathematical operations can be weighted summation or convolution operations after channel concatenation, etc.

[0052] Specifically, in step 130, such as Figure 2As shown, detailed target data is input into a spectral-spatial feature deep fusion network. Multispectral features are fused through a multi-layered cross-channel attention fusion module, outputting high-confidence search and rescue target locations, including:

[0053] The processing flow of the SS-DFN network begins with the parallel extraction of features. When a set of synchronous multispectral data is input into the network, multiple parallel feature extraction branches within the SS-DFN network are used to extract preliminary features from a single spectral channel in the shallow layers (early network layers) of the network, obtaining single-channel features such as edges, colors, textures (visible light), hotspot intensity (thermal infrared), and specific reflectance (near / shortwave infrared).

[0054] The key step lies in the deep fusion of cross-channel features. After extracting the initial single-channel features, the SS-DFN network does not hastily stitch them together, but rather engages them deeply through a series of carefully designed, multi-layered cross-channel attention fusion modules. As mentioned above, these modules, based on cross-channel attention and cross-channel fusion mechanisms, enable the communication and fusion of features from different key spectral channels, specifically including:

[0055] By using a shallow cross-channel attention mechanism to align and associate these single-channel features, the correspondence between different channel features in spatial location is learned, and clues of different single-channel features are associated (e.g., associating hotspots and unnatural textures that overlap in spatial location). A shallow cross-channel fusion mechanism is used to generate preliminary fusion features, and a low-confidence region that may contain a target is initially formed.

[0056] By utilizing a deep cross-channel attention mechanism, high-level semantic features of different channels are aligned and associated. Then, a deep cross-channel fusion mechanism is used to weight and fuse the aligned and associated high-level semantic features to generate deep fused features.

[0057] Repeat the above process until the network depth or feature accuracy requirements are met, and use the decoder or output layer to obtain recognition results with high confidence and accurate annotation of the search and rescue target location and target outline.

[0058] To deepen understanding, consider a specific scenario: a missing person partially obscured by foliage in a dense forest. In a visible light image, AI might only see scattered, non-human-shaped patches of color. In a thermal infrared image, it could detect a blurry, warm area, but this might be unstable due to heat loss. This is where the fusion mechanism comes into play. The non-natural color features extracted by the visible light channel guide the network to focus more on the weak thermal signals at the corresponding locations in the thermal infrared feature map. Simultaneously, if the short-wave infrared channel detects spectral reflectance signals consistent with nylon clothing in this area, this signal significantly enhances the confidence of other channels in determining that the area is man-made rather than natural. Through this method, the SS-DFN network dynamically learns the inherent correlation between different spectral features—that is, what spatial forms are typically accompanied by what thermal signals and spectral reflectance features. After multi-layered deep fusion, the network outputs no longer a simple yes or no judgment, but a highly confident recognition result with precise annotation of the target's location and contour. The advantage of this fusion method lies in its powerful completion capability. Even if a target is completely invisible in a certain spectral channel, as long as there are clues in other channels that can be correlated, the network still has a high probability of being able to piece together these scattered and weak chains of evidence, ultimately achieving accurate target identification. This enables search and rescue systems to maintain high recognition rates even under complex concealment, camouflage, or extreme weather conditions where traditional technologies are ineffective, achieving a leap from simple feature overlay to true multi-dimensional information collaborative cognition, and providing the most reliable target indication for subsequent rescue operations.

[0059] Once the system confirms the location of the search and rescue target with high confidence, the search and rescue operation transitions from the search phase to the rescue phase. Traditional rescue route planning often relies on existing electronic maps (GIS), and its core algorithm typically finds the shortest path on a two-dimensional plane that avoids static obstacles (such as buildings and rivers). However, in real-world wilderness search and rescue environments, this simplified model has a fatal flaw: it ignores the complex undulations of the terrain, the ubiquitous dynamic environmental risks, and the actual accessibility of rescue resources.

[0060] To compensate for the above deficiencies, multimodal path planning is performed in step 140. Specifically, this includes:

[0061] Step 141: Based on the location of the search and rescue target, construct a multi-dimensional dynamic decision-making model that comprehensively considers three-dimensional terrain, dynamic risks, and resource accessibility.

[0062] The first step is to construct a high-precision, real-time 3D environment model. Unlike relying on outdated satellite maps, this system utilizes data rapidly generated by LiDAR onboard a UAV during the detailed survey phase (step 120) or through oblique photogrammetry to construct a digital surface model (DSM) with centimeter-level precision in real time. This 3D environment model not only includes macroscopic terrain information such as mountains, gullies, and slopes, but also encompasses microscopic obstacles such as vegetation height, fallen trees, and large rocks. This transforms the basis of path planning from a flat map into a virtual sandbox highly consistent with the real world, allowing for detailed computer analysis. Based on this, the physical costs of any path, such as length, slope, and cumulative elevation gain, can be accurately calculated.

[0063] The second step involves integrating multispectral dynamic risk data. Simply possessing three-dimensional terrain data is insufficient to guarantee safety. The most innovative aspect here lies in introducing the concept of dynamic risk quantification, further utilizing multispectral data continuously collected by the drone during flight to conduct real-time risk assessment and labeling of the environment. This process is akin to overlaying multiple dynamic risk layers onto a three-dimensional sand table. For example, by analyzing thermal infrared (TIR) ​​data, the system can identify potential fire hazard areas, geothermal instability zones, and even concealed underground cavities (due to their potentially different surface temperatures). By analyzing near-infrared (NIR) and short-wave infrared (SWIR) data, the system can calculate indicators such as the Normalized Difference Water Index (NDWI), thereby accurately locating swamps and slippery muddy areas—areas posing significant risks to ground personnel and vehicles. Furthermore, by combining slope data and soil moisture analysis, it can predict and identify high-risk areas for secondary disasters such as landslides and debris flows. This risk information is no longer static but is updated in real-time with continuous drone reconnaissance and environmental changes (such as rainfall).

[0064] Step 142: The search and rescue route planning algorithm is activated to solve a multi-dimensional dynamic search and rescue decision-making planning model to obtain the optimal search and rescue strategy, serving as a multimodal search and rescue route plan. The route planning algorithm is only officially launched after accurate terrain and dynamic risks are understood. However, its goal is not simply to find the shortest path, but to solve for a multi-objective, multi-modal optimal rescue strategy. When searching for feasible paths, the algorithm simultaneously optimizes multiple objectives: path length, travel time, physical exertion (calculated based on slope), and the comprehensive risk value of path crossing. Ultimately, the system will not provide only a single optimal path, but will generate several alternative plans, each with its own characteristics, and present them to the commander in a highly intuitive way. For example: Option A is the shortest path for ground personnel to walk, but it has a high risk level and takes about 2 hours; Option B is a slightly longer path, but it perfectly avoids all identified high-risk areas and has the highest safety level, taking about 3.5 hours; Option C suggests that ground personnel proceed to a specific coordinate point, and the system has already identified a flat, open area near that point that is free from the threat of high-voltage lines, making it the best location for helicopter rappelling or airdropping supplies.

[0065] The aforementioned planning approach, which deeply integrates three-dimensional terrain, multispectral risk perception, and resource accessibility (such as communication signal coverage and helicopter landing points), elevates search and rescue decision-making from two-dimensional route planning to three-dimensional action strategy planning. It provides commanders with not just a simple arrow, but a dynamic and executable operational map that includes comprehensive considerations such as time, risk, and resource allocation. This maximizes the safety of rescue personnel and improves the success rate of search and rescue operations in the ever-changing search and rescue environment.

[0066] The value of an advanced technological system lies not only in its powerful automation capabilities, but also in its ability to efficiently collaborate with the wisdom of human experts and learn from each practice. Traditional decision support systems are often one-way reporting tools; they present the analysis results to humans, and the decision-making process ends there.

[0067] In the final step 150, the present invention aims to construct a search and rescue decision-making scheme that far surpasses traditional decision-making schemes, is complete, and features a two-way interactive human-machine symbiotic ecosystem. It includes two closely related core concepts: a closed loop of human-machine collaboration during mission execution and the system's self-evolution after the mission ends.

[0068] The technical objective of this invention is not to design a black box that attempts to replace the commander. On the contrary, it is designed as an extremely transparent and highly intuitive situational awareness platform.

[0069] First, the human-machine collaboration closed loop in task execution includes:

[0070] After the information is processed and analyzed in the aforementioned steps, all key information—including real-time multispectral imagery from drones, target locations identified by AI (with close-up shots and confidence scores), a 3D terrain model, dynamic risk layers, and multiple rescue route options recommended by the system (each with detailed estimated time, risk index, and resource requirements)—is seamlessly integrated and presented on a large screen in the command center via a graphical, interactive interface. This presentation method itself is empowering. Commanders no longer need to deal with fragmented data and reports, but can gain a comprehensive view of the entire situation. More importantly, this interface is interactive. Commanders can review, compare, and select system-recommended options based on their experience and intuition. They can click on any recommended route, and the human-computer interaction system will immediately simulate key perspectives and risk points along the route. They can also fine-tune the plan, such as manually modifying a route node, and the system will immediately recalculate the risks and time changes resulting from the adjustment. This interactivity ensures a perfect combination of human experience and wisdom with the powerful computing capabilities of machines. Once the commander makes a final decision (e.g., selecting Plan B and instructing the helicopter to divert to point C), this decision is received by the system as a new instruction and transmitted in real time to the drones or rescue teams ahead, guiding their specific actions and forming a closed-loop instruction system. This closed loop is continuous and dynamic. While the rescue team is moving as planned, the drones ahead do not cease their work; they continue to act as aerial sentinels, conducting advance reconnaissance along the rescue route. If a sudden landslide occurs ahead, or the planned helicopter landing point becomes unsafe due to strong winds, the drone's sensors will immediately detect these changes. The system will immediately update the risk model, issue an alert on the command interface, and automatically plan a new evasive route. This real-time reconnaissance-early warning-replanning capability ensures that the rescue operation can adapt to changes in the search and rescue environment at any time, forming a safety closed loop throughout the operation.

[0071] Secondly, and more significantly, is the system's self-evolution after the mission. The end of a search and rescue mission marks the beginning of learning and optimization for this system. All data generated during the mission—from raw multispectral imagery to every judgment made by the AI ​​(whether right or wrong), to the commander's final decision, and the actual results of the rescue operation (e.g., the actual time spent by the rescue team, unforeseen obstacles encountered along the way)—is completely and structurally stored in the system's memory. This database becomes the core data resource for the system's self-optimization. After the mission ends, the system automatically initiates a debriefing and learning process. For example, if the AI ​​fails to identify a target in a certain scenario, but it is later confirmed that the target does exist, the system will mark this case as a missed sample and use it to retrain and fine-tune the spectral-spatial feature deep fusion network in step 130, thereby performing better in similar scenarios in the future. Similarly, if the rescue team encounters difficulties on a path assessed as low-risk by the system, the system will analyze the reasons and adjust the weight parameters of the multi-dimensional dynamic search and rescue decision-making planning model in step 140. The commander's decisions themselves serve as a crucial monitoring signal, allowing the system to learn when human expert choices outperform the algorithm's initial recommendations. Through this post-mission self-optimization mechanism, the system is no longer a static tool but an intelligent agent capable of learning and continuously growing from real-world experience. Each successful rescue reinforces its correct model, and each challenge encountered corrects its potential flaws. This allows the search and rescue system, built upon these methodological steps, to continuously evolve its adaptability and intelligence with increased usage, ensuring more accurate and reliable decision support in future search and rescue missions. This is the fundamental difference between the technical solution of this invention and all one-off, static systems, and the core guarantee for its long-term technological advancement and practical value.

[0072] In the foregoing embodiments, this invention provides an intelligent collaborative search and rescue method for drones based on a closed loop of perception-decision-action-learning. Unlike all existing linear, segmented workflows, this method is not a single technology, but a complete and novel operational mode: AI performs initial analysis in real-time, driving the drone to conduct detailed searches (intelligent data collection); detailed target data then triggers deep recognition (precise targeting); the recognition results immediately initiate path planning considering dynamic risks (strategy generation); and all information is provided to the commander for human-machine collaborative decision-making. Finally, the mission data can also feed back into the system for self-optimization throughout the entire process. This process represents a system-wide, holistic protection, a completely new and intelligent search and rescue paradigm. Specifically, it includes three key technological supports:

[0073] (1) A task-driven adaptive multispectral acquisition method, including a two-stage intelligent acquisition mechanism that combines high-altitude overview screening with close-range detailed investigation. Specifically, a lightweight airborne AI model is used to analyze multispectral data in real time to autonomously discover potential points of interest (POIs), and the flight mode is automatically adjusted according to the POI coordinates for detailed investigation, which ensures efficient and intelligent acquisition of high-value data.

[0074] (2) The weak target recognition method based on deep spectral-spatial feature fusion is to design the architecture and fusion mechanism of the deep spectral-spatial feature fusion network (SS-DFN), such as cross-channel attention mechanism and cross-channel fusion mechanism, to realize multi-level deep interaction and fusion of multispectral data at the feature level, thereby identifying weak targets with inconspicuous features under a single sensor, and ensuring the interpretation of complex data.

[0075] (3) A real-time search and rescue path planning method that integrates multidimensional dynamic risks, including designing a dynamic search and rescue path planning algorithm that combines multispectral risk perception with three-dimensional terrain, using multispectral data to quantify environmental risks (such as fire points and slippery areas) in real time, and superimposing them as dynamic costs onto the three-dimensional environment model to generate a multi-mode rescue strategy that takes into account safety, efficiency and resource accessibility, so as to ensure that safe and smart search and rescue decisions are made.

[0076] In one embodiment, the present invention provides a closed-loop intelligent collaborative search and rescue device for unmanned aerial vehicles (UAVs). The device is based on a search and rescue UAV equipped with a lightweight integrated multispectral imager, an edge computing unit, and an autonomous flight control system, enabling intelligent collaborative search and rescue. The lightweight integrated multispectral imager is equipped with a synchronization trigger controller to ensure precise alignment of data frames when synchronously triggering data from different key spectral channels. The device includes:

[0077] The first module is used for high-altitude overview and rapid anomaly screening, including: synchronously acquiring multispectral data using a lightweight integrated multispectral imager; recording and processing multispectral data in real time using an edge computing unit, marking abnormal areas, and generating potential points of interest.

[0078] The second module is used to conduct detailed close-range reconnaissance based on potential points of interest and mission-driven approaches using the autonomous flight control system to obtain detailed target data.

[0079] The third module is used to input detailed target data into the spectral-spatial feature deep fusion network, and fuse multispectral features through a multi-level cross-channel attention fusion module to output the location of the search and rescue target with high confidence; the spectral-spatial feature deep fusion network is designed to realize intelligent identification of search and rescue targets;

[0080] The fourth module is used for multimodal path planning, including: based on the location of the search and rescue target and the real-time 3D environment model, by fusing multispectral dynamic risk data, constructing a multidimensional dynamic search and rescue decision planning model; designing a search and rescue path planning algorithm, solving the multidimensional dynamic search and rescue decision planning model to obtain the optimal search and rescue strategy as a multimodal search and rescue path scheme;

[0081] The fifth module is used to visualize the multimodal search and rescue path plan through a human-computer interaction interface and receive the commander's decision instructions. After the mission is completed, iteratively optimize the intelligent identification process and the multi-dimensional dynamic search and rescue decision planning model based on the action results.

[0082] On the other hand, the present invention provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the UAV closed-loop intelligent collaborative search and rescue method provided in any of the above embodiments. The computer device can be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device stores sample data. The network interface of the computer device is used for communication with external terminals via a network connection.

[0083] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the UAV closed-loop intelligent collaborative search and rescue method provided in any of the above embodiments.

[0084] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0085] Matters not covered in this invention are common knowledge. The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered to be within the scope of this specification.

[0086] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A closed-loop intelligent collaborative search and rescue method for unmanned aerial vehicles (UAVs), characterized in that, The method is based on a search and rescue drone equipped with a lightweight integrated multispectral imager, an edge computing unit, and an autonomous flight control system to achieve intelligent collaborative search and rescue; the lightweight integrated multispectral imager is equipped with a synchronous trigger controller to ensure the precise alignment of data frames when synchronously triggering data from different key spectral channels. The aforementioned closed-loop intelligent collaborative search and rescue method for unmanned aerial vehicles includes: Step 110 involves conducting an aerial overview and rapid anomaly screening, including: simultaneously acquiring multispectral data using a lightweight integrated multispectral imager; recording and processing multispectral data in real time using an edge computing unit, marking abnormal areas, and generating potential points of interest. Step 120: Using the autonomous flight control system, conduct a detailed close-range survey based on potential points of interest and mission-driven approach to obtain detailed target data; Step 130: Detailed target data is input into a spectral-spatial feature deep fusion network. Multispectral features are fused through a multi-layered cross-channel attention fusion module to output the search and rescue target location with high confidence. The spectral-spatial feature deep fusion network is designed to achieve intelligent identification of search and rescue targets. The network includes at least multiple parallel feature extraction branches and a multi-layered cross-channel attention fusion module. The multi-layered cross-channel attention fusion module includes shallow and deep layers. Each branch in the parallel feature extraction branches is specifically responsible for extracting preliminary features from its respective key spectral channels. The cross-channel attention fusion module utilizes cross-channel attention and cross-channel fusion mechanisms to achieve mutual communication and fusion of features from different key spectral channels. Detailed target data is input into a deep spectral-spatial feature fusion network; The features of key spectral channels are extracted separately using parallel feature extraction branches to obtain single-channel features; By utilizing the cross-channel attention mechanism of the shallow cross-channel attention fusion module, the intrinsic correlation between different single-channel features is learned, and the cross-channel fusion mechanism of the corresponding layer is used to generate preliminary fused features; The high-level semantic features of different key spectral channels are aligned and associated using the cross-channel attention mechanism of the deep cross-channel attention fusion module, and the high-level semantic features after alignment and association are weighted and fused using the cross-channel fusion mechanism of the corresponding layer to generate deep fused features. Repeat the above process until the network depth or feature accuracy requirements are met, and use the decoder or output layer to output the recognition result with high confidence and accurate annotation of the search and rescue target location and target outline; Step 140 involves multimodal path planning, including: constructing a multidimensional dynamic search and rescue decision planning model based on the search and rescue target location and real-time 3D environment model by fusing multispectral dynamic risk data; designing a search and rescue path planning algorithm; solving the multidimensional dynamic search and rescue decision planning model to obtain the optimal search and rescue strategy, which serves as the multimodal search and rescue path scheme. Step 141: Based on the location of the search and rescue target, construct a multi-dimensional dynamic decision-making model, including: Constructing a high-precision, real-time 3D environment model includes using detailed target data to construct a digital surface model with centimeter-level precision in real time; the 3D environment model contains both macroscopic terrain information and microscopic obstacle information. By integrating three-dimensional environmental models with multispectral dynamic risk data, a multi-dimensional dynamic decision-making model is constructed, including real-time risk assessment and labeling of the environment through quantification of dynamic risks. The optimization objectives of the multi-dimensional dynamic decision-making model include: path length, travel time, physical exertion, and the comprehensive risk value of path crossing; Step 142: Activate the search and rescue path planning algorithm to solve the multi-dimensional dynamic search and rescue decision planning model to obtain the optimal search and rescue strategy as a multimodal search and rescue path scheme. When searching for feasible paths, the search and rescue path planning algorithm simultaneously optimizes multiple objectives: path length, travel time, physical exertion calculated based on slope, and comprehensive risk value of path crossing, providing more than one optimal path and generating alternative schemes. Step 150: Visualize the multimodal search and rescue path scheme through the human-computer interaction interface and receive the commander's decision instructions. After the mission is completed, iteratively optimize the intelligent recognition process and the multidimensional dynamic search and rescue decision planning model based on the action results.

2. The UAV closed-loop intelligent collaborative search and rescue method according to claim 1, characterized in that, The edge computing unit employs a lightweight real-time AI analysis model to process synchronously acquired multispectral data in real time and mark potential points of interest in abnormal areas.

3. The UAV closed-loop intelligent collaborative search and rescue method according to claim 2, characterized in that, The aforementioned detailed, proximity-based investigation, driven by potential points of interest and task-driven approaches, includes: Based on the marking of one or more potential points of interest, the drone automatically switches from overview mode to close-up detailed investigation mode; The drone's autonomous flight control system plans an optimal approach route based on the coordinates of the marked potential points of interest. It automatically lowers its flight altitude, adjusts the gimbal angle, and slows down its flight speed to perform multi-angle, high-resolution hovering or circling photography of the potential points of interest, obtaining detailed target data.

4. The UAV closed-loop intelligent collaborative search and rescue method according to claim 3, characterized in that, The key spectral channels include at least the visible light, near-infrared, short-wave infrared, and thermal infrared spectral channels.

5. The UAV closed-loop intelligent collaborative search and rescue method according to claim 4, characterized in that, The single-channel feature includes: Target spatial geometric information extracted by visible light branching; The intensity, distribution, and shape of the heat source extracted by thermal infrared branching; Near-infrared and short-wave infrared branches extract unique spectral reflectance characteristics associated with vegetation, soil, and artificial fabrics.

6. The UAV closed-loop intelligent collaborative search and rescue method according to claim 5, characterized in that, Step 150 includes: Step 151, through interventionist human-computer interaction, realize a closed loop of human-computer collaboration in task execution, including: Design an interactive human-computer interface; The multimodal search and rescue path scheme and its generation process are presented on the human-computer interaction interface. Commanders of search and rescue missions use human-computer interaction interfaces to intervene and make decisions regarding the mission. Step 152, perform system self-evolution after the task is completed, including: All data generated during the previous search and rescue mission will be stored completely and in a structured manner in the database of the hardware system where the human-computer interaction interface is located. Design an automatic review and learning program to mark historical data as successful and unsuccessful, use the marked data to retrain the spectral-spatial feature deep fusion network in step 130, and correct the weight parameters of the multi-dimensional dynamic search and rescue decision planning model in step 140.

7. A closed-loop intelligent collaborative search and rescue device for unmanned aerial vehicles (UAVs), characterized in that, The device is based on a search and rescue drone equipped with a lightweight integrated multispectral imager, an edge computing unit, and an autonomous flight control system, enabling intelligent collaborative search and rescue. The lightweight integrated multispectral imager is equipped with a synchronous trigger controller to ensure precise alignment of data frames when synchronously triggering data from different key spectral channels. The device includes: The first module is used for high-altitude overview and rapid anomaly screening, including: synchronously acquiring multispectral data using a lightweight integrated multispectral imager; recording and processing multispectral data in real time using an edge computing unit, marking abnormal areas, and generating potential points of interest. The second module is used to conduct detailed close-range reconnaissance based on potential points of interest and mission-driven approaches using the autonomous flight control system to obtain detailed target data. The third module is used to input detailed target data into a spectral-spatial feature deep fusion network, fuse multispectral features through a multi-level cross-channel attention fusion module, and output the location of the search and rescue target with high confidence. The spectral-spatial feature deep fusion network is designed to achieve intelligent identification of search and rescue targets. The spectral-spatial feature deep fusion network includes at least multiple parallel feature extraction branches and a multi-level cross-channel attention fusion module. The multi-level cross-channel attention fusion module includes shallow and deep cross-channel attention fusion modules. Each branch in the parallel feature extraction branches is specifically responsible for extracting preliminary features from its respective key spectral channels. The cross-channel attention fusion module utilizes cross-channel attention and cross-channel fusion mechanisms to achieve mutual communication and fusion of features from different key spectral channels. Detailed target data is input into a deep spectral-spatial feature fusion network; The features of key spectral channels are extracted separately using parallel feature extraction branches to obtain single-channel features; By utilizing the cross-channel attention mechanism of the shallow cross-channel attention fusion module, the intrinsic correlation between different single-channel features is learned, and the cross-channel fusion mechanism of the corresponding layer is used to generate preliminary fused features; The high-level semantic features of different key spectral channels are aligned and associated using the cross-channel attention mechanism of the deep cross-channel attention fusion module, and the high-level semantic features after alignment and association are weighted and fused using the cross-channel fusion mechanism of the corresponding layer to generate deep fused features. Repeat the above process until the network depth or feature accuracy requirements are met, and use the decoder or output layer to output the recognition result with high confidence and accurate annotation of the search and rescue target location and target outline; The fourth module is used for multimodal path planning, including: based on the location of the search and rescue target and the real-time 3D environment model, by fusing multispectral dynamic risk data, constructing a multidimensional dynamic search and rescue decision planning model; designing a search and rescue path planning algorithm, solving the multidimensional dynamic search and rescue decision planning model to obtain the optimal search and rescue strategy as a multimodal search and rescue path scheme; Based on the location of the search and rescue target, a multi-dimensional dynamic decision-making model is constructed, including: Constructing a high-precision, real-time 3D environment model includes using detailed target data to construct a digital surface model with centimeter-level precision in real time; the 3D environment model contains both macroscopic terrain information and microscopic obstacle information. By integrating three-dimensional environmental models with multispectral dynamic risk data, a multi-dimensional dynamic decision-making model is constructed, including real-time risk assessment and labeling of the environment through quantification of dynamic risks. The optimization objectives of the multi-dimensional dynamic decision-making model include: path length, travel time, physical exertion, and the comprehensive risk value of path crossing; The search and rescue path planning algorithm is activated to solve the multi-dimensional dynamic search and rescue decision planning model to obtain the optimal search and rescue strategy as a multimodal search and rescue path scheme. When searching for feasible paths, the search and rescue path planning algorithm simultaneously optimizes multiple objectives: path length, travel time, physical exertion calculated based on slope, and comprehensive risk value of path crossing, in order to provide more than one optimal path and generate alternative schemes. The fifth module is used to visualize the multimodal search and rescue path plan through a human-computer interaction interface and receive the commander's decision instructions. After the mission is completed, iteratively optimize the intelligent identification process and the multi-dimensional dynamic search and rescue decision planning model based on the action results.

Images