Search and rescue system and search and rescue method using the same

Abstract

The system allows drones to be remotely controlled from anywhere in the world via the VR-Metaverse space, and by using AI analysis to determine optimal search patterns and thresholds for sensor information, it enables safe and efficient rescue operations with minimal on-site personnel. Solution: The operator wears a VR display device from anywhere in the world and remotely controls the drone from the search site via the metaverse. AI analyzes environmental data collected on-site to derive the optimal search pattern, and AI analyzes the drone's multi-sensor information to determine thresholds and identify the area where rescuers are located. While ensuring the safety of the operator through global remote communication, a three-mode control system—manual operation, AI automatic flight, and safe return—enables efficient rescue operations with minimal personnel on-site.

Description

[Technical Field]

[0001] This invention relates to a system for supporting rescue operations at marine accidents and disaster sites. In particular, it relates to a system that utilizes VR and metaverse technologies to enable operators to remotely control drones at search sites from anywhere in the world, and enables efficient and safe rescue operations by determining optimal search patterns and threshold values ​​for sensor information through AI analysis. [Background technology]

[0002] Rescue operations following maritime accidents or disasters are usually carried out by rescuers dispatched directly to the scene, but due to worsening weather and sea conditions and the risk of secondary disasters, it is difficult to ensure the safety of rescuers while responding quickly. In particular, when conducting rescue operations in remote or dangerous areas, the number of personnel that can be deployed to the scene is limited, making it urgent to establish efficient search methods.

[0003] In recent years, disaster relief technology using drones has been attracting attention, but conventional technology generally involves a drone being directly operated by a human operator near the site. For example, Japanese Patent Laid-Open Publication No. 2019-123456, "Drone-Based Disaster Site Monitoring System" (Patent Document 1), discloses a technology for grasping the situation by operating a drone near the site, but does not fully consider ensuring the safety of the operator or overcoming geographical constraints.

[0004] Furthermore, US Patent US 10,877,542 B2 "Remote UAV Operation System" (Patent Document 2) proposes remote control technology, but it is limited to operation via simple video transmission and does not mention three-dimensional situational awareness in the metaverse space or the derivation of optimal search patterns through AI analysis.

[0005] Furthermore, Patent Document 3 discloses a search pattern generation technology using AI, but does not take into consideration the ability to link with the VR-Metaverse environment or the ability to identify areas where people needing rescue are present by judging thresholds in sensor information.

[0006] In the prior art, 1. Ensuring operator safety through remote VR control 2. Intuitive 3D operation in VR-Metaverse space 3. Deriving optimal search patterns that adapt to the environment through AI analysis 4. Threshold determination system using AI analysis of sensor information The elements of No system was proposed. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2019-123456 [Patent Document 2] U.S. Patent No. 10,877,542 [Patent Document 3] U.S. Patent Application Publication No. 2020 / 0301427 Summary of the Invention [Problem to be solved by the invention]

[0008] In order to solve the above problems, the present invention aims to provide a system that integrates VR-Metaverse technology, drones, and AI analysis, allowing operators to safely operate drones at search sites from anywhere in the world, and enables efficient and safe rescue operations with a minimum number of on-site personnel by determining optimal search patterns and thresholds using AI analysis.

[0009] Specifically, we aim to solve the following problems: 1. The operator can access the search site remotely from anywhere in the world, completely eliminating the risk of secondary disasters. 2. Effective drone control from a remote location can be achieved through intuitive operation in the VR-Metaverse space. 3. Automatically derive the optimal search pattern according to the situation through AI analysis based on local environmental data. 4. By using AI to analyze drone sensor information and determine thresholds, the area in need of rescue can be identified with high accuracy. 5. Minimize the number of personnel deployed to the scene and ensure safe and efficient rescue operations. [Means for solving the problem]

[0010] In order to solve the above problems, the present invention provides a remote integrated search system including the following components:

[0011] According to an embodiment of the present invention, as shown in FIG. 1, the system comprises the following elements: 1. VR display device (10): A device that presents three-dimensional visual information to operators anywhere in the world 2. Metaverse Generation Server (20): Generates a 3D virtual environment that reflects the actual search site. 3. Field Data Collection Unit (30): Collects environmental data at the search site. 4. Remotely Controlled Drone (40): Flying and controlled by remote control 5. AI Analysis System (50): AI analysis of environmental data and sensor information 6. Remote Communication System (60): Connecting the World and the Site 7. Multi-sensor device (70): Various sensors mounted on the drone 8. Optimal search pattern derivation unit (80): Generates optimal search patterns through AI analysis 9. Threshold determination unit (90): Identifies the area where rescue-needing people exist by analyzing sensor information 10. Drone Safety Control Unit (100): A system that controls three modes: manual control, AI automatic flight, and safe return. 11. Situation-responsive drone deployment unit (110): Calculates and deploys the optimal number of drones depending on the area size, urgency, and difficulty of search.

[0012] The main technical features of the present invention are as follows: 1. Ensure operator safety through remote VR control accessible from anywhere in the world 2. Three-dimensional and intuitive drone operation in the VR-Metaverse space 3. Automatically deriving optimal search patterns that adapt to the situation through AI analysis of local environmental data 4. High-accuracy identification of areas where people need rescue are present through threshold determination using AI analysis of multi-sensor information 5. Safety is ensured through three control modes: manual control, AI automatic flight, and safe return. 6. Situation-responsive drone deployment system based on area size, urgency, and search difficulty 7. Minimizing on-site personnel while simultaneously achieving safety [Effects of the Invention]

[0013] The present invention is expected to have the following effects. 1. Complete elimination of the risk of secondary disasters due to remote operation of pilots on a global scale 2. Effective drone control from anywhere in the world through VR-Metaverse operation 3. AI analysis optimizes search patterns, significantly improving search efficiency 4. Threshold Judgment System for Improving Accuracy in Finding Victims in Need of Rescue 5.3-mode control system ensures high level of drone safety 6. Achieving safe and efficient rescue operations by minimizing the number of personnel deployed to the scene 7. Utilizing advanced pilots on a global scale, completely transcending geographical limitations [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a diagram showing the overall system configuration of the present invention; [Figure 2] FIG. 10 is a diagram showing the correspondence between environmental conditions and optimal search patterns. [Figure 3] FIG. 10 is a diagram showing the process of identifying the area where a person in need of rescue exists using a threshold determination system. [Figure 4] This is a diagram showing an example taken around 2 NM south of Enoshima. DETAILED DESCRIPTION OF THE INVENTION

[0015] As shown in Figure 1, this system consists of a "remote control side" located somewhere in the world and a "search site side." On the remote control side, a pilot anywhere in the world can wear a VR display device (10) and operate a drone in a three-dimensional virtual environment created by the metaverse generation server (20), even from a location thousands of kilometers away from the actual search site.

[0016] At the search site, local data is input by the local crew into the local data collection unit (30). That is, local data is input into the local data collection unit (30). This information (local data) is environmental data. The search site and the remote control side are connected by a remote communication system (60), which allows information sharing and control command transmission in real time.

[0017] As shown in Table 1 (environmental data to be collected), the following environmental data on the search site is input and collected by the on-site data collection unit (30): [Table 1]

[0018] The environmental data input and collected by the field data collection unit (30) is transmitted to the optimum search pattern derivation unit (80). The optimum search pattern derivation unit (80) derives an optimum search pattern by the following AI analysis process: 1. Structural analysis of multidimensional environmental data using tensor decomposition: A mathematical method for decomposing environmental data (wind direction, wind speed, wave height, tidal current, etc.) expressed as multidimensional arrays into low-dimensional structural patterns and extracting correlations between the data. 2. Classification of environmental condition patterns using neural networks: A method to classify current environmental conditions into known patterns using neural networks trained on data from past rescue cases. 3. Drift prediction calculation using nonlinear fluid simulation: A calculation method that uses numerical analysis to predict the drift trajectory of an object under complex weather and sea conditions. 4. Calculating the probabilistic search area using Monte Carlo simulation: A method for calculating the probability distribution of the presence of victims using a statistical method using random sampling. 5. Optimal route search algorithm based on graph theory: A mathematical method that represents the search area as a graph structure and calculates the shortest and most efficient search route.

[0019] As shown in Table 2 (environmental conditions and optimal search patterns), the optimal one is automatically selected from the five basic search patterns (A to E) shown in Figure 2 according to the combination of environmental conditions. [Table 2]

[0020] The operator wearing the VR display device (10) can comprehensively grasp the following information, including information reflecting environmental data, within the three-dimensional virtual environment created by the metaverse generation server (20): 1. Visualization of actual topographical data and meteorological and oceanographic conditions at the search site 2. 3D display of optimal search patterns derived through AI analysis 3. Real-time footage and sensor information from drones 4. Probability distribution heat map of the area where rescuers are present

[0021] The pilot's operating instructions are transmitted to the remotely controlled drone (40) via a remote communication system (60). The communication system has a redundant configuration of 5G / 6G terrestrial lines and LEO satellite communications, ensuring communication quality through remote control.

[0022] The multi-sensor device (70) mounted on the remote-controlled drone (40) acquires information from the following sensors. The acquired information is sent to the metaverse generation server (20) along with the results of the threshold determination unit (90) described below, and displayed on the VR display device (10): 1. High-resolution visible light camera: for extracting visual features of human body shape 2. Infrared camera: for detecting thermal image patterns based on body heat 3. Laser sensor: for detecting underwater objects

[0023] The threshold determination unit (90) performs the following AI analysis process on the multi-sensor information acquired by the multi-sensor device (70): Figure 3: Process for identifying areas where rescue victims are present using threshold judgment 1.Sensor information integration processing Each sensor data is processed as an integrated matrix by tensor transformation 2. Feature extraction process The following features are extracted using a convolutional neural network: · Human body shape features (head and limb positional relationships) Life jacket color pattern Body temperature distribution pattern 3. Threshold judgment processing Execute a judgment based on the threshold set for the extracted feature quantity: - Threshold determination of human body shape confidence · Threshold judgment of color pattern compatibility - Threshold judgment of thermal image pattern compatibility - Threshold determination of overall score 4. Identifying the area where people needing rescue exist Integrate detection results that exceed the threshold to identify areas where people needing rescue exist

[0024] The location of the identified person in need of rescue is visualized in the metaverse space with a red highlight, and drone operators around the world are notified in real time.

[0025] The drone safety control unit (100) controls the following three modes to ensure the safe operation of the remote-controlled drone (40): 1. Manual control mode In this mode, a pilot somewhere in the world directly controls the drone through the VR-Metaverse space. Flight direction, speed, and altitude are controlled based on the pilot's instructions. This mode is applicable when communication quality is good and the pilot is actively operating the drone. 2. AI automatic flight mode A mode in which the drone flies autonomously based on the search pattern generated by the optimal search pattern derivation unit (80). This mode is automatically switched to in the following situations: When communication is lost: When communication with the pilot is lost for more than 5 seconds Optimal search execution: When an efficient search pattern needs to be executed - When instructed by the pilot: If the pilot selects automatic flight, the AI ​​will continue to execute the pre-set search pattern during automatic flight and automatically adjust the flight path according to environmental changes (changes in wind direction, wind speed, etc.). 3. Safe Return Mode To ensure the safety of the drone, it will automatically initiate a return flight to its base under the following conditions: When the battery level falls below a set threshold (usually 30%) - When weather conditions deteriorate and safety limits are exceeded - When a system abnormality is detected When an emergency return command is received, the safe return mode automatically selects the safest route, rather than the shortest distance, and flies autonomously to the return base.

[0026] The modes are automatically switched over, with priority given to the Safe Return Mode, AI Auto Flight Mode, and Manual Control Mode, ensuring drone safety in any situation.

[0027] The situation-responsive drone deployment unit (110) automatically calculates the optimal number of drones depending on the search conditions, realizing flexible deployment. 1. Basic deployment according to area size Calculate the basic number of drones based on the size of the search area: Small areas (less than 1 square kilometer): Basic deployment of 2-3 aircraft Medium-sized areas (1-5 square kilometers): Standard deployment of 4-6 aircraft Large area (over 5 square kilometers): Large-scale deployment of 7-10 aircraft 2. Deployment adjustment according to the urgency of the incident Dynamically adjust the number of drones depending on the urgency of the incident: Highest level of urgency: Increase to 1.5 times the basic number of deployments High Urgency: Maintain basic deployment numbers Medium Urgency: Adjusted to 0.8 times the basic deployment number The urgency is determined comprehensively based on factors such as the number of victims, elapsed time, weather conditions, and chances of survival. 3. Special deployment according to search difficulty Additional deployment will be determined according to the search difficulty based on sea conditions and topographical complexity: High difficulty (strong winds, high waves, poor visibility): Special deployment of 2-3 additional aircraft Medium difficulty (moderately bad conditions): Deploy 1-2 additional reinforcements Low difficulty (good conditions): Basic deployment only 4. Adaptive Redeployment: If the situation changes during a search operation, adjust drone deployment in real time: When a person in need of rescue is found: Concentrate deployment around the area where the person was found When weather conditions worsen: Reduced aircraft deployment to ensure safety When expanding the search range: Expand the range by deploying additional units

[0028] The situation-responsive drone deployment unit (110) makes it possible to make the most effective use of limited drone resources and establish a flexible search system that suits the situation.

[0029] The operation of this system is carried out in the following steps: 1. Initial Setup Phase Environmental data of the search site is input and collected by the local data collection unit (30). The situation-responsive drone deployment unit (110) evaluates the area size, urgency, and difficulty of the search, and calculates the optimal number of drones. The drone safety control unit (100) initializes each drone in manual control mode. 2. AI analysis phase The optimal search pattern derivation unit (80) performs AI analysis of the environmental data and generates an optimal search pattern according to the number of drones to be deployed. 3. VR connection phase The VR display device (10) is connected to the Metaverse space. The VR display device (10) is worn by drone operators around the world who are remotely controlling the drones. The VR display device (10) may display integrated information from multiple drones. 4. Remote Control Phase Remotely controlled drones (40) are operated and flown near the search site. At this time, the drone's remote control operator checks the optimal search pattern for each drone in the metaverse space and operates them in manual control mode. 5. AI monitoring phase The threshold determination unit (90) performs real-time AI analysis of the multi-sensor information (obtained by the multi-sensor device (70)) of all drones to monitor the area where rescuers are present. At the same time, the drone safety control unit (100) constantly monitors the remaining battery power and communication status of all drones. 6. Adaptive Redeployment Phase The situation-responsive drone deployment unit (110) adjusts deployment in real time according to changing conditions: When a person in need of rescue is found: Concentrate deployment in the area where the person was found When weather worsens: Adjustment of deployment to ensure safety When the search area changes: Redistribute the number of aircraft according to the area 7. Automatic mode switching phase Perform automatic switching for each drone as needed: When communication is lost: Switches to AI automatic flight mode and continues to execute the pre-set search pattern When the battery is low: Switch to safe return mode and automatically return to base. When communication is restored: Can return to manual control mode 8. Discovery and Notification Phase If any of the drones detects a condition exceeding a threshold, the area where a person in need of rescue is located is identified and notified to the pilot and local crew. 9.Rescue support phase Multiple drones mark the locations of those in need of rescue, assisting local minimum rescue teams in safely rescuing them.

[0030] As a specific example of the present invention, we will explain an assumed example of search activities in a maritime accident approximately 2 NM south of Enoshima. Figure 4 shows the assumed operation scenario.

[0031] Accident summary and dynamic drone deployment The scenario is that a small yacht capsized about 2 NM (approximately 3.7 km) south of Enoshima, with one crew member missing. The following environmental data is input and collected by the on-site data collection unit (30): Wind direction: Northeast 45 degrees, wind speed: 12 meters per second (strong) Wave height: 2.5 meters (medium), Current: Southwest 2 knots (medium) Visibility: 5 km (medium) Estimated time of distress: 2 hours Last sighting location: 35.28 degrees north latitude, 139.48 degrees east longitude (approximately 2 NM south of Enoshima) The situational drone deployment unit (110) makes drone deployment decisions based on an assessment of: Area size: Approximately 10 square kilometers (large area) → Basic deployment: 7 aircraft Urgency: Single distress; Elapsed time: 2 hours (high urgency) → Maintain basic deployment · Search difficulty: Strong winds, medium wave height (medium difficulty) → Deploy one additional aircraft Final deployment decision: A total of eight drones will be deployed

[0032] Optimal search pattern derived through AI analysis The optimal search pattern derivation unit (80) selects Pattern B (grid-type high-density search) based on Table 2, taking into account the conditions of strong wind speed, medium current, and reverse current. The search area is divided into eight sectors to accommodate the deployment of eight drones, and each drone is assigned a sector. Furthermore, a nonlinear fluid simulation is used to set an elliptical priority search area (major axis 3 km, minor axis 2 km) biased southwestward from the last sighting position, with two drones deployed in the high-priority sector and one drone in the medium-priority sector.

[0033] Remote VR operation and drone safety control A pilot in New York puts on a VR display device (10) from his home and connects to the metaverse space generated by the metaverse generation server (20). The drone safety control unit (100) is initialized in manual control mode, and the pilot begins operating the remotely controlled drone (40) from a location approximately 10,000 kilometers away. Due to the time difference, it is daytime in the local area, but it is midnight in the pilot's local time.

[0034] Search operations and safety control The system automatically performs the following safety controls during search operations: If communication becomes temporarily unstable: The system automatically switches to AI automatic flight mode and continues executing the pre-set grid search pattern. If communication is restored: Return to manual control mode. When the battery level drops below the set threshold, the system transitions to preparation for safe return mode.

[0035] Person in need of rescue found The threshold determination unit (90) performs the following determination process on the information acquired from the multi-sensor device: -Human body shape confidence: Comparison with the set threshold of 80% - Life jacket color compatibility: Comparison with the set threshold of 70% - Temperature pattern compatibility: Compare with the set threshold of 75% Overall judgment score: Judgment based on the set threshold of 85% When each threshold is exceeded, the area where rescuers are present is identified at the drift position in the southwest direction, and the area is transferred to the metaverse. The pilot is notified by a red highlight in the space.

[0036] Safe rescue operations The pilot in New York confirms the location off the coast of Enoshima and confirms that the person in question is in need of rescue. The precise location information is transmitted to the local crew and the Japan Coast Guard. The drone continues to hover above the person in question, and when the remaining battery power reaches a set threshold (usually 30%), it automatically returns to base in safe return mode. This allows for a safe rescue operation with minimal on-site personnel.

[0037] In this example, the risk of secondary disasters is completely eliminated by allowing the operator to access the site from anywhere in the world, approximately 10,000 kilometers away. The three-mode control system ensures drone safety, while the optimal search pattern and threshold determination using AI analysis enable more efficient search activities than conventional systematic search methods. This also demonstrates the feasibility of a 24-hour global rescue support system that transcends time zones. [Industrial Applicability]

[0038] In addition to salvage, the system of the present invention can be applied in the following fields: 1. Mountain rescue: Searching for victims in rugged mountainous areas 2. Disaster site survey: Understanding the situation in dangerous areas due to earthquakes, tsunamis, volcanic eruptions, etc. 3. Nuclear Accident Response: Unmanned Survey Activities in Radiation Zones 4. Anti-terrorism measures: Supporting dangerous tasks such as explosive disposal 5. Infrastructure inspection: Inspection of equipment at high altitudes and dangerous locations

[0039] By combining safety assured by remote VR control with efficiency improved by AI analysis, it has great industrial value as an innovative solution for a variety of dangerous work fields. [Explanation of symbols]

[0040] 10 VR display device 20 Metaverse Generation Server 30 Field Data Collection Department 40 Remote Control Drone 50 AI analysis system 60 Telecommunications Systems 70 Multi-sensor device 80 Optimal search pattern derivation part 90 Threshold judgment unit 100 Drone Safety Control Unit 110 Situational Drone Deployment Unit

Claims

1. 1. A search and rescue system, comprising: a VR display device that presents three-dimensional visual information to the operator; A metaverse generation means for generating a three-dimensional virtual environment that reflects the actual search site; a field data collection unit into which environmental data of the search site is input; an optimal search pattern derivation unit that performs AI analysis on the environmental data to derive an optimal search pattern; a plurality of remotely controlled drones that operate in the actual search site based on operation instructions from the pilot; a multi-sensor device mounted on the remote-controlled drone to acquire information about the actual search site; A threshold determination unit is provided that performs AI analysis on the information acquired by the multi-sensor device and identifies a zone where a person in need of rescue exists when a certain threshold is exceeded, The VR display device displays a result of the determination by the threshold determination unit, the multi-sensor device includes a visible light camera, an infrared camera, and a laser sensor; The threshold determination unit is a search and rescue system characterized in that it identifies the area where rescue-requiring persons exist using thresholds set for each of the human body shape certainty, color pattern compatibility, and body temperature pattern compatibility, and a threshold for the overall determination score that combines these.

2. 2. The search and rescue system according to claim 1, wherein the environmental data includes wind direction, wind speed, wave height, current direction, current speed, visibility, estimated time to distress, and last sighting location.

3. A manual control mode in which the pilot directly controls the aircraft; An AI automatic flight mode that flies autonomously based on the optimal search pattern when communication is lost or when an efficient search is performed; It has a safe return mode that automatically returns to the base when the battery level is low or in an emergency, The search and rescue system of claim 1, characterized in that the remote communication system ensures communication quality through remote control by using a redundant configuration of a 5G / 6G terrestrial communication network and a LEO satellite communication network, and further comprises a drone safety control unit that divides the search area into a plurality of sectors and assigns each remotely controlled drone to a different search sector, and that includes a plurality of the remotely controlled drones.

4. The search and rescue system described in claim 1, further comprising a situation-responsive drone deployment unit that determines and deploys the number of the multiple remotely controlled drones according to the area size, urgency, and difficulty of the search.

5. A search and rescue method using the search and rescue system according to any one of claims 1 to 4, comprising: (a) collecting environmental data of the search site by the on-site data collection unit; (b) a step of deriving an optimal search pattern by the optimal search pattern derivation unit performing AI analysis on the environmental data of the search site; (c) displaying the optimum search pattern on the VR display device; (d) transmitting information from the multi-sensor device to the threshold determination unit; (e) A search and rescue method characterized by comprising a step in which the threshold determination unit performs AI analysis of the information obtained by the multi-sensor device and identifies the area in which a person in need of rescue is present when a certain threshold is exceeded.

Images