system

Abstract

We provide the system. [Solution] Means for detecting the occurrence of a disaster, A means of activating the flying object, A means for receiving video data captured by the aforementioned aircraft, A means for analyzing the aforementioned video data to generate a three-dimensional map, A means of formulating a relief plan based on the aforementioned three-dimensional map, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] At the time of a disaster, it is required to quickly and accurately grasp the damage situation and provide information for planning effective rescue activities. However, with conventional methods, there is a problem that information collection takes time and the accuracy of analysis is insufficient, resulting in a delay in effective response.

Means for Solving the Problems

[0005] The present invention provides a system that detects the occurrence of a disaster and quickly activates a plurality of flying objects. The flying objects capture the damage situation and transmit the video data to a server. The server analyzes the received data with an artificial intelligence module and generates a three-dimensional map. By formulating a rescue plan based on this map, it enables quick and effective rescue activities.

[0006] A "disaster" is an event caused by natural phenomena or man-made accidents that inflicts significant damage on a local community.

[0007] "Detection" means that a device or system senses a specific phenomenon or event and recognizes it as information.

[0008] "Flying object" refers to a mechanical device used for movement through the air, including unmanned aerial vehicles such as drones.

[0009] "To start up" refers to bringing a machine or device into an operational state, and more specifically, to turning on the power and beginning its operation.

[0010] "To photograph" is the act of recording visual information using an imaging device such as a camera.

[0011] "Video data" refers to data that stores visual information recorded by cameras or other devices in a digital format.

[0012] "Receiving" refers to the act of receiving data or signals transmitted from an external source and incorporating them into the system.

[0013] "Analyzing" is the process of thoroughly examining complex data and understanding its meaning and structure.

[0014] A "three-dimensional map" is a map that represents physical space in three dimensions, providing visual information including the position and shape of objects.

[0015] A "relief plan" is a strategic plan formulated to efficiently carry out life-saving operations and provide supplies during disasters and emergencies.

[0016] An "artificial intelligence module" is a functional unit of a computer program that automatically performs data analysis and decision-making. [Brief explanation of the drawing]

[0017] [Figure 1]It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, let's explain the terminology used in the following explanation.

[0020] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0021] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] The system of this invention aims to quickly and accurately grasp the extent of damage during a disaster and to plan effective relief activities. This system uses multiple unmanned aerial vehicles (flying objects) to capture images of the disaster area with cameras mounted on them and collect data in real time.

[0039] Upon detecting a disaster, the server immediately connects with the management terminal and automatically activates the designated aircraft. The server then sets the optimal flight route for each aircraft based on information about the predicted damage area and instructs them to collect video data. The aircraft then follow the set route and transmit video data of the damage to the server.

[0040] The server analyzes the received video data using an artificial intelligence module. This analysis generates a three-dimensional map visualizing the extent of the damage. This 3D map can be immediately viewed on the user's device. Users can visually grasp the scale and scope of the damage on their device and make quick decisions.

[0041] The server further combines 3D maps with other geographic information systems to automatically calculate the safest and most effective routes for transporting relief supplies. This relief plan is immediately notified to the user's terminal, allowing the user to carry out efficient relief operations based on it.

[0042] As a concrete example, consider a scenario where an earthquake occurs in a certain region. After detecting the disaster, the server, with the cooperation of the local government, activates multiple aircraft and directs them towards areas believed to be severely affected. The aircraft quickly collect video data, which the server uses to analyze and visualize the extent of the damage. Users can then access this information through their terminals and initiate effective rescue operations based on safe rescue routes. This system can significantly shorten the initial response time to a disaster and contribute to improving the survival rate.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The server detects the occurrence of a disaster from the disaster monitoring system.

[0046] Step 2:

[0047] The server connects to the local government's management terminal and transmits activation commands for multiple aircraft.

[0048] Step 3:

[0049] The server determines the flight path for each aircraft based on pre-configured geographical information of the disaster area.

[0050] Step 4:

[0051] The server transmits the planned flight route to each aircraft and provides flight instructions.

[0052] Step 5:

[0053] The aircraft will fly over the affected area while capturing video data.

[0054] Step 6:

[0055] To transmit video data to the server in real time, the server continuously receives data.

[0056] Step 7:

[0057] The server inputs the received video data into an artificial intelligence module and begins analyzing the extent of the damage.

[0058] Step 8:

[0059] The server uses the analysis results to generate a three-dimensional map and create visualization data.

[0060] Step 9:

[0061] The server sends the visualization data to the user's terminal.

[0062] Step 10:

[0063] Users can view a 3D map through their device to understand the extent of the damage.

[0064] Step 11:

[0065] The server combines three-dimensional maps and geographical information to calculate the optimal route for transporting relief supplies.

[0066] Step 12:

[0067] The server sends the calculation results to the user's terminal and notifies them of a detailed rescue plan.

[0068] Step 13:

[0069] The user begins rescue operations based on the rescue plan displayed on their device.

[0070] (Example 1)

[0071] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0072] When natural disasters occur, it is crucial to quickly and accurately assess the extent of the damage. However, conventional methods often resulted in delays in relief efforts due to the time it took to gather information from affected areas. Furthermore, the inability to collect and analyze data in real time hindered the development of relief plans. Therefore, a more efficient, accurate, and rapid disaster information management system is needed.

[0073] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0074] In this invention, the server includes means for detecting the occurrence of a disaster, means for automatically activating an unmanned aerial vehicle (UAV), and means for setting an optimal flight route for the UAV. This enables the UAV to quickly collect information in the disaster area immediately after a disaster occurs and to carry out efficient and accurate rescue operations.

[0075] "Means for detecting the occurrence of a disaster" refers to a mechanism that uses a sensor system to detect the occurrence of a disaster in real time and notify of the anomaly.

[0076] "Means for automatically activating unmanned aerial vehicles" refers to a method of quickly activating pre-registered unmanned aerial vehicles via remote control when a disaster is detected.

[0077] "Means for setting the optimal flight route for the unmanned aerial vehicle" refers to a process that automatically calculates and instructs the flight path so that the unmanned aerial vehicle can efficiently collect data, taking into account geographical information and disaster information.

[0078] "A means of generating three-dimensional geographic information by analyzing video data with an artificial intelligence processing unit" refers to a technology that uses artificial intelligence technology to process acquired video footage and generate information that visualizes the damage situation in three dimensions.

[0079] "Means for formulating and notifying relief plans" refers to a method of planning effective relief activities based on generated three-dimensional geographic information and immediately communicating them to relevant agencies and personnel.

[0080] In an embodiment of this invention, a system is provided that utilizes unmanned aerial vehicles and artificial intelligence technology to support rapid and accurate information gathering and rescue operations in the event of a disaster.

[0081] The server monitors data in real time from sensors installed to detect disasters (such as seismometers and weather sensors). When an anomaly is detected, the server remotely activates an unmanned aerial vehicle (UAV) and directs it to the designated area. The UAV is equipped with a high-performance camera and a location information system, which allows it to collect high-precision video data of the disaster area.

[0082] The server uses algorithms to calculate the route to the disaster area in optimizing the flight path. This utilizes historical data, current weather, and topographic information. Captured video is streamed to the server in real time, where it proceeds to an image analysis process using a generated AI model. Here, the video data is converted into visual data that represents the damaged area in three dimensions.

[0083] The analysis results are displayed as three-dimensional geographic information on the user's terminal, allowing the user to quickly assess damage and plan rescue routes. Furthermore, the server integrates with a Geographic Information System (GIS) to formulate an optimal rescue plan based on the calculated rescue routes and notify the user.

[0084] As a concrete example, if an earthquake occurs in a certain area, when a sensor detects the tremor, the server immediately activates an unmanned aerial vehicle (UAV) and begins flying towards the affected area. Once the UAV has collected video data and the server has finished analyzing it, the user's terminal will be notified of a three-dimensional damage map along with effective rescue routes. Based on this information, local governments and rescue organizations can take swift action.

[0085] An example of an input prompt for a generating AI model might be: "Create a three-dimensional map of the area severely damaged by the earthquake and calculate the optimal rescue route."

[0086] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0087] Step 1:

[0088] The server receives data from sensors installed to detect disasters. The input is real-time data from the sensors, which the server analyzes to detect abnormal values. Specifically, the server uses a data processing algorithm to check signals that exceed a certain threshold. The output is disaster alert trigger information.

[0089] Step 2:

[0090] Upon confirming the occurrence of a disaster, the server sends a command to activate the unmanned aerial vehicles (UAVs). The input is disaster alert information, and the server accesses the UAV management system to remotely activate the necessary aircraft. Specifically, it sets a flight schedule and sends specific flight preparation instructions to the aircraft. The output is the status information of the activated UAVs.

[0091] Step 3:

[0092] The server calculates and transmits the optimal flight route for the unmanned aerial vehicle (UAV). Inputs include historical disaster data, real-time weather information, and topographic maps. Based on this information, the server performs algorithmic processing to determine an efficient route. Specifically, it simulates flight conditions and automatically generates a path that avoids obstacles. The output is transmitted to the UAV as a flight plan.

[0093] Step 4:

[0094] The unmanned aerial vehicle (UAV) flies along a designated route and collects video data with its onboard camera. The input is the flight plan from the server. During flight, the UAV periodically adjusts the camera angle and takes video. The output is real-time video data.

[0095] Step 5:

[0096] The server receives video data transmitted from unmanned aerial vehicles (UAVs) and analyzes it using a generated AI model. The input is video data from the UAV. The server executes a video analysis program to identify the affected area and generate three-dimensional geographic information. Specifically, it detects and analyzes anomalies in the video and creates a three-dimensional map using the model. The output is a three-dimensional damage map.

[0097] Step 6:

[0098] The server develops a rescue plan based on the generated 3D map and notifies the user's terminal. The input is a 3D damage map. The server uses a geographic information system to calculate a safe and effective rescue route and proposes it to the user. Specifically, it visualizes the calculated route and sends it to the user's terminal for visual confirmation. The output is detailed rescue plan information and a map.

[0099] Step 7:

[0100] The user reviews the rescue plan information received on their terminal and begins rescue operations. Inputs are rescue plans and map information from the server. The user takes swift action in coordination with relevant organizations and teams. Specific actions include arranging necessary equipment and personnel and executing the plan. Outputs are status reports of the deployed rescue operations.

[0101] (Application Example 1)

[0102] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0103] While rapid and efficient relief efforts are required during disasters, it is difficult to immediately grasp the situation in the affected area and formulate an optimal relief plan. Furthermore, flexible plan modifications are necessary to respond immediately to changes in the situation on the ground. The objective of this invention is to provide a system that supports appropriate rescue decisions by accurately visualizing the current situation in the affected area in real time.

[0104] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0105] In this invention, the server includes means for detecting the occurrence of a disaster, means for activating an aircraft, means for receiving video data captured by the aircraft, means for analyzing the video data to generate a three-dimensional map, means for formulating a rescue plan based on the three-dimensional map, and means for displaying the rescue plan on a visual device in real time. This enables rapid and appropriate rescue operations based on an accurate understanding of the extent of the damage.

[0106] "Means for detecting the occurrence of a disaster" refers to devices or systems used to detect the occurrence of natural disasters or man-made emergencies.

[0107] "Means for activating an aircraft" refers to control devices and programs that enable unmanned aerial vehicles or drones to begin autonomous operation.

[0108] "Means for receiving video data" refers to communication devices or programs for acquiring image and video information transmitted from an aircraft.

[0109] "Methods for analyzing video data to generate three-dimensional maps" refer to devices or programs that use image processing technology and artificial intelligence to create three-dimensional maps of real-world terrain and structures based on received video data.

[0110] "Means for formulating relief plans" refer to systems and methodologies for assessing the extent of the damage and determining the optimal routes for supplying materials and scheduling relief activities.

[0111] "Means for displaying rescue plans on visual devices in real time" refers to technology that instantly displays generated rescue plans and 3D maps through smart glasses or displays, providing information to field personnel.

[0112] The system that realizes this invention consists of multiple unmanned aerial vehicles, a central server, and user terminals. When a disaster occurs, the server uses disaster detection means to confirm the emergency situation and automatically activates multiple unmanned aerial vehicles. The activated aerial vehicles collect video data using camera equipment while flying over the disaster area and transmit it to the server in real time.

[0113] The server uses an AI analysis module to process the received video data and generate a map that visualizes the damage in three dimensions. This map integrates multiple video data to clearly show the scale and extent of the damage. This process uses machine learning algorithms to represent the situation in the affected area in three dimensions through the analysis of pixel data.

[0114] Based on the generated 3D maps, the server develops the safest and most effective rescue plan. This plan combines complex geographic information systems to calculate the optimal transport route. The calculated rescue plan and visualized maps are immediately transmitted to the user's terminal and displayed on smart glasses or other visual devices.

[0115] Users can use this information to instantly grasp the situation and make quick decisions regarding relief efforts. For example, in the event of flood damage, unmanned aerial vehicles (UAVs) can take detailed photographs of the overflowing river and the flooding situation in the surrounding area. The server generates a 3D map of the location and extent of the damage, providing visual information to staff waiting at the logistics center to support the safe delivery of supplies.

[0116] Example prompt: "Develop an application to calculate the optimal delivery route to disaster-stricken areas used by a logistics center. This system will analyze video data acquired from unmanned aerial vehicles in real time, and staff will be able to view the data using smart glasses."

[0117] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0118] Step 1:

[0119] The server detects the occurrence of a disaster. Sensors and communication networks that detect natural disasters and man-made emergencies provide real-time information about the occurrence of a disaster. Inputs are data from sensors and communication networks, while outputs are information about the type and scale of the disaster.

[0120] Step 2:

[0121] The server activates the unmanned aerial vehicle (UAV). After confirming the occurrence of a disaster, it automatically issues deployment orders to multiple UAVs. The input is the disaster information obtained in step 1, and the output is the deployment orders to the UAVs.

[0122] Step 3:

[0123] The unmanned aerial vehicle (UAV) collects video data from the disaster area and transmits it to a server. It films the surrounding situation while flying and sends the footage to the server via wireless communication. The input is visual information from the disaster area, and the output is the raw video data transmitted to the server.

[0124] Step 4:

[0125] The server inputs video data into an AI analysis module and analyzes the damage situation in three dimensions. Using the input video data, a machine learning algorithm analyzes the pixel information and visualizes the damage as a three-dimensional map. The output is three-dimensional mapping information of structures and terrain.

[0126] Step 5:

[0127] The server plans logistics routes based on a 3D map. It utilizes geographic information systems and existing traffic data to develop safe and efficient logistics plans. Inputs are 3D maps and traffic data, and output is a relief plan that includes detailed delivery routes.

[0128] Step 6:

[0129] The server transmits rescue plans and 3D maps to the user's terminal. This allows the user to check the situation in real time and carry out appropriate rescue activities. The input is the rescue plan and 3D map, and the output is information displayed on smart glasses or visual devices.

[0130] Step 7:

[0131] Users use smart glasses to make on-site decisions based on plans and map information. Input is visual information displayed on the device, and output is the rescue operations and logistics actions that are carried out.

[0132] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0133] This invention combines a system designed for rapid and accurate assessment of damage during disasters and effective planning of relief operations with an emotion engine to recognize user emotions. The system utilizes multiple unmanned aerial vehicles (flying objects) equipped with cameras to capture images of the disaster area and collect data in real time.

[0134] The server, upon detecting a disaster, coordinates with the management terminal and activates multiple aircraft. Each aircraft is assigned an optimal flight route and heads towards designated locations to photograph the damage. The video data captured by the aircraft is transmitted to the server, where it is analyzed in real time by an artificial intelligence module. This generates a three-dimensional map of the damage, which users can visually view through their terminals.

[0135] Furthermore, the system incorporates an emotion engine that analyzes user reactions to understand the user's psychological state. The server inputs facial expression and voice data collected from the user into the emotion engine to analyze the user's emotional state.

[0136] For example, in the event of an earthquake, the server immediately begins assessing the situation in the affected area and performs emotional analysis on users on-site (rescue teams and local coordinators). If a user is experiencing stress, the server automatically uses this information to provide supplementary information and additional support, assisting with command and control. This enables effective responses without causing psychological pressure.

[0137] Thus, the system of the present invention makes it possible to integrate the assessment of the physical situation and the provision of psychological support in disaster-stricken areas, thereby improving the overall efficiency of disaster response.

[0138] The following describes the processing flow.

[0139] Step 1:

[0140] The server detects the occurrence of a disaster from the disaster monitoring system.

[0141] Step 2:

[0142] The server connects to the local government's management terminal and transmits activation commands for multiple aircraft.

[0143] Step 3:

[0144] The server uses geographical information to determine the optimal flight route for each aircraft and issues flight instructions.

[0145] Step 4:

[0146] The aircraft will fly over the affected area while capturing high-resolution video data.

[0147] Step 5:

[0148] The server receives video data transmitted from the aircraft in real time.

[0149] Step 6:

[0150] The server analyzes the received video data using an artificial intelligence module and generates a three-dimensional map.

[0151] Step 7:

[0152] The server sends the generated 3D map to the user's terminal to visualize the extent of the damage.

[0153] Step 8:

[0154] Users can view a 3D map on their device to understand the scale and extent of the damage.

[0155] Step 9:

[0156] The server activates the emotion engine and analyzes the facial expression and voice data obtained from the user.

[0157] Step 10:

[0158] The server analyzes the user's emotional state and determines their stress and anxiety levels.

[0159] Step 11:

[0160] Based on the analysis results, the server will send additional support information and encouraging messages to the user as needed.

[0161] Step 12:

[0162] Users receive information transmitted through their devices and can efficiently carry out rescue operations while feeling emotionally supported.

[0163] Step 13:

[0164] The server calculates the optimal route for transporting relief supplies based on a 3D map and emotional state data, and notifies the user.

[0165] Step 14:

[0166] Users will follow the relief plan displayed on their devices to begin transporting supplies and rescuing victims.

[0167] (Example 2)

[0168] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0169] While it is crucial to quickly and accurately assess the extent of damage during a disaster and to conduct effective relief operations, conventional systems have limitations in physically assessing the situation, and furthermore, they present challenges such as significant psychological burdens on those working on the ground.

[0170] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0171] In this invention, the server includes means for detecting a disaster, means for starting an aircraft, means for receiving image data captured by the aircraft, means for analyzing the image data to generate a three-dimensional map, means for formulating a support plan based on the three-dimensional map, means for analyzing the emotional state of a user, and means for providing support information based on the emotional state. This makes it possible to integrate the understanding of the physical situation with psychological support and improve the efficiency of disaster response.

[0172] A "disaster" refers to a serious situation in which human lives or property are endangered by natural phenomena or man-made factors.

[0173] An "aircraft" refers to an unmanned aerial vehicle that flies through the air and is equipped with specific equipment or devices to perform a mission.

[0174] "Image data" refers to visual information acquired by a camera mounted on an aircraft.

[0175] A "three-dimensional map" refers to a map generated based on three-dimensional spatial information, which visualizes the terrain, height, and shape of objects.

[0176] A "support plan" refers to a plan or policy designed to facilitate relief and recovery efforts in the event of a disaster.

[0177] "Users" refers to individuals or organizations that use the system to obtain information and carry out disaster response activities.

[0178] "Emotional state" refers to information that describes the user's psychological and emotional condition, including emotions such as stress and anxiety.

[0179] "Support information" refers to information about psychological support and behavioral guidelines provided according to the user's emotional state.

[0180] A "machine learning module" refers to a part of a system that processes information using algorithms that automatically analyze data and find patterns.

[0181] This invention provides a system that enables rapid and accurate assessment of damage during a disaster, and facilitates more effective rescue operations. This system consists of multiple elements, including an aircraft, a server, terminals, and an artificial intelligence module and an emotion engine.

[0182] The server first automatically launches an aircraft upon detecting a disaster. The aircraft is equipped with cameras and acquires image data of the affected area in real time. The acquired image data is sent to the server, which analyzes this data using an artificial intelligence module. In this analysis, object recognition technology is used to evaluate the extent of building damage and changes in terrain, and a 3D map is generated based on this. This 3D map is provided to the user via a terminal, allowing the user to visually confirm the extent of the damage.

[0183] Furthermore, the server collects facial expression and voice data through terminals to understand the emotional state of users working in the field. This data is analyzed by an emotion engine to evaluate the user's psychological state. For example, if the server determines that a user is experiencing high levels of stress, it automatically provides appropriate support information and takes measures to alleviate their psychological burden.

[0184] This system will enable not only the assessment of physical damage but also psychological support, improving the overall efficiency of disaster response. A concrete example is rescue operations during an earthquake. An example of a prompt message might be, "Please describe a system that assesses the damage situation in disaster-stricken areas in real time after an earthquake while also analyzing the psychological state of users on-site."

[0185] Thus, the invention realizes a new disaster response system unlike any other by combining high-performance hardware and advanced software.

[0186] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0187] Step 1:

[0188] The server detects the occurrence of a disaster using information from sensors and external alarm systems. Based on this information, it immediately issues commands to launch multiple aircraft. The aircraft then move to the disaster area according to pre-programmed routes.

[0189] Step 2:

[0190] The aircraft captures image data of the disaster area using its onboard high-resolution camera. During this process, it utilizes the camera's autofocus and exposure adjustment functions to obtain optimal images. The acquired image data is transmitted to a server in real time. Here, the input is the video footage of the disaster area, and the output is the raw data transmitted to the server.

[0191] Step 3:

[0192] The server passes the received image data as input to an artificial intelligence module, which then analyzes the extent of the damage. Object recognition algorithms are used in the data analysis to evaluate the degree of building damage and changes in terrain. A three-dimensional map of the affected area is generated as output of this process.

[0193] Step 4:

[0194] The server transfers the generated 3D map to the user's terminal. The user can then visually confirm the extent of the damage by manipulating this map on their terminal. Specifically, they can rotate the map and zoom in and out.

[0195] Step 5:

[0196] The server collects facial expression and voice data from the user to understand their emotional state. The device's camera and microphone function as input devices, and the data is sent to the server. The input data is analyzed by an emotion engine.

[0197] Step 6:

[0198] The server uses the results of the emotion engine's analysis to generate and provide optimal support information to the user. If the server detects that the user is experiencing stress, it automatically sends information including supplementary explanations and psychological support to reduce the user's burden. This output consists of information and instructions provided to the user.

[0199] (Application Example 2)

[0200] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0201] Existing disaster response information gathering systems are limited to understanding the physical extent of damage, and have the drawback of not adequately supporting the psychological well-being of those working on the ground. Similarly, in security settings, there is a need not only to detect anomalies but also to understand people's psychological states and respond quickly and accurately. Therefore, a system is needed that can alleviate psychological pressure and enable more effective security monitoring.

[0202] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0203] In this invention, the server includes a device for detecting disaster events, a device for starting an aircraft, and a device for receiving image information acquired by the aircraft. This makes it possible to comprehensively assess the extent of damage during a disaster and provide psychological support at security sites.

[0204] A "disaster event" is an event that causes damage, such as a natural phenomenon or an accident.

[0205] An "aircraft" is defined as a vehicle capable of flying through the air and equipped with devices for collecting data such as video and audio.

[0206] "Image information" refers to visual data acquired by aircraft, which shows the situation at disaster sites and monitored areas.

[0207] A "spatial map" is three-dimensional geographical information generated by analyzing image data, allowing for an accurate understanding of the situation on site.

[0208] A "support plan" is a set of action guidelines formulated based on spatial maps and analysis results, aimed at disaster response and security management.

[0209] "Emotional information" refers to data that indicates the psychological state of the person receiving support, and is extracted from elements such as facial expressions and voice.

[0210] "Supplementary information" refers to information provided to the supervisor for additional guidance and support based on the analyzed emotional information.

[0211] A "machine learning module" is a program or system that processes large amounts of data and learns patterns and features from it.

[0212] This invention provides a system that offers rapid and accurate situation assessment and psychological support during disasters and security situations. The system mainly consists of a server, multiple stored aircraft, and user terminals.

[0213] When the server detects a disaster or security event, it quickly launches an aircraft. The aircraft then acquires image data from the scene and transmits it to the server in real time. The server uses machine learning modules and image processing technologies to analyze this image data and generate a spatial map. Specifically, it utilizes Python and OpenCV. Furthermore, it uses Google Cloud's natural language processing API to analyze sentiment information.

[0214] The generated spatial map and emotional information are transmitted to the user's terminal and used to develop support plans. Users operating the terminal receive supplementary information based on the psychological state of the person being monitored in the security environment and take appropriate measures. For example, if anxiety due to congestion is detected during event monitoring in a shopping mall, security guards can receive timely instructions and take the most appropriate action.

[0215] As a concrete example, the prompt message for analyzing surveillance camera footage from a commercial facility is as follows:

[0216] "We analyze footage from surveillance cameras installed within commercial facilities to detect congestion and suspicious activity in real time, and evaluate the psychological state of security guards."

[0217] This invention improves overall efficiency in disaster response and security monitoring.

[0218] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0219] Step 1:

[0220] The server detects disaster events and security events. Inputs are data from external sensors and existing monitoring systems, and outputs are signals to activate aircraft. When data exceeding a specific threshold is detected, the system reacts immediately.

[0221] Step 2:

[0222] The server starts the aircraft and directs it to the designated area. The input is the start signal generated in step 1, and the output is the aircraft's flight plan. The aircraft uses GPS to calculate the optimal path and fly accordingly.

[0223] Step 3:

[0224] The aircraft captures on-site image information and transmits it to the server in real time. The input is video data from the camera mounted on the aircraft, and the output is a data stream to the server. The aircraft adjusts its altitude and angle to capture images under optimal conditions.

[0225] Step 4:

[0226] The server analyzes the received image information and generates a spatial map. The input is the video data received in step 3, and the output is a three-dimensional spatial map. Here, Python and OpenCV are used to process the video data.

[0227] Step 5:

[0228] The server creates a support plan based on the generated spatial map and sends it to the user terminal. The input is a three-dimensional spatial map, and the output is a support plan document. Detailed information on the map is displayed on the user terminal.

[0229] Step 6:

[0230] The server processes emotional information to analyze the user's psychological state. Input is the user's facial expressions and voice data, and output is the emotional assessment result. Emotional analysis is performed using Google Cloud's natural language processing API.

[0231] Step 7:

[0232] The server sends supplementary information to the user terminal based on the analysis results. The input is the emotion assessment result, and the output is specific instructions and support information. The user terminal provides the received information through screen display and audio.

[0233] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0234] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0235] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0236] [Second Embodiment]

[0237] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0238] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0239] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0240] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0241] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0242] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0243] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0244] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0245] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0246] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0247] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0248] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0249] The system of this invention aims to quickly and accurately grasp the extent of damage in the event of a disaster and to plan effective relief activities. This system uses multiple unmanned aerial vehicles (flying objects) and cameras mounted on them to capture images of the disaster area and collect data in real time.

[0250] Upon detecting a disaster, the server immediately connects with the management terminal and automatically activates the designated aircraft. The server then sets the optimal flight route for each aircraft based on information about the predicted damage area and instructs them to collect video data. The aircraft then follow the set route and transmit video data of the damage to the server.

[0251] The server analyzes the received video data using an artificial intelligence module. This analysis generates a three-dimensional map visualizing the extent of the damage. This 3D map can be immediately viewed on the user's device. Users can visually grasp the scale and scope of the damage on their device and make quick decisions.

[0252] The server further combines 3D maps with other geographic information systems to automatically calculate the safest and most effective routes for transporting relief supplies. This relief plan is immediately notified to the user's terminal, allowing the user to carry out efficient relief operations based on it.

[0253] As a concrete example, consider a scenario where an earthquake occurs in a certain region. After detecting the disaster, the server, with the cooperation of the local government, activates multiple aircraft and directs them towards areas believed to be severely affected. The aircraft quickly collect video data, which the server uses to analyze and visualize the extent of the damage. Users can then access this information through their terminals and initiate effective rescue operations based on safe rescue routes. This system can significantly shorten the initial response time to a disaster and contribute to improving the survival rate.

[0254] The following describes the processing flow.

[0255] Step 1:

[0256] The server detects the occurrence of a disaster from the disaster monitoring system.

[0257] Step 2:

[0258] The server connects to the local government's management terminal and transmits activation commands for multiple aircraft.

[0259] Step 3:

[0260] The server determines the flight path for each aircraft based on pre-configured geographical information of the disaster area.

[0261] Step 4:

[0262] The server transmits the planned flight route to each aircraft and provides flight instructions.

[0263] Step 5:

[0264] The aircraft will fly over the affected area while capturing video data.

[0265] Step 6:

[0266] To transmit video data to the server in real time, the server continuously receives data.

[0267] Step 7:

[0268] The server inputs the received video data into an artificial intelligence module and begins analyzing the extent of the damage.

[0269] Step 8:

[0270] The server uses the analysis results to generate a three-dimensional map and create visualization data.

[0271] Step 9:

[0272] The server sends the visualization data to the user's terminal.

[0273] Step 10:

[0274] Users can view a 3D map through their device to understand the extent of the damage.

[0275] Step 11:

[0276] The server combines three-dimensional maps and geographical information to calculate the optimal route for transporting relief supplies.

[0277] Step 12:

[0278] The server sends the calculation results to the user's terminal and notifies a detailed rescue plan.

[0279] Step 13:

[0280] Based on the rescue plan displayed on the terminal, the user starts the rescue activities.

[0281] (Example 1)

[0282] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0283] When a natural disaster occurs, it is required to quickly and accurately grasp the damage situation. However, in the conventional method, it often takes a long time to collect information on the disaster area, and the rescue activities are often delayed. In addition, real-time data collection and analysis cannot be performed, which also hinders the formulation of rescue plans. Therefore, a more efficient, accurate, and rapid disaster information management system is needed.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0285] In this invention, the server includes means for detecting the occurrence of a disaster, means for automatically starting an unmanned aerial vehicle, and means for setting an optimal flight route for the unmanned aerial vehicle. Thereby, it becomes possible to quickly collect information on the disaster area by the unmanned aerial vehicle immediately after the occurrence of the disaster and perform efficient and accurate rescue activities.

[0286] The "means for detecting the occurrence of a disaster" is a mechanism that uses a sensor system to sense the occurrence of a disaster in real time and notify an abnormality.

[0287] The "means for automatically starting an unmanned aerial vehicle" is a method of quickly putting a pre-registered unmanned aerial vehicle into an operating state by remote control when a disaster is detected.

[0288] "Means for setting the optimal flight route for the unmanned aerial vehicle" refers to a process that automatically calculates and instructs the flight path so that the unmanned aerial vehicle can efficiently collect data, taking into account geographical information and disaster information.

[0289] "A means of generating three-dimensional geographic information by analyzing video data with an artificial intelligence processing unit" refers to a technology that uses artificial intelligence technology to process acquired video footage and generate information that visualizes the damage situation in three dimensions.

[0290] "Means for formulating and notifying relief plans" refers to a method of planning effective relief activities based on generated three-dimensional geographic information and immediately communicating them to relevant agencies and personnel.

[0291] In an embodiment of this invention, a system is provided that utilizes unmanned aerial vehicles and artificial intelligence technology to support rapid and accurate information gathering and rescue operations in the event of a disaster.

[0292] The server monitors data in real time from sensors installed to detect disasters (such as seismometers and weather sensors). When an anomaly is detected, the server remotely activates an unmanned aerial vehicle (UAV) and directs it to the designated area. The UAV is equipped with a high-performance camera and a location information system, which allows it to collect high-precision video data of the disaster area.

[0293] The server uses algorithms to calculate the route to the disaster area in optimizing the flight path. This utilizes historical data, current weather, and topographic information. Captured video is streamed to the server in real time, where it proceeds to an image analysis process using a generated AI model. Here, the video data is converted into visual data that represents the damaged area in three dimensions.

[0294] The analysis results are displayed as three-dimensional geographic information on the user's terminal, allowing the user to quickly assess damage and plan rescue routes. Furthermore, the server integrates with a Geographic Information System (GIS) to formulate an optimal rescue plan based on the calculated rescue routes and notify the user.

[0295] As a concrete example, if an earthquake occurs in a certain area, when a sensor detects the tremor, the server immediately activates an unmanned aerial vehicle (UAV) and begins flying towards the affected area. Once the UAV has collected video data and the server has finished analyzing it, the user's terminal will be notified of a three-dimensional damage map along with effective rescue routes. Based on this information, local governments and rescue organizations can take swift action.

[0296] An example of an input prompt for a generating AI model might be: "Create a three-dimensional map of the area severely damaged by the earthquake and calculate the optimal rescue route."

[0297] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0298] Step 1:

[0299] The server receives data from sensors installed to detect disasters. The input is real-time data from the sensors, which the server analyzes to detect abnormal values. Specifically, the server uses a data processing algorithm to check signals that exceed a certain threshold. The output is disaster alert trigger information.

[0300] Step 2:

[0301] When the server confirms the occurrence of a disaster, it sends a startup command for the unmanned aircraft. The input is disaster alert information. The server accesses the management system of the unmanned aircraft and remotely starts the necessary aircraft. As specific operations, it sets the flight schedule and sends specific flight preparation instructions to the aircraft. The output is the status information of the started unmanned aircraft.

[0302] Step 3:

[0303] The server calculates the optimal flight route for the unmanned aircraft and sends it. The input is past disaster data, real-time weather information, topographic maps, etc. The server performs algorithm processing based on this information to determine an efficient route. As specific operations, it simulates flight conditions and automatically generates a route to avoid obstacles. The output is sent to the unmanned aircraft as a flight plan.

[0304] Step 4:

[0305] The unmanned aircraft flies along the designated route and collects video data with the onboard camera. The input is the flight plan from the server. During the flight, as specific operations, the unmanned aircraft periodically takes videos while adjusting the angle of the camera. The output is real-time video data.

[0306] Step 5:

[0307] The server receives the video data sent from the unmanned aircraft and analyzes it using the generated AI model. The input is the video data from the unmanned aircraft. The server executes a video analysis program to identify the affected area and generate three-dimensional geographical information. As specific operations, it detects and analyzes abnormalities in the video and creates a three-dimensional map using the model. The output is a three-dimensional damage map.

[0308] Step 6:

[0309] The server develops a rescue plan based on the generated 3D map and notifies the user's terminal. The input is a 3D damage map. The server uses a geographic information system to calculate a safe and effective rescue route and proposes it to the user. Specifically, it visualizes the calculated route and sends it to the user's terminal for visual confirmation. The output is detailed rescue plan information and a map.

[0310] Step 7:

[0311] The user reviews the rescue plan information received on their terminal and begins rescue operations. Inputs are rescue plans and map information from the server. The user takes swift action in coordination with relevant organizations and teams. Specific actions include arranging necessary equipment and personnel and executing the plan. Outputs are status reports of the deployed rescue operations.

[0312] (Application Example 1)

[0313] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0314] While rapid and efficient relief efforts are required during disasters, it is difficult to immediately grasp the situation in the affected area and formulate an optimal relief plan. Furthermore, flexible plan modifications are necessary to respond immediately to changes in the situation on the ground. The objective of this invention is to provide a system that supports appropriate rescue decisions by accurately visualizing the current situation in the affected area in real time.

[0315] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0316] In this invention, the server includes means for detecting the occurrence of a disaster, means for activating an aircraft, means for receiving video data captured by the aircraft, means for analyzing the video data to generate a three-dimensional map, means for formulating a rescue plan based on the three-dimensional map, and means for displaying the rescue plan on a visual device in real time. This enables rapid and appropriate rescue operations based on an accurate understanding of the extent of the damage.

[0317] "Means for detecting the occurrence of a disaster" refers to devices or systems used to detect the occurrence of natural disasters or man-made emergencies.

[0318] "Means for activating an aircraft" refers to control devices and programs that enable unmanned aerial vehicles or drones to begin autonomous operation.

[0319] "Means for receiving video data" refers to communication devices or programs for acquiring image and video information transmitted from an aircraft.

[0320] "Methods for analyzing video data to generate three-dimensional maps" refer to devices or programs that use image processing technology and artificial intelligence to create three-dimensional maps of real-world terrain and structures based on received video data.

[0321] "Means for formulating relief plans" refer to systems and methodologies for assessing the extent of the damage and determining the optimal routes for supplying materials and scheduling relief activities.

[0322] "Means for displaying rescue plans on visual devices in real time" refers to technology that instantly displays generated rescue plans and 3D maps through smart glasses or displays, providing information to field personnel.

[0323] The system that realizes this invention consists of multiple unmanned aerial vehicles, a central server, and user terminals. When a disaster occurs, the server uses disaster detection means to confirm the emergency situation and automatically activates multiple unmanned aerial vehicles. The activated aerial vehicles collect video data using camera equipment while flying over the disaster area and transmit it to the server in real time.

[0324] The server uses an AI analysis module to process the received video data and generate a map that visualizes the damage in three dimensions. This map integrates multiple video data to clearly show the scale and extent of the damage. This process uses machine learning algorithms to represent the situation in the affected area in three dimensions through the analysis of pixel data.

[0325] Based on the generated 3D maps, the server develops the safest and most effective rescue plan. This plan combines complex geographic information systems to calculate the optimal transport route. The calculated rescue plan and visualized maps are immediately transmitted to the user's terminal and displayed on smart glasses or other visual devices.

[0326] Users can use this information to instantly grasp the situation and make quick decisions regarding relief efforts. For example, in the event of flood damage, unmanned aerial vehicles (UAVs) can take detailed photographs of the overflowing river and the flooding situation in the surrounding area. The server generates a 3D map of the location and extent of the damage, providing visual information to staff waiting at the logistics center to support the safe delivery of supplies.

[0327] Example prompt: "Develop an application to calculate the optimal delivery route to disaster-stricken areas used by a logistics center. This system will analyze video data acquired from unmanned aerial vehicles in real time, and staff will be able to view the data using smart glasses."

[0328] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0329] Step 1:

[0330] The server detects the occurrence of a disaster. Sensors and communication networks that detect natural disasters and man-made emergencies provide real-time information about the occurrence of a disaster. Inputs are data from sensors and communication networks, while outputs are information about the type and scale of the disaster.

[0331] Step 2:

[0332] The server activates the unmanned aerial vehicle (UAV). After confirming the occurrence of a disaster, it automatically issues deployment orders to multiple UAVs. The input is the disaster information obtained in step 1, and the output is the deployment orders to the UAVs.

[0333] Step 3:

[0334] The unmanned aerial vehicle (UAV) collects video data from the disaster area and transmits it to a server. It films the surrounding situation while flying and sends the footage to the server via wireless communication. The input is visual information from the disaster area, and the output is the raw video data transmitted to the server.

[0335] Step 4:

[0336] The server inputs video data into an AI analysis module and analyzes the damage situation in three dimensions. Using the input video data, a machine learning algorithm analyzes the pixel information and visualizes the damage as a three-dimensional map. The output is three-dimensional mapping information of structures and terrain.

[0337] Step 5:

[0338] The server plans logistics routes based on a 3D map. It utilizes geographic information systems and existing traffic data to develop safe and efficient logistics plans. Inputs are 3D maps and traffic data, and output is a relief plan that includes detailed delivery routes.

[0339] Step 6:

[0340] The server transmits rescue plans and 3D maps to the user's terminal. This allows the user to check the situation in real time and carry out appropriate rescue activities. The input is the rescue plan and 3D map, and the output is information displayed on smart glasses or visual devices.

[0341] Step 7:

[0342] Users use smart glasses to make on-site decisions based on plans and map information. Input is visual information displayed on the device, and output is the rescue operations and logistics actions that are carried out.

[0343] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0344] This invention combines a system designed for rapid and accurate assessment of damage during disasters and effective planning of relief operations with an emotion engine to recognize user emotions. The system utilizes multiple unmanned aerial vehicles (flying objects) equipped with cameras to capture images of the disaster area and collect data in real time.

[0345] The server, upon detecting a disaster, coordinates with the management terminal and activates multiple aircraft. Each aircraft is assigned an optimal flight route and heads towards designated locations to photograph the damage. The video data captured by the aircraft is transmitted to the server, where it is analyzed in real time by an artificial intelligence module. This generates a three-dimensional map of the damage, which users can visually view through their terminals.

[0346] Furthermore, the system incorporates an emotion engine that analyzes user reactions to understand the user's psychological state. The server inputs facial expression and voice data collected from the user into the emotion engine to analyze the user's emotional state.

[0347] For example, in the event of an earthquake, the server immediately begins assessing the situation in the affected area and performs emotional analysis on users on-site (rescue teams and local coordinators). If a user is experiencing stress, the server automatically uses this information to provide supplementary information and additional support, assisting with command and control. This enables effective responses without causing psychological pressure.

[0348] Thus, the system of the present invention makes it possible to integrate the assessment of the physical situation and the provision of psychological support in disaster-stricken areas, thereby improving the overall efficiency of disaster response.

[0349] The following describes the processing flow.

[0350] Step 1:

[0351] The server detects the occurrence of a disaster from the disaster monitoring system.

[0352] Step 2:

[0353] The server connects to the local government's management terminal and transmits activation commands for multiple aircraft.

[0354] Step 3:

[0355] The server uses geographical information to determine the optimal flight route for each aircraft and issues flight instructions.

[0356] Step 4:

[0357] The aircraft will fly over the affected area while capturing high-resolution video data.

[0358] Step 5:

[0359] The server receives video data transmitted from the aircraft in real time.

[0360] Step 6:

[0361] The server analyzes the received video data using an artificial intelligence module and generates a three-dimensional map.

[0362] Step 7:

[0363] The server sends the generated 3D map to the user's terminal to visualize the extent of the damage.

[0364] Step 8:

[0365] Users can view a 3D map on their device to understand the scale and extent of the damage.

[0366] Step 9:

[0367] The server activates the emotion engine and analyzes the facial expression and voice data obtained from the user.

[0368] Step 10:

[0369] The server analyzes the user's emotional state and determines their stress and anxiety levels.

[0370] Step 11:

[0371] Based on the analysis results, the server will send additional support information and encouraging messages to the user as needed.

[0372] Step 12:

[0373] Users receive information transmitted through their devices and can efficiently carry out rescue operations while feeling emotionally supported.

[0374] Step 13:

[0375] The server calculates the optimal route for transporting relief supplies based on a 3D map and emotional state data, and notifies the user.

[0376] Step 14:

[0377] Users will follow the relief plan displayed on their devices to begin transporting supplies and rescuing victims.

[0378] (Example 2)

[0379] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0380] While it is crucial to quickly and accurately assess the extent of damage during a disaster and to conduct effective relief operations, conventional systems have limitations in physically assessing the situation, and furthermore, they present challenges such as significant psychological burdens on those working on the ground.

[0381] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0382] In this invention, the server includes means for detecting a disaster, means for starting an aircraft, means for receiving image data captured by the aircraft, means for analyzing the image data to generate a three-dimensional map, means for formulating a support plan based on the three-dimensional map, means for analyzing the emotional state of a user, and means for providing support information based on the emotional state. This makes it possible to integrate the understanding of the physical situation with psychological support and improve the efficiency of disaster response.

[0383] A "disaster" refers to a serious situation in which human lives or property are endangered by natural phenomena or man-made factors.

[0384] An "aircraft" refers to an unmanned aerial vehicle that flies through the air and is capable of carrying specific equipment or devices to perform a mission.

[0385] "Image data" refers to visual information acquired by a camera mounted on an aircraft.

[0386] A "three-dimensional map" refers to a map generated based on three-dimensional spatial information, which visualizes the terrain, height, and shape of objects.

[0387] A "support plan" refers to a plan or policy designed to facilitate relief and recovery efforts in the event of a disaster.

[0388] "Users" refers to individuals or organizations that use the system to obtain information and carry out disaster response activities.

[0389] "Emotional state" refers to information that describes the user's psychological and emotional condition, including emotions such as stress and anxiety.

[0390] "Support information" refers to information about psychological support and behavioral guidelines provided according to the user's emotional state.

[0391] A "machine learning module" refers to a part of a system that processes information using algorithms that automatically analyze data and find patterns.

[0392] This invention provides a system that enables rapid and accurate assessment of damage during a disaster, and facilitates more effective rescue operations. This system consists of multiple elements, including an aircraft, a server, terminals, and an artificial intelligence module and an emotion engine.

[0393] The server first automatically launches an aircraft upon detecting a disaster. The aircraft is equipped with cameras and acquires image data of the affected area in real time. The acquired image data is sent to the server, which analyzes this data using an artificial intelligence module. In this analysis, object recognition technology is used to evaluate the extent of building damage and changes in terrain, and a 3D map is generated based on this. This 3D map is provided to the user via a terminal, allowing the user to visually confirm the extent of the damage.

[0394] Furthermore, the server collects facial expression and voice data through terminals to understand the emotional state of users working in the field. This data is analyzed by an emotion engine to evaluate the user's psychological state. For example, if the server determines that a user is experiencing high levels of stress, it automatically provides appropriate support information and takes measures to alleviate their psychological burden.

[0395] This system will enable not only the assessment of physical damage but also psychological support, improving the overall efficiency of disaster response. A concrete example is rescue operations during an earthquake. An example of a prompt message might be, "Please describe a system that assesses the damage situation in disaster-stricken areas in real time after an earthquake while also analyzing the psychological state of users on-site."

[0396] Thus, the invention realizes a new disaster response system unlike any other by combining high-performance hardware and advanced software.

[0397] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0398] Step 1:

[0399] The server detects the occurrence of a disaster using information from sensors and external alarm systems. Based on this information, it immediately issues commands to launch multiple aircraft. The aircraft then move to the disaster area according to pre-programmed routes.

[0400] Step 2:

[0401] The aircraft captures image data of the disaster area using its onboard high-resolution camera. During this process, it utilizes the camera's autofocus and exposure adjustment functions to obtain optimal images. The acquired image data is transmitted to a server in real time. Here, the input is the video footage of the disaster area, and the output is the raw data transmitted to the server.

[0402] Step 3:

[0403] The server passes the received image data as input to an artificial intelligence module, which then analyzes the extent of the damage. Object recognition algorithms are used in the data analysis to evaluate the degree of building damage and changes in terrain. A three-dimensional map of the affected area is generated as output of this process.

[0404] Step 4:

[0405] The server transfers the generated 3D map to the user's terminal. The user can then visually confirm the extent of the damage by manipulating this map on their terminal. Specifically, they can rotate the map and zoom in and out.

[0406] Step 5:

[0407] The server collects facial expression and voice data from the user to understand their emotional state. The device's camera and microphone function as input devices, and the data is sent to the server. The input data is analyzed by an emotion engine.

[0408] Step 6:

[0409] The server uses the results of the emotion engine's analysis to generate and provide optimal support information to the user. If the server detects that the user is experiencing stress, it automatically sends information including supplementary explanations and psychological support to reduce the user's burden. This output consists of information and instructions provided to the user.

[0410] (Application Example 2)

[0411] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0412] Existing disaster response information gathering systems are limited to understanding the physical extent of damage, and have the drawback of not adequately supporting the psychological well-being of those working on the ground. Similarly, in security settings, there is a need not only to detect anomalies but also to understand people's psychological states and respond quickly and accurately. Therefore, a system is needed that can alleviate psychological pressure and enable more effective security monitoring.

[0413] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0414] In this invention, the server includes a device for detecting disaster events, a device for starting an aircraft, and a device for receiving image information acquired by the aircraft. This makes it possible to comprehensively assess the extent of damage during a disaster and provide psychological support at security sites.

[0415] A "disaster event" is an event that causes damage, such as a natural phenomenon or an accident.

[0416] An "aircraft" is defined as a vehicle capable of flying through the air and equipped with devices for collecting data such as video and audio.

[0417] "Image information" refers to visual data acquired by aircraft, which shows the situation at disaster sites and monitored areas.

[0418] A "spatial map" is three-dimensional geographical information generated by analyzing image data, allowing for an accurate understanding of the situation on site.

[0419] A "support plan" is a set of action guidelines formulated based on spatial maps and analysis results, aimed at disaster response and security management.

[0420] "Emotional information" refers to data that indicates the psychological state of the person receiving support, and is extracted from elements such as facial expressions and voice.

[0421] "Supplementary information" refers to information provided to the supervisor for additional guidance and support based on the analyzed emotional information.

[0422] A "machine learning module" is a program or system that processes large amounts of data and learns patterns and features from it.

[0423] This invention provides a system that offers rapid and accurate situation assessment and psychological support during disasters and security situations. The system mainly consists of a server, multiple stored aircraft, and user terminals.

[0424] When the server detects a disaster or security event, it quickly launches an aircraft. The aircraft then acquires image data from the scene and transmits it to the server in real time. The server uses machine learning modules and image processing techniques to analyze this image data and generate a spatial map. Specifically, it utilizes Python and OpenCV. Furthermore, it uses Google Cloud's natural language processing API to analyze sentiment information.

[0425] The generated spatial map and emotional information are transmitted to the user's terminal and used to develop support plans. Users operating the terminal receive supplementary information based on the psychological state of the person being monitored in the security environment and take appropriate measures. For example, if anxiety due to congestion is detected during event monitoring in a shopping mall, security guards can receive timely instructions and take the most appropriate action.

[0426] As a concrete example, the prompt message for analyzing surveillance camera footage from a commercial facility is as follows:

[0427] "We analyze footage from surveillance cameras installed within commercial facilities to detect congestion and suspicious activity in real time, and evaluate the psychological state of security guards."

[0428] This invention improves overall efficiency in disaster response and security monitoring.

[0429] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0430] Step 1:

[0431] The server detects disaster events and security events. Inputs are data from external sensors and existing monitoring systems, and outputs are signals to activate aircraft. When data exceeding a specific threshold is detected, the system reacts immediately.

[0432] Step 2:

[0433] The server starts the aircraft and directs it to the designated area. The input is the start signal generated in step 1, and the output is the aircraft's flight plan. The aircraft uses GPS to calculate the optimal path and fly accordingly.

[0434] Step 3:

[0435] The aircraft captures on-site image information and transmits it to the server in real time. The input is video data from the camera mounted on the aircraft, and the output is a data stream to the server. The aircraft adjusts its altitude and angle to capture images under optimal conditions.

[0436] Step 4:

[0437] The server analyzes the received image information and generates a spatial map. The input is the video data received in step 3, and the output is a three-dimensional spatial map. Here, Python and OpenCV are used to process the video data.

[0438] Step 5:

[0439] The server creates a support plan based on the generated spatial map and sends it to the user terminal. The input is a three-dimensional spatial map, and the output is a support plan document. Detailed information on the map is displayed on the user terminal.

[0440] Step 6:

[0441] The server processes emotional information to analyze the user's psychological state. Input is the user's facial expressions and voice data, and output is the emotional assessment result. Emotional analysis is performed using Google Cloud's natural language processing API.

[0442] Step 7:

[0443] The server sends supplementary information to the user terminal based on the analysis results. The input is the emotion assessment result, and the output is specific instructions and support information. The user terminal provides the received information through screen display and audio.

[0444] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0445] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0446] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0447] [Third Embodiment]

[0448] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0449] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0450] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0451] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0452] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0453] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0454] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0455] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0456] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0457] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0458] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0459] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0460] The system of this invention aims to quickly and accurately grasp the extent of damage in the event of a disaster and to plan effective relief activities. This system uses multiple unmanned aerial vehicles (flying objects) and cameras mounted on them to capture images of the disaster area and collect data in real time.

[0461] Upon detecting a disaster, the server immediately connects with the management terminal and automatically activates the designated aircraft. The server then sets the optimal flight route for each aircraft based on information about the predicted damage area and instructs them to collect video data. The aircraft then follow the set route and transmit video data of the damage to the server.

[0462] The server analyzes the received video data using an artificial intelligence module. This analysis generates a three-dimensional map visualizing the extent of the damage. This 3D map can be immediately viewed on the user's device. Users can visually grasp the scale and scope of the damage on their device and make quick decisions.

[0463] The server further combines 3D maps with other geographic information systems to automatically calculate the safest and most effective routes for transporting relief supplies. This relief plan is immediately notified to the user's terminal, allowing the user to carry out efficient relief operations based on it.

[0464] As a concrete example, consider a scenario where an earthquake occurs in a certain region. After detecting the disaster, the server, with the cooperation of the local government, activates multiple aircraft and directs them towards areas believed to be severely affected. The aircraft quickly collect video data, which the server uses to analyze and visualize the extent of the damage. Users can then access this information through their terminals and initiate effective rescue operations based on safe rescue routes. This system can significantly shorten the initial response time to a disaster and contribute to improving the survival rate.

[0465] The following describes the processing flow.

[0466] Step 1:

[0467] The server detects the occurrence of a disaster from the disaster monitoring system.

[0468] Step 2:

[0469] The server connects to the local government's management terminal and transmits activation commands for multiple aircraft.

[0470] Step 3:

[0471] The server determines the flight path for each aircraft based on pre-configured geographical information of the disaster area.

[0472] Step 4:

[0473] The server transmits the planned flight route to each aircraft and provides flight instructions.

[0474] Step 5:

[0475] The aircraft will fly over the affected area while capturing video data.

[0476] Step 6:

[0477] To transmit video data to the server in real time, the server continuously receives data.

[0478] Step 7:

[0479] The server inputs the received video data into an artificial intelligence module and begins analyzing the extent of the damage.

[0480] Step 8:

[0481] The server uses the analysis results to generate a three-dimensional map and create visualization data.

[0482] Step 9:

[0483] The server sends the visualization data to the user's terminal.

[0484] Step 10:

[0485] Users can view a 3D map through their device to understand the extent of the damage.

[0486] Step 11:

[0487] The server combines three-dimensional maps and geographical information to calculate the optimal route for transporting relief supplies.

[0488] Step 12:

[0489] The server sends the calculation results to the user's terminal and notifies them of a detailed rescue plan.

[0490] Step 13:

[0491] The user begins rescue operations based on the rescue plan displayed on their device.

[0492] (Example 1)

[0493] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0494] When natural disasters occur, it is crucial to quickly and accurately assess the extent of the damage. However, conventional methods often resulted in delays in relief efforts due to the time it took to gather information from affected areas. Furthermore, the inability to collect and analyze data in real time hindered the development of relief plans. Therefore, a more efficient, accurate, and rapid disaster information management system is needed.

[0495] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0496] In this invention, the server includes means for detecting the occurrence of a disaster, means for automatically activating an unmanned aerial vehicle (UAV), and means for setting an optimal flight route for the UAV. This enables the UAV to quickly collect information in the disaster area immediately after a disaster occurs and to carry out efficient and accurate rescue operations.

[0497] "Means for detecting the occurrence of a disaster" refers to a mechanism that uses a sensor system to detect the occurrence of a disaster in real time and notify of the anomaly.

[0498] "Means for automatically activating unmanned aerial vehicles" refers to a method of quickly activating pre-registered unmanned aerial vehicles via remote control when a disaster is detected.

[0499] "Means for setting the optimal flight route for the unmanned aerial vehicle" refers to a process that automatically calculates and instructs the flight path so that the unmanned aerial vehicle can efficiently collect data, taking into account geographical information and disaster information.

[0500] "A means of generating three-dimensional geographic information by analyzing video data with an artificial intelligence processing unit" refers to a technology that uses artificial intelligence technology to process acquired video footage and generate information that visualizes the damage situation in three dimensions.

[0501] "Means for formulating and notifying relief plans" refers to a method of planning effective relief activities based on generated three-dimensional geographic information and immediately communicating them to relevant agencies and personnel.

[0502] In an embodiment of this invention, a system is provided that utilizes unmanned aerial vehicles and artificial intelligence technology to support rapid and accurate information gathering and rescue operations in the event of a disaster.

[0503] The server monitors data in real time from sensors installed to detect disasters (such as seismometers and weather sensors). When an anomaly is detected, the server remotely activates an unmanned aerial vehicle (UAV) and directs it to the designated area. The UAV is equipped with a high-performance camera and a location information system, which allows it to collect high-precision video data of the disaster area.

[0504] The server uses algorithms to calculate the route to the disaster area in optimizing the flight path. This utilizes historical data, current weather, and topographic information. Captured video is streamed to the server in real time, where it proceeds to an image analysis process using a generated AI model. Here, the video data is converted into visual data that represents the damaged area in three dimensions.

[0505] The analysis results are displayed as three-dimensional geographic information on the user's terminal, allowing the user to quickly assess damage and plan rescue routes. Furthermore, the server integrates with a Geographic Information System (GIS) to formulate an optimal rescue plan based on the calculated rescue routes and notify the user.

[0506] As a concrete example, if an earthquake occurs in a certain area, when a sensor detects the tremor, the server immediately activates an unmanned aerial vehicle (UAV) and begins flying towards the affected area. Once the UAV has collected video data and the server has finished analyzing it, the user's terminal will be notified of a three-dimensional damage map along with effective rescue routes. Based on this information, local governments and rescue organizations can take swift action.

[0507] An example of an input prompt for a generating AI model might be: "Create a three-dimensional map of the area severely damaged by the earthquake and calculate the optimal rescue route."

[0508] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0509] Step 1:

[0510] The server receives data from sensors installed to detect disasters. The input is real-time data from the sensors, which the server analyzes to detect abnormal values. Specifically, the server uses a data processing algorithm to check signals that exceed a certain threshold. The output is disaster alert trigger information.

[0511] Step 2:

[0512] Upon confirming the occurrence of a disaster, the server sends a command to activate the unmanned aerial vehicles (UAVs). The input is disaster alert information, and the server accesses the UAV management system to remotely activate the necessary aircraft. Specifically, it sets a flight schedule and sends specific flight preparation instructions to the aircraft. The output is the status information of the activated UAVs.

[0513] Step 3:

[0514] The server calculates and transmits the optimal flight route for the unmanned aerial vehicle (UAV). Inputs include historical disaster data, real-time weather information, and topographic maps. Based on this information, the server performs algorithmic processing to determine an efficient route. Specifically, it simulates flight conditions and automatically generates a path that avoids obstacles. The output is transmitted to the UAV as a flight plan.

[0515] Step 4:

[0516] The unmanned aerial vehicle (UAV) flies along a designated route and collects video data with its onboard camera. The input is the flight plan from the server. As a specific action during flight, the UAV periodically takes video while adjusting the camera angle. The output is real-time video data.

[0517] Step 5:

[0518] The server receives video data transmitted from unmanned aerial vehicles (UAVs) and analyzes it using a generated AI model. The input is video data from the UAV. The server executes a video analysis program to identify the affected area and generate three-dimensional geographic information. Specifically, it detects and analyzes anomalies in the video and creates a three-dimensional map using the model. The output is a three-dimensional damage map.

[0519] Step 6:

[0520] The server develops a rescue plan based on the generated 3D map and notifies the user's terminal. The input is a 3D damage map. The server uses a geographic information system to calculate a safe and effective rescue route and proposes it to the user. Specifically, it visualizes the calculated route and sends it to the user's terminal for visual confirmation. The output is detailed rescue plan information and a map.

[0521] Step 7:

[0522] The user reviews the rescue plan information received on their terminal and begins rescue operations. Inputs are rescue plans and map information from the server. The user takes swift action in coordination with relevant organizations and teams. Specific actions include arranging necessary equipment and personnel and executing the plan. Outputs are status reports of the deployed rescue operations.

[0523] (Application Example 1)

[0524] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0525] While rapid and efficient relief efforts are required during disasters, it is difficult to immediately grasp the situation in the affected area and formulate an optimal relief plan. Furthermore, flexible plan modifications are necessary to respond immediately to changes in the situation on the ground. The objective of this invention is to provide a system that supports appropriate rescue decisions by accurately visualizing the current situation in the affected area in real time.

[0526] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0527] In this invention, the server includes means for detecting the occurrence of a disaster, means for activating an aircraft, means for receiving video data captured by the aircraft, means for analyzing the video data to generate a three-dimensional map, means for formulating a rescue plan based on the three-dimensional map, and means for displaying the rescue plan on a visual device in real time. This enables rapid and appropriate rescue operations based on an accurate understanding of the extent of the damage.

[0528] "Means for detecting the occurrence of a disaster" refers to devices or systems used to detect the occurrence of natural disasters or man-made emergencies.

[0529] "Means for activating an aircraft" refers to control devices and programs that enable unmanned aerial vehicles or drones to begin autonomous operation.

[0530] "Means for receiving video data" refers to communication devices or programs for acquiring image and video information transmitted from an aircraft.

[0531] "Methods for analyzing video data to generate three-dimensional maps" refer to devices or programs that use image processing technology and artificial intelligence to create three-dimensional maps of real-world terrain and structures based on received video data.

[0532] "Means for formulating relief plans" refer to systems and methodologies for assessing the extent of the damage and determining the optimal routes for supplying materials and scheduling relief activities.

[0533] "Means for displaying rescue plans on visual devices in real time" refers to technology that instantly displays generated rescue plans and 3D maps through smart glasses or displays, providing information to field personnel.

[0534] The system that realizes this invention consists of multiple unmanned aerial vehicles, a central server, and user terminals. When a disaster occurs, the server uses disaster detection means to confirm the emergency situation and automatically activates multiple unmanned aerial vehicles. The activated aerial vehicles collect video data using camera equipment while flying over the disaster area and transmit it to the server in real time.

[0535] The server uses an AI analysis module to process the received video data and generate a map that visualizes the damage in three dimensions. This map integrates multiple video data to clearly show the scale and extent of the damage. This process uses machine learning algorithms to represent the situation in the affected area in three dimensions through the analysis of pixel data.

[0536] Based on the generated 3D maps, the server develops the safest and most effective rescue plan. This plan combines complex geographic information systems to calculate the optimal transport route. The calculated rescue plan and visualized maps are immediately transmitted to the user's terminal and displayed on smart glasses or other visual devices.

[0537] Users can use this information to instantly grasp the situation and make quick decisions regarding relief efforts. For example, in the event of flood damage, unmanned aerial vehicles (UAVs) can take detailed photographs of the overflowing river and the flooding situation in the surrounding area. The server generates a 3D map of the location and extent of the damage, providing visual information to staff waiting at the logistics center to support the safe delivery of supplies.

[0538] Example prompt: "Develop an application to calculate the optimal delivery route to disaster-stricken areas used by a logistics center. This system will analyze video data acquired from unmanned aerial vehicles in real time, and staff will be able to view the data using smart glasses."

[0539] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0540] Step 1:

[0541] The server detects the occurrence of a disaster. Sensors and communication networks that detect natural disasters and man-made emergencies provide real-time information about the occurrence of a disaster. Inputs are data from sensors and communication networks, while outputs are information about the type and scale of the disaster.

[0542] Step 2:

[0543] The server activates the unmanned aerial vehicle (UAV). After confirming the occurrence of a disaster, it automatically issues deployment orders to multiple UAVs. The input is the disaster information obtained in step 1, and the output is the deployment orders to the UAVs.

[0544] Step 3:

[0545] The unmanned aerial vehicle (UAV) collects video data from the disaster area and transmits it to a server. It films the surrounding situation while flying and sends the footage to the server via wireless communication. The input is visual information from the disaster area, and the output is the raw video data transmitted to the server.

[0546] Step 4:

[0547] The server inputs video data into an AI analysis module and analyzes the damage situation in three dimensions. Using the input video data, a machine learning algorithm analyzes the pixel information and visualizes the damage as a three-dimensional map. The output is three-dimensional mapping information of structures and terrain.

[0548] Step 5:

[0549] The server plans logistics routes based on a 3D map. It utilizes geographic information systems and existing traffic data to develop safe and efficient logistics plans. Inputs are 3D maps and traffic data, and output is a relief plan that includes detailed delivery routes.

[0550] Step 6:

[0551] The server transmits rescue plans and 3D maps to the user's terminal. This allows the user to check the situation in real time and carry out appropriate rescue activities. The input is the rescue plan and 3D map, and the output is information displayed on smart glasses or visual devices.

[0552] Step 7:

[0553] Users use smart glasses to make on-site decisions based on plans and map information. Input is visual information displayed on the device, and output is the rescue operations and logistics actions that are carried out.

[0554] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0555] This invention combines a system designed for rapid and accurate assessment of damage during disasters and effective planning of relief operations with an emotion engine to recognize user emotions. The system utilizes multiple unmanned aerial vehicles (flying objects) equipped with cameras to capture images of the disaster area and collect data in real time.

[0556] The server, upon detecting a disaster, coordinates with the management terminal and activates multiple aircraft. Each aircraft is assigned an optimal flight route and heads towards designated locations to photograph the damage. The video data captured by the aircraft is transmitted to the server, where it is analyzed in real time by an artificial intelligence module. This generates a three-dimensional map of the damage, which users can visually view through their terminals.

[0557] Furthermore, the system incorporates an emotion engine that analyzes user reactions to understand the user's psychological state. The server inputs facial expression and voice data collected from the user into the emotion engine to analyze the user's emotional state.

[0558] For example, in the event of an earthquake, the server immediately begins assessing the situation in the affected area and performs emotional analysis on users on-site (rescue teams and local coordinators). If a user is experiencing stress, the server automatically uses this information to provide supplementary information and additional support, assisting with command and control. This enables effective responses without causing psychological pressure.

[0559] Thus, the system of the present invention makes it possible to integrate the assessment of the physical situation and the provision of psychological support in disaster-stricken areas, thereby improving the overall efficiency of disaster response.

[0560] The following describes the processing flow.

[0561] Step 1:

[0562] The server detects the occurrence of a disaster from the disaster monitoring system.

[0563] Step 2:

[0564] The server connects to the local government's management terminal and transmits activation commands for multiple aircraft.

[0565] Step 3:

[0566] The server uses geographical information to determine the optimal flight route for each aircraft and issues flight instructions.

[0567] Step 4:

[0568] The aircraft will fly over the affected area while capturing high-resolution video data.

[0569] Step 5:

[0570] The server receives video data transmitted from the aircraft in real time.

[0571] Step 6:

[0572] The server analyzes the received video data using an artificial intelligence module and generates a three-dimensional map.

[0573] Step 7:

[0574] The server sends the generated 3D map to the user's terminal to visualize the extent of the damage.

[0575] Step 8:

[0576] Users can view a 3D map on their device to understand the scale and extent of the damage.

[0577] Step 9:

[0578] The server activates the emotion engine and analyzes the facial expression and voice data obtained from the user.

[0579] Step 10:

[0580] The server analyzes the user's emotional state and determines their stress and anxiety levels.

[0581] Step 11:

[0582] Based on the analysis results, the server will send additional support information and encouraging messages to the user as needed.

[0583] Step 12:

[0584] Users receive information transmitted through their devices and can efficiently carry out rescue operations while feeling emotionally supported.

[0585] Step 13:

[0586] The server calculates the optimal route for transporting relief supplies based on a 3D map and emotional state data, and notifies the user.

[0587] Step 14:

[0588] Users will follow the relief plan displayed on their devices to begin transporting supplies and rescuing victims.

[0589] (Example 2)

[0590] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0591] While it is crucial to quickly and accurately assess the extent of damage during a disaster and to conduct effective relief operations, conventional systems have limitations in physically assessing the situation, and furthermore, they present challenges such as significant psychological burdens on those working on the ground.

[0592] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0593] In this invention, the server includes means for detecting a disaster, means for starting an aircraft, means for receiving image data captured by the aircraft, means for analyzing the image data to generate a three-dimensional map, means for formulating a support plan based on the three-dimensional map, means for analyzing the emotional state of a user, and means for providing support information based on the emotional state. This makes it possible to integrate the understanding of the physical situation with psychological support and improve the efficiency of disaster response.

[0594] A "disaster" refers to a serious situation in which human lives or property are endangered by natural phenomena or man-made factors.

[0595] An "aircraft" refers to an unmanned aerial vehicle that flies through the air and is capable of carrying specific equipment or devices to perform a mission.

[0596] "Image data" refers to visual information acquired by a camera mounted on an aircraft.

[0597] A "three-dimensional map" refers to a map generated based on three-dimensional spatial information, which visualizes the terrain, height, and shape of objects.

[0598] A "support plan" refers to a plan or policy designed to facilitate relief and recovery efforts in the event of a disaster.

[0599] "Users" refers to individuals or organizations that use the system to obtain information and carry out disaster response activities.

[0600] "Emotional state" refers to information that describes the user's psychological and emotional condition, including emotions such as stress and anxiety.

[0601] "Support information" refers to information about psychological support and behavioral guidelines provided according to the user's emotional state.

[0602] A "machine learning module" refers to a part of a system that processes information using algorithms that automatically analyze data and find patterns.

[0603] This invention provides a system that enables rapid and accurate assessment of damage during a disaster, and facilitates more effective rescue operations. This system consists of multiple elements, including an aircraft, a server, terminals, and an artificial intelligence module and an emotion engine.

[0604] The server first automatically launches an aircraft upon detecting a disaster. The aircraft is equipped with cameras and acquires image data of the affected area in real time. The acquired image data is sent to the server, which analyzes this data using an artificial intelligence module. In this analysis, object recognition technology is used to evaluate the extent of building damage and changes in terrain, and a 3D map is generated based on this. This 3D map is provided to the user via a terminal, allowing the user to visually confirm the extent of the damage.

[0605] Furthermore, the server collects facial expression and voice data through terminals to understand the emotional state of users working in the field. This data is analyzed by an emotion engine to evaluate the user's psychological state. For example, if the server determines that a user is experiencing high levels of stress, it automatically provides appropriate support information and takes measures to alleviate their psychological burden.

[0606] This system will enable not only the assessment of physical damage but also psychological support, improving the overall efficiency of disaster response. A concrete example is rescue operations during an earthquake. An example of a prompt message might be, "Please describe a system that assesses the damage situation in disaster-stricken areas in real time after an earthquake while also analyzing the psychological state of users on-site."

[0607] Thus, the invention realizes a new disaster response system unlike any other by combining high-performance hardware and advanced software.

[0608] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0609] Step 1:

[0610] The server detects the occurrence of a disaster using information from sensors and external alarm systems. Based on this information, it immediately issues commands to launch multiple aircraft. The aircraft then move to the disaster area according to pre-programmed routes.

[0611] Step 2:

[0612] The aircraft captures image data of the disaster area using its onboard high-resolution camera. During this process, it utilizes the camera's autofocus and exposure adjustment functions to obtain optimal images. The acquired image data is transmitted to a server in real time. Here, the input is the video footage of the disaster area, and the output is the raw data transmitted to the server.

[0613] Step 3:

[0614] The server passes the received image data as input to an artificial intelligence module, which then analyzes the extent of the damage. Object recognition algorithms are used in the data analysis to evaluate the degree of building damage and changes in terrain. A three-dimensional map of the affected area is generated as output of this process.

[0615] Step 4:

[0616] The server transfers the generated 3D map to the user's terminal. The user can then visually confirm the extent of the damage by manipulating this map on their terminal. Specifically, they can rotate the map and zoom in and out.

[0617] Step 5:

[0618] The server collects facial expression and voice data from the user to understand their emotional state. The device's camera and microphone function as input devices, and the data is sent to the server. The input data is analyzed by an emotion engine.

[0619] Step 6:

[0620] The server uses the results of the emotion engine's analysis to generate and provide optimal support information to the user. If the server detects that the user is experiencing stress, it automatically sends information including supplementary explanations and psychological support to reduce the user's burden. This output consists of information and instructions provided to the user.

[0621] (Application Example 2)

[0622] Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0623] Existing disaster response information gathering systems are limited to understanding the physical extent of damage, and have the drawback of not adequately supporting the psychological well-being of those working on the ground. Similarly, in security settings, there is a need not only to detect anomalies but also to understand people's psychological states and respond quickly and accurately. Therefore, a system is needed that can alleviate psychological pressure and enable more effective security monitoring.

[0624] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0625] In this invention, the server includes a device for detecting disaster events, a device for starting an aircraft, and a device for receiving image information acquired by the aircraft. This makes it possible to comprehensively assess the extent of damage during a disaster and provide psychological support at security sites.

[0626] A "disaster event" is an event that causes damage, such as a natural phenomenon or an accident.

[0627] An "aircraft" is defined as a vehicle capable of flying through the air and equipped with devices for collecting data such as video and audio.

[0628] "Image information" refers to visual data acquired by aircraft, which shows the situation at disaster sites and monitored areas.

[0629] A "spatial map" is three-dimensional geographical information generated by analyzing image data, allowing for an accurate understanding of the situation on site.

[0630] A "support plan" is a set of action guidelines formulated based on spatial maps and analysis results, aimed at disaster response and security management.

[0631] "Emotional information" refers to data that indicates the psychological state of the person receiving support, and is extracted from elements such as facial expressions and voice.

[0632] "Supplementary information" refers to information provided to the supervisor for additional guidance and support based on the analyzed emotional information.

[0633] A "machine learning module" is a program or system that processes large amounts of data and learns patterns and features from it.

[0634] This invention provides a system that offers rapid and accurate situation assessment and psychological support during disasters and security situations. The system mainly consists of a server, multiple stored aircraft, and user terminals.

[0635] When the server detects a disaster or security event, it quickly launches an aircraft. The aircraft then acquires image data from the scene and transmits it to the server in real time. The server uses machine learning modules and image processing techniques to analyze this image data and generate a spatial map. Specifically, it utilizes Python and OpenCV. Furthermore, it uses Google Cloud's natural language processing API to analyze sentiment information.

[0636] The generated spatial map and emotional information are transmitted to the user's terminal and used to develop support plans. Users operating the terminal receive supplementary information based on the psychological state of the person being monitored in the security environment and take appropriate measures. For example, if anxiety due to congestion is detected during event monitoring in a shopping mall, security guards can receive timely instructions and take the most appropriate action.

[0637] As a concrete example, the prompt message for analyzing surveillance camera footage from a commercial facility is as follows:

[0638] "We analyze footage from surveillance cameras installed within commercial facilities to detect congestion and suspicious activity in real time, and evaluate the psychological state of security guards."

[0639] This invention improves overall efficiency in disaster response and security monitoring.

[0640] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0641] Step 1:

[0642] The server detects disaster events and security events. Inputs are data from external sensors and existing monitoring systems, and outputs are signals to activate aircraft. When data exceeding a specific threshold is detected, the system reacts immediately.

[0643] Step 2:

[0644] The server starts the aircraft and directs it to the designated area. The input is the start signal generated in step 1, and the output is the aircraft's flight plan. The aircraft uses GPS to calculate the optimal path and fly accordingly.

[0645] Step 3:

[0646] The aircraft captures on-site image information and transmits it to the server in real time. The input is video data from the camera mounted on the aircraft, and the output is a data stream to the server. The aircraft adjusts its altitude and angle to capture images under optimal conditions.

[0647] Step 4:

[0648] The server analyzes the received image information and generates a spatial map. The input is the video data received in step 3, and the output is a three-dimensional spatial map. Here, Python and OpenCV are used to process the video data.

[0649] Step 5:

[0650] The server creates a support plan based on the generated spatial map and sends it to the user terminal. The input is a three-dimensional spatial map, and the output is a support plan document. Detailed information on the map is displayed on the user terminal.

[0651] Step 6:

[0652] The server processes emotional information to analyze the user's psychological state. Input is the user's facial expressions and voice data, and output is the emotional assessment result. Emotional analysis is performed using Google Cloud's natural language processing API.

[0653] Step 7:

[0654] The server sends supplementary information to the user terminal based on the analysis results. The input is the emotion assessment result, and the output is specific instructions and support information. The user terminal provides the received information through screen display and audio.

[0655] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0656] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0657] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0658] [Fourth Embodiment]

[0659] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0660] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0661] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0662] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0663] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0664] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0665] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0666] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0667] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0668] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0669] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0670] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0671] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0672] The system of this invention aims to quickly and accurately grasp the extent of damage in the event of a disaster and to plan effective relief activities. This system uses multiple unmanned aerial vehicles (flying objects) and cameras mounted on them to capture images of the disaster area and collect data in real time.

[0673] Upon detecting a disaster, the server immediately connects with the management terminal and automatically activates the designated aircraft. The server then sets the optimal flight route for each aircraft based on information about the predicted damage area and instructs them to collect video data. The aircraft then follow the set route and transmit video data of the damage to the server.

[0674] The server analyzes the received video data using an artificial intelligence module. This analysis generates a three-dimensional map visualizing the extent of the damage. This 3D map can be immediately viewed on the user's device. Users can visually grasp the scale and scope of the damage on their device and make quick decisions.

[0675] The server further combines 3D maps with other geographic information systems to automatically calculate the safest and most effective routes for transporting relief supplies. This relief plan is immediately notified to the user's terminal, allowing the user to carry out efficient relief operations based on it.

[0676] As a concrete example, consider a scenario where an earthquake occurs in a certain region. After detecting the disaster, the server, with the cooperation of the local government, activates multiple aircraft and directs them towards areas believed to be severely affected. The aircraft quickly collect video data, which the server uses to analyze and visualize the extent of the damage. Users can then access this information through their terminals and initiate effective rescue operations based on safe rescue routes. This system can significantly shorten the initial response time to a disaster and contribute to improving the survival rate.

[0677] The following describes the processing flow.

[0678] Step 1:

[0679] The server detects the occurrence of a disaster from the disaster monitoring system.

[0680] Step 2:

[0681] The server connects to the local government's management terminal and transmits activation commands for multiple aircraft.

[0682] Step 3:

[0683] The server determines the flight path for each aircraft based on pre-configured geographical information of the disaster area.

[0684] Step 4:

[0685] The server transmits the planned flight route to each aircraft and provides flight instructions.

[0686] Step 5:

[0687] The aircraft will fly over the affected area while capturing video data.

[0688] Step 6:

[0689] To transmit video data to the server in real time, the server continuously receives data.

[0690] Step 7:

[0691] The server inputs the received video data into an artificial intelligence module and begins analyzing the extent of the damage.

[0692] Step 8:

[0693] The server uses the analysis results to generate a three-dimensional map and create visualization data.

[0694] Step 9:

[0695] The server sends the visualization data to the user's terminal.

[0696] Step 10:

[0697] Users can view a 3D map through their device to understand the extent of the damage.

[0698] Step 11:

[0699] The server combines three-dimensional maps and geographical information to calculate the optimal route for transporting relief supplies.

[0700] Step 12:

[0701] The server sends the calculation results to the user's terminal and notifies them of a detailed rescue plan.

[0702] Step 13:

[0703] The user begins rescue operations based on the rescue plan displayed on their device.

[0704] (Example 1)

[0705] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0706] When natural disasters occur, it is crucial to quickly and accurately assess the extent of the damage. However, conventional methods often resulted in delays in relief efforts due to the time it took to gather information from affected areas. Furthermore, the inability to collect and analyze data in real time hindered the development of relief plans. Therefore, a more efficient, accurate, and rapid disaster information management system is needed.

[0707] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0708] In this invention, the server includes means for detecting the occurrence of a disaster, means for automatically activating an unmanned aerial vehicle (UAV), and means for setting an optimal flight route for the UAV. This enables the UAV to quickly collect information in the disaster area immediately after a disaster occurs and to carry out efficient and accurate rescue operations.

[0709] "Means for detecting the occurrence of a disaster" refers to a mechanism that uses a sensor system to detect the occurrence of a disaster in real time and notify of the anomaly.

[0710] "Means for automatically activating unmanned aerial vehicles" refers to a method of quickly activating pre-registered unmanned aerial vehicles via remote control when a disaster is detected.

[0711] "Means for setting the optimal flight route for the unmanned aerial vehicle" refers to a process that automatically calculates and instructs the flight path so that the unmanned aerial vehicle can efficiently collect data, taking into account geographical information and disaster information.

[0712] "A means of generating three-dimensional geographic information by analyzing video data with an artificial intelligence processing unit" refers to a technology that uses artificial intelligence technology to process acquired video footage and generate information that visualizes the damage situation in three dimensions.

[0713] "Means for formulating and notifying relief plans" refers to a method of planning effective relief activities based on generated three-dimensional geographic information and immediately communicating them to relevant agencies and personnel.

[0714] In an embodiment of this invention, a system is provided that utilizes unmanned aerial vehicles and artificial intelligence technology to support rapid and accurate information gathering and rescue operations in the event of a disaster.

[0715] The server monitors data in real time from sensors installed to detect disasters (such as seismometers and weather sensors). When an anomaly is detected, the server remotely activates an unmanned aerial vehicle (UAV) and directs it to the designated area. The UAV is equipped with a high-performance camera and a location information system, which allows it to collect high-precision video data of the disaster area.

[0716] The server uses algorithms to calculate the route to the disaster area in optimizing the flight path. This utilizes historical data, current weather, and topographic information. Captured video is streamed to the server in real time, where it proceeds to an image analysis process using a generated AI model. Here, the video data is converted into visual data that represents the damaged area in three dimensions.

[0717] The analysis results are displayed as three-dimensional geographic information on the user's terminal, allowing the user to quickly assess damage and plan rescue routes. Furthermore, the server integrates with a Geographic Information System (GIS) to formulate an optimal rescue plan based on the calculated rescue routes and notify the user.

[0718] As a concrete example, if an earthquake occurs in a certain area, when a sensor detects the tremor, the server immediately activates an unmanned aerial vehicle (UAV) and begins flying towards the affected area. Once the UAV has collected video data and the server has finished analyzing it, the user's terminal will be notified of a three-dimensional damage map along with effective rescue routes. Based on this information, local governments and rescue organizations can take swift action.

[0719] An example of an input prompt for a generating AI model might be: "Create a three-dimensional map of the area severely damaged by the earthquake and calculate the optimal rescue route."

[0720] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0721] Step 1:

[0722] The server receives data from sensors installed to detect disasters. The input is real-time data from the sensors, which the server analyzes to detect abnormal values. Specifically, the server uses a data processing algorithm to check signals that exceed a certain threshold. The output is disaster alert trigger information.

[0723] Step 2:

[0724] Upon confirming the occurrence of a disaster, the server sends a command to activate the unmanned aerial vehicles (UAVs). The input is disaster alert information, and the server accesses the UAV management system to remotely activate the necessary aircraft. Specifically, it sets a flight schedule and sends specific flight preparation instructions to the aircraft. The output is the status information of the activated UAVs.

[0725] Step 3:

[0726] The server calculates and transmits the optimal flight route for the unmanned aerial vehicle (UAV). Inputs include historical disaster data, real-time weather information, and topographic maps. Based on this information, the server performs algorithmic processing to determine an efficient route. Specifically, it simulates flight conditions and automatically generates a path that avoids obstacles. The output is transmitted to the UAV as a flight plan.

[0727] Step 4:

[0728] The unmanned aerial vehicle (UAV) flies along a designated route and collects video data with its onboard camera. The input is the flight plan from the server. As a specific action during flight, the UAV periodically takes video while adjusting the camera angle. The output is real-time video data.

[0729] Step 5:

[0730] The server receives video data transmitted from unmanned aerial vehicles (UAVs) and analyzes it using a generated AI model. The input is video data from the UAV. The server executes a video analysis program to identify the affected area and generate three-dimensional geographic information. Specifically, it detects and analyzes anomalies in the video and creates a three-dimensional map using the model. The output is a three-dimensional damage map.

[0731] Step 6:

[0732] The server develops a rescue plan based on the generated 3D map and notifies the user's terminal. The input is a 3D damage map. The server uses a geographic information system to calculate a safe and effective rescue route and proposes it to the user. Specifically, it visualizes the calculated route and sends it to the user's terminal for visual confirmation. The output is detailed rescue plan information and a map.

[0733] Step 7:

[0734] The user reviews the rescue plan information received on their terminal and begins rescue operations. Inputs are rescue plans and map information from the server. The user takes swift action in coordination with relevant organizations and teams. Specific actions include arranging necessary equipment and personnel and executing the plan. Outputs are status reports of the deployed rescue operations.

[0735] (Application Example 1)

[0736] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0737] While rapid and efficient relief efforts are required during disasters, it is difficult to immediately grasp the situation in the affected area and formulate an optimal relief plan. Furthermore, flexible plan modifications are necessary to respond immediately to changes in the situation on the ground. The objective of this invention is to provide a system that supports appropriate rescue decisions by accurately visualizing the current situation in the affected area in real time.

[0738] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0739] In this invention, the server includes means for detecting the occurrence of a disaster, means for activating an aircraft, means for receiving video data captured by the aircraft, means for analyzing the video data to generate a three-dimensional map, means for formulating a rescue plan based on the three-dimensional map, and means for displaying the rescue plan on a visual device in real time. This enables rapid and appropriate rescue operations based on an accurate understanding of the extent of the damage.

[0740] "Means for detecting the occurrence of a disaster" refers to devices or systems used to detect the occurrence of natural disasters or man-made emergencies.

[0741] "Means for activating an aircraft" refers to control devices and programs that enable unmanned aerial vehicles or drones to begin autonomous operation.

[0742] "Means for receiving video data" refers to communication devices or programs for acquiring image and video information transmitted from an aircraft.

[0743] "Methods for analyzing video data to generate three-dimensional maps" refer to devices or programs that use image processing technology and artificial intelligence to create three-dimensional maps of real-world terrain and structures based on received video data.

[0744] "Means for formulating relief plans" refer to systems and methodologies for assessing the extent of the damage and determining the optimal routes for supplying materials and scheduling relief activities.

[0745] "Means for displaying rescue plans on visual devices in real time" refers to technology that instantly displays generated rescue plans and 3D maps through smart glasses or displays, providing information to field personnel.

[0746] The system that realizes this invention consists of multiple unmanned aerial vehicles, a central server, and user terminals. When a disaster occurs, the server uses disaster detection means to confirm the emergency situation and automatically activates multiple unmanned aerial vehicles. The activated aerial vehicles collect video data using camera equipment while flying over the disaster area and transmit it to the server in real time.

[0747] The server uses an AI analysis module to process the received video data and generate a map that visualizes the damage in three dimensions. This map integrates multiple video data to clearly show the scale and extent of the damage. This process uses machine learning algorithms to represent the situation in the affected area in three dimensions through the analysis of pixel data.

[0748] Based on the generated 3D maps, the server develops the safest and most effective rescue plan. This plan combines complex geographic information systems to calculate the optimal transport route. The calculated rescue plan and visualized maps are immediately transmitted to the user's terminal and displayed on smart glasses or other visual devices.

[0749] Users can use this information to instantly grasp the situation and make quick decisions regarding relief efforts. For example, in the event of flood damage, unmanned aerial vehicles (UAVs) can take detailed photographs of the overflowing river and the flooding situation in the surrounding area. The server generates a 3D map of the location and extent of the damage, providing visual information to staff waiting at the logistics center to support the safe delivery of supplies.

[0750] Example prompt: "Develop an application to calculate the optimal delivery route to disaster-stricken areas used by a logistics center. This system will analyze video data acquired from unmanned aerial vehicles in real time, and staff will be able to view the data using smart glasses."

[0751] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0752] Step 1:

[0753] The server detects the occurrence of a disaster. Sensors and communication networks that detect natural disasters and man-made emergencies provide real-time information about the occurrence of a disaster. Inputs are data from sensors and communication networks, while outputs are information about the type and scale of the disaster.

[0754] Step 2:

[0755] The server activates the unmanned aerial vehicle (UAV). After confirming the occurrence of a disaster, it automatically issues deployment orders to multiple UAVs. The input is the disaster information obtained in step 1, and the output is the deployment orders to the UAVs.

[0756] Step 3:

[0757] The unmanned aerial vehicle (UAV) collects video data from the disaster area and transmits it to a server. It films the surrounding situation while flying and sends the footage to the server via wireless communication. The input is visual information from the disaster area, and the output is the raw video data transmitted to the server.

[0758] Step 4:

[0759] The server inputs video data into an AI analysis module and analyzes the damage situation in three dimensions. Using the input video data, a machine learning algorithm analyzes the pixel information and visualizes the damage as a three-dimensional map. The output is three-dimensional mapping information of structures and terrain.

[0760] Step 5:

[0761] The server plans logistics routes based on a 3D map. It utilizes geographic information systems and existing traffic data to develop safe and efficient logistics plans. Inputs are 3D maps and traffic data, and output is a relief plan that includes detailed delivery routes.

[0762] Step 6:

[0763] The server transmits rescue plans and 3D maps to the user's terminal. This allows the user to check the situation in real time and carry out appropriate rescue activities. The input is the rescue plan and 3D map, and the output is information displayed on smart glasses or visual devices.

[0764] Step 7:

[0765] Users use smart glasses to make on-site decisions based on plans and map information. Input is visual information displayed on the device, and output is the rescue operations and logistics actions that are carried out.

[0766] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0767] This invention combines a system designed for rapid and accurate assessment of damage during disasters and effective planning of relief operations with an emotion engine to recognize user emotions. The system utilizes multiple unmanned aerial vehicles (flying objects) equipped with cameras to capture images of the disaster area and collect data in real time.

[0768] The server, upon detecting a disaster, coordinates with the management terminal and activates multiple aircraft. Each aircraft is assigned an optimal flight route and heads towards designated locations to photograph the damage. The video data captured by the aircraft is transmitted to the server, where it is analyzed in real time by an artificial intelligence module. This generates a three-dimensional map of the damage, which users can visually view through their terminals.

[0769] Furthermore, the system incorporates an emotion engine that analyzes user reactions to understand the user's psychological state. The server inputs facial expression and voice data collected from the user into the emotion engine to analyze the user's emotional state.

[0770] For example, in the event of an earthquake, the server immediately begins assessing the situation in the affected area and performs emotional analysis on users on-site (rescue teams and local coordinators). If a user is experiencing stress, the server automatically uses this information to provide supplementary information and additional support, assisting with command and control. This enables effective responses without causing psychological pressure.

[0771] Thus, the system of the present invention makes it possible to integrate the assessment of the physical situation and the provision of psychological support in disaster-stricken areas, thereby improving the overall efficiency of disaster response.

[0772] The following describes the processing flow.

[0773] Step 1:

[0774] The server detects the occurrence of a disaster from the disaster monitoring system.

[0775] Step 2:

[0776] The server connects to the local government's management terminal and transmits activation commands for multiple aircraft.

[0777] Step 3:

[0778] The server uses geographical information to determine the optimal flight route for each aircraft and issues flight instructions.

[0779] Step 4:

[0780] The aircraft will fly over the affected area while capturing high-resolution video data.

[0781] Step 5:

[0782] The server receives video data transmitted from the aircraft in real time.

[0783] Step 6:

[0784] The server analyzes the received video data using an artificial intelligence module and generates a three-dimensional map.

[0785] Step 7:

[0786] The server sends the generated 3D map to the user's terminal to visualize the extent of the damage.

[0787] Step 8:

[0788] Users can view a 3D map on their device to understand the scale and extent of the damage.

[0789] Step 9:

[0790] The server activates the emotion engine and analyzes the facial expression and voice data obtained from the user.

[0791] Step 10:

[0792] The server analyzes the user's emotional state and determines their stress and anxiety levels.

[0793] Step 11:

[0794] Based on the analysis results, the server will send additional support information and encouraging messages to the user as needed.

[0795] Step 12:

[0796] Users receive information transmitted through their devices and can efficiently carry out rescue operations while feeling emotionally supported.

[0797] Step 13:

[0798] The server calculates the optimal route for transporting relief supplies based on a 3D map and emotional state data, and notifies the user.

[0799] Step 14:

[0800] Users will follow the relief plan displayed on their devices to begin transporting supplies and rescuing victims.

[0801] (Example 2)

[0802] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0803] While it is crucial to quickly and accurately assess the extent of damage during a disaster and to conduct effective relief operations, conventional systems have limitations in physically assessing the situation, and furthermore, they present challenges such as significant psychological burdens on those working on the ground.

[0804] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0805] In this invention, the server includes means for detecting a disaster, means for starting an aircraft, means for receiving image data captured by the aircraft, means for analyzing the image data to generate a three-dimensional map, means for formulating a support plan based on the three-dimensional map, means for analyzing the emotional state of a user, and means for providing support information based on the emotional state. This makes it possible to integrate the understanding of the physical situation with psychological support and improve the efficiency of disaster response.

[0806] A "disaster" refers to a serious situation in which human lives or property are endangered by natural phenomena or man-made factors.

[0807] An "aircraft" refers to an unmanned aerial vehicle that flies through the air and is capable of carrying specific equipment or devices to perform a mission.

[0808] "Image data" refers to visual information acquired by a camera mounted on an aircraft.

[0809] A "three-dimensional map" refers to a map generated based on three-dimensional spatial information, which visualizes the terrain, height, and shape of objects.

[0810] A "support plan" refers to a plan or policy designed to facilitate relief and recovery efforts in the event of a disaster.

[0811] "Users" refers to individuals or organizations that use the system to obtain information and carry out disaster response activities.

[0812] "Emotional state" refers to information that describes the user's psychological and emotional condition, including emotions such as stress and anxiety.

[0813] "Support information" refers to information about psychological support and behavioral guidelines provided according to the user's emotional state.

[0814] A "machine learning module" refers to a part of a system that processes information using algorithms that automatically analyze data and find patterns.

[0815] This invention provides a system that enables rapid and accurate assessment of damage during a disaster, and facilitates more effective rescue operations. This system consists of multiple elements, including an aircraft, a server, terminals, and an artificial intelligence module and an emotion engine.

[0816] The server first automatically launches an aircraft upon detecting a disaster. The aircraft is equipped with cameras and acquires image data of the affected area in real time. The acquired image data is sent to the server, which analyzes this data using an artificial intelligence module. In this analysis, object recognition technology is used to evaluate the extent of building damage and changes in terrain, and a 3D map is generated based on this. This 3D map is provided to the user via a terminal, allowing the user to visually confirm the extent of the damage.

[0817] Furthermore, the server collects facial expression and voice data through terminals to understand the emotional state of users working in the field. This data is analyzed by an emotion engine to evaluate the user's psychological state. For example, if the server determines that a user is experiencing high levels of stress, it automatically provides appropriate support information and takes measures to alleviate their psychological burden.

[0818] This system will enable not only the assessment of physical damage but also psychological support, improving the overall efficiency of disaster response. A concrete example is rescue operations during an earthquake. An example of a prompt message might be, "Please describe a system that assesses the damage situation in disaster-stricken areas in real time after an earthquake while also analyzing the psychological state of users on-site."

[0819] Thus, the invention realizes a new disaster response system unlike any other by combining high-performance hardware and advanced software.

[0820] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0821] Step 1:

[0822] The server detects the occurrence of a disaster using information from sensors and external alarm systems. Based on this information, it immediately issues commands to launch multiple aircraft. The aircraft then move to the disaster area according to pre-programmed routes.

[0823] Step 2:

[0824] The aircraft captures image data of the disaster area using its onboard high-resolution camera. During this process, it utilizes the camera's autofocus and exposure adjustment functions to obtain optimal images. The acquired image data is transmitted to a server in real time. Here, the input is the video footage of the disaster area, and the output is the raw data transmitted to the server.

[0825] Step 3:

[0826] The server passes the received image data as input to an artificial intelligence module, which then analyzes the extent of the damage. Object recognition algorithms are used in the data analysis to evaluate the degree of building damage and changes in terrain. A three-dimensional map of the affected area is generated as output of this process.

[0827] Step 4:

[0828] The server transfers the generated 3D map to the user's terminal. The user can then visually confirm the extent of the damage by manipulating this map on their terminal. Specifically, they can rotate the map and zoom in and out.

[0829] Step 5:

[0830] The server collects facial expression and voice data from the user to understand their emotional state. The device's camera and microphone function as input devices, and the data is sent to the server. The input data is analyzed by an emotion engine.

[0831] Step 6:

[0832] The server uses the results of the emotion engine's analysis to generate and provide optimal support information to the user. If the server detects that the user is experiencing stress, it automatically sends information including supplementary explanations and psychological support to reduce the user's burden. This output consists of information and instructions provided to the user.

[0833] (Application Example 2)

[0834] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0835] Existing disaster response information gathering systems are limited to understanding the physical extent of damage, and have the drawback of not adequately supporting the psychological well-being of those working on the ground. Similarly, in security settings, there is a need not only to detect anomalies but also to understand people's psychological states and respond quickly and accurately. Therefore, a system is needed that can alleviate psychological pressure and enable more effective security monitoring.

[0836] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0837] In this invention, the server includes a device for detecting disaster events, a device for starting an aircraft, and a device for receiving image information acquired by the aircraft. This makes it possible to comprehensively assess the extent of damage during a disaster and provide psychological support at security sites.

[0838] A "disaster event" is an event that causes damage, such as a natural phenomenon or an accident.

[0839] An "aircraft" is defined as a vehicle capable of flying through the air and equipped with devices for collecting data such as video and audio.

[0840] "Image information" refers to visual data acquired by aircraft, which shows the situation at disaster sites and monitored areas.

[0841] A "spatial map" is three-dimensional geographical information generated by analyzing image data, allowing for an accurate understanding of the situation on site.

[0842] A "support plan" is a set of action guidelines formulated based on spatial maps and analysis results, aimed at disaster response and security management.

[0843] "Emotional information" refers to data that indicates the psychological state of the person receiving support, and is extracted from elements such as facial expressions and voice.

[0844] "Supplementary information" refers to information provided to the supervisor for additional guidance and support based on the analyzed emotional information.

[0845] A "machine learning module" is a program or system that processes large amounts of data and learns patterns and features from it.

[0846] This invention provides a system that offers rapid and accurate situation assessment and psychological support during disasters and security situations. The system mainly consists of a server, multiple stored aircraft, and user terminals.

[0847] When the server detects a disaster or security event, it quickly launches an aircraft. The aircraft then acquires image data from the scene and transmits it to the server in real time. The server uses machine learning modules and image processing techniques to analyze this image data and generate a spatial map. Specifically, it utilizes Python and OpenCV. Furthermore, it uses Google Cloud's natural language processing API to analyze sentiment information.

[0848] The generated spatial map and emotional information are transmitted to the user's terminal and used to develop support plans. Users operating the terminal receive supplementary information based on the psychological state of the person being monitored in the security environment and take appropriate measures. For example, if anxiety due to congestion is detected during event monitoring in a shopping mall, security guards can receive timely instructions and take the most appropriate action.

[0849] As a concrete example, the prompt message for analyzing surveillance camera footage from a commercial facility is as follows:

[0850] "We analyze footage from surveillance cameras installed within commercial facilities to detect congestion and suspicious activity in real time, and evaluate the psychological state of security guards."

[0851] This invention improves overall efficiency in disaster response and security monitoring.

[0852] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0853] Step 1:

[0854] The server detects disaster events and security events. Inputs are data from external sensors and existing monitoring systems, and outputs are signals to activate aircraft. When data exceeding a specific threshold is detected, the system reacts immediately.

[0855] Step 2:

[0856] The server starts the aircraft and directs it to the designated area. The input is the start signal generated in step 1, and the output is the aircraft's flight plan. The aircraft uses GPS to calculate the optimal path and fly accordingly.

[0857] Step 3:

[0858] The aircraft captures on-site image information and transmits it to the server in real time. The input is video data from the camera mounted on the aircraft, and the output is a data stream to the server. The aircraft adjusts its altitude and angle to capture images under optimal conditions.

[0859] Step 4:

[0860] The server analyzes the received image information and generates a spatial map. The input is the video data received in step 3, and the output is a three-dimensional spatial map. Here, Python and OpenCV are used to process the video data.

[0861] Step 5:

[0862] The server creates a support plan based on the generated spatial map and sends it to the user terminal. The input is a three-dimensional spatial map, and the output is a support plan document. Detailed information on the map is displayed on the user terminal.

[0863] Step 6:

[0864] The server processes emotional information to analyze the user's psychological state. Input is the user's facial expressions and voice data, and output is the emotional assessment result. Emotional analysis is performed using Google Cloud's natural language processing API.

[0865] Step 7:

[0866] The server sends supplementary information to the user terminal based on the analysis results. The input is the emotion assessment result, and the output is specific instructions and support information. The user terminal provides the received information through screen display and audio.

[0867] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0868] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0869] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0870] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0871] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0872] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0873] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0874] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0875] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0876] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0877] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0878] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0879] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0880] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0881] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0882] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0883] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0884] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0885] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0886] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0887] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0888] The following is further disclosed regarding the embodiments described above.

[0889] (Claim 1)

[0890] Means for detecting the occurrence of a disaster,

[0891] A means of activating the flying object,

[0892] A means for receiving video data captured by the aforementioned aircraft,

[0893] A means for analyzing the aforementioned video data to generate a three-dimensional map,

[0894] A means of formulating a relief plan based on the aforementioned three-dimensional map,

[0895] A system that includes this.

[0896] (Claim 2)

[0897] The system according to claim 1, wherein the aforementioned flying object is a plurality of flying objects.

[0898] (Claim 3)

[0899] The system according to claim 1, wherein the analysis uses an artificial intelligence module.

[0900] "Example 1"

[0901] (Claim 1)

[0902] Means for detecting the occurrence of a disaster,

[0903] A means of automatically launching an unmanned aerial vehicle,

[0904] A means for setting an optimal flight route for the aforementioned unmanned aircraft,

[0905] A means for receiving video data captured by the aforementioned unmanned aerial vehicle,

[0906] A means for analyzing the aforementioned video data with an artificial intelligence processing unit to generate three-dimensional geographic information,

[0907] A means of formulating and notifying relief plans based on the aforementioned three-dimensional geographic information,

[0908] A system that includes this.

[0909] (Claim 2)

[0910] The system according to claim 1, wherein the unmanned aerial vehicle is a plurality of flying machines.

[0911] (Claim 3)

[0912] The system according to claim 1, wherein the analysis uses a generative artificial intelligence model.

[0913] "Application Example 1"

[0914] (Claim 1)

[0915] Means for detecting the occurrence of a disaster,

[0916] A means of activating the flying object,

[0917] A means for receiving video data captured by the aforementioned aircraft,

[0918] A means for analyzing the aforementioned video data to generate a three-dimensional map,

[0919] A means of formulating a relief plan based on the aforementioned three-dimensional map,

[0920] Means for displaying the aforementioned rescue plan on a visual device in real time,

[0921] A system that includes this.

[0922] (Claim 2)

[0923] The system according to claim 1, wherein the aforementioned flying object is a plurality of flying objects.

[0924] (Claim 3)

[0925] The system according to claim 1, wherein the analysis uses an artificial intelligence module.

[0926] "Example 2 of combining an emotion engine"

[0927] (Claim 1)

[0928] Means for detecting disasters,

[0929] Means of starting an aircraft,

[0930] Means for receiving image data captured by the aforementioned aircraft,

[0931] A means for analyzing the aforementioned image data to generate a three-dimensional map,

[0932] A means of formulating a support plan based on the aforementioned three-dimensional map,

[0933] A means of analyzing the emotional state of users,

[0934] A means for providing support information based on the aforementioned emotional state,

[0935] A system that includes this.

[0936] (Claim 2)

[0937] The system according to claim 1, wherein the aircraft is a plurality of aircraft.

[0938] (Claim 3)

[0939] The system according to claim 1, wherein the analysis uses a machine learning module.

[0940] "Application example 2 when combining with an emotional engine"

[0941] (Claim 1)

[0942] A device for detecting disaster events,

[0943] A device for starting an aircraft,

[0944] A device for receiving image information acquired by the aforementioned aircraft,

[0945] A device that analyzes the aforementioned image information to generate a spatial map,

[0946] A device for creating a support plan based on the aforementioned spatial map,

[0947] A device that analyzes the emotional information of the monitored subject based on the aforementioned support plan,

[0948] A device that provides auxiliary information to a monitoring device based on the aforementioned emotional information,

[0949] A system that includes this.

[0950] (Claim 2)

[0951] The system according to claim 1, wherein the aircraft is a plurality of aircraft.

[0952] (Claim 3)

[0953] The system according to claim 1, wherein the analysis uses a machine learning module. [Explanation of Symbols]

[0954] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for detecting the occurrence of a disaster, A means of activating the flying object, A means for receiving video data captured by the aforementioned aircraft, A means for analyzing the aforementioned video data to generate a three-dimensional map, A means of formulating a relief plan based on the aforementioned three-dimensional map, A system that includes this.

2. The system according to claim 1, wherein the aforementioned aircraft is a plurality of aircraft.

3. The system according to claim 1, wherein the analysis uses an artificial intelligence module.

Images