system

Abstract

We provide the system. [Solution] Means of observing the environment using aircraft and ground-based carriers, Information processing means including artificial intelligence for analyzing data acquired from the aforementioned aircraft and ground transport vehicles, A means of optimizing resource allocation and formulating rescue procedures based on the location and condition of disaster victims, Means for transmitting instructions to the aircraft and ground transport aircraft for supplying relief supplies based on the rescue procedure, A means of providing disaster information and evacuation routes to citizens using communication means connected to portable devices, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Manual rescue activities during natural disasters make it difficult to quickly identify disaster victims and efficiently distribute relief supplies, often resulting in a slow start. Also, since there is a large reliance on human resources, there are limitations and it is difficult to ensure the safety of disaster victims. The present invention aims to solve these problems and achieve rapid and efficient rescue activities.

Means for Solving the Problems

[0005] This invention includes means for observing the environment using aircraft and ground transport vehicles. Furthermore, it provides data processing means using artificial intelligence for analyzing data acquired from these observation means. It speeds up rescue operations by optimizing resource allocation based on the location and condition of victims and formulating efficient rescue procedures. Based on the formulated rescue procedures, it provides a system that transmits instructions for supplying relief goods to aircraft and ground transport vehicles, thereby realizing effective rescue operations.

[0006] An "aircraft" is an unmanned aerial vehicle that flies through the air and collects environmental data along a designated route.

[0007] A "ground transport vehicle" is an autonomous, mobile land-based vehicle that performs observations and data collection while moving along the ground.

[0008] "Means of observation" refers to technologies that use sensors and cameras mounted on aircraft and ground transport vehicles to collect environmental information and visible data.

[0009] "Data processing means including artificial intelligence" refers to a collection of algorithms and software used to analyze acquired data in real time and evaluate the situation in disaster-stricken areas.

[0010] "Location and status of disaster victims" refers to information indicating the specific location and health and safety status of people affected by a natural disaster.

[0011] "Means for optimizing resource allocation and formulating rescue procedures" refers to a function that automatically creates a plan for effectively using available relief resources and efficiently carrying out rescue operations, based on information about the victims.

[0012] "Directions for the supply of relief goods" is the process of sending instructions to aircraft and ground transport vehicles to deliver the most needed supplies to the disaster area. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] The following describes a specific process for implementing the autonomous disaster relief system of the present invention.

[0035] Server Role

[0036] The server is a central hub for managing the vast amount of data transmitted from aircraft and ground transport vehicles during disasters. Specifically, the server rapidly analyzes the received real-time data and utilizes artificial intelligence modules to identify victims and create maps of the affected areas. In the analysis, image recognition technology is used to pinpoint the location of victims, and voice recognition technology is used to detect their voices and movements. This enables more accurate rescue operations. Furthermore, the server formulates efficient rescue procedures and transmits this information as commands to aircraft and ground transport vehicles.

[0037] Terminal role

[0038] Aircraft and ground carriers, acting as terminals, utilize their different capabilities to cover disaster areas. Aircraft, with their ability to move quickly over wide areas, primarily conduct aerial observations. High-resolution cameras and infrared sensors are used for data collection, and this collected information is immediately transmitted to the server. Ground carriers acquire detailed environmental data and move through the disaster area while avoiding obstacles. In this way, terminals utilize sensors and communication functions to collect information accurately and quickly, and carry out rescue operations with the support of the server.

[0039] User roles

[0040] Users can monitor the progress of rescue operations through their control terminals and intervene manually as needed. In particular, in areas with complex terrain or where obstacles arise, users can make quick decisions to improve the efficiency of life-saving efforts. Users can also adjust resource allocation using analysis results from the server, allowing for a rapid response to high-priority rescue missions.

[0041] Specific example

[0042] For example, if a large-scale earthquake causes part of a city to collapse, the server creates a map of the affected area and identifies the locations of victims. When an aircraft flies over the collapsed buildings and uses thermal cameras to locate victims under the rubble, ground transport aircraft head to that location to deliver relief supplies. Users can monitor this data in real time and send new commands to the server as needed. This system makes it possible to provide rapid and effective support to victims.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The server receives sensor data and image data transmitted from terminal aircraft and ground transport vehicles. This data is important as basic information for understanding the current situation in the disaster-stricken areas.

[0046] Step 2:

[0047] The aircraft, acting as the terminal, flies within a designated patrol area, using its onboard cameras and infrared sensors to collect real-time video and thermal information. The collected data is immediately transmitted to a server.

[0048] Step 3:

[0049] The ground transporter, acting as the terminal, moves across the ground while avoiding obstacles, and uses voice and heat sensors to locate potential locations of victims buried under rubble, detecting their voices and body temperature. This information is also immediately transmitted to the server.

[0050] Step 4:

[0051] The server uses image recognition algorithms based on the received data to identify individuals and locations that may be victims of the disaster. This process analyzes the obtained thermal and audio data to prioritize the identification of high-risk areas.

[0052] Step 5:

[0053] The server develops rescue procedures based on the location information of identified victims. These procedures include determining the type and quantity of relief supplies needed and optimizing resource allocation, such as which terminals should use which resources.

[0054] Step 6:

[0055] Based on the established rescue procedures, the server transmits specific action instructions to the aircraft and ground transport vehicles, which act as terminals. The aircraft airdrops the relief supplies to the designated locations, and the ground transport vehicles move to the detailed locations to distribute the supplies.

[0056] Step 7:

[0057] Users use dedicated control terminals to monitor information from the server and track the progress of rescue operations. They can also send new commands to the server as needed to further adjust the rescue efforts.

[0058] Step 8:

[0059] Users monitor the progress of rescue operations and provide feedback on the procedures developed by the server. This feedback will be used to determine future resource allocation and prioritization.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] When a disaster strikes, it is crucial to quickly and accurately ascertain the location and condition of victims and to conduct effective rescue operations. Current systems lack the accuracy of real-time information analysis and the immediacy of rescue instructions, resulting in reduced efficiency in rescue efforts. Furthermore, a system is needed that can flexibly respond to widespread changes in the situation and formulate optimal rescue procedures. Therefore, key challenges in this technological field are improving the accuracy of information analysis and achieving optimal resource allocation according to the disaster situation.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for conducting wide-area observations using aircraft and ground transporters, information processing means including artificial intelligence for analyzing information collected from the aircraft and ground transporters in real time, and means for formulating rescue procedures and setting priorities for rescue operations based on the location and condition of victims and the situation in the affected area. This enables effective rescue operations through the rapid identification of victims and appropriate allocation of resources.

[0065] An "aircraft" is a means of air transport, such as airplanes and drones, that can fly through the air and conduct observations over a wide area.

[0066] A "ground transport vehicle" is a mechanical device that moves along the ground and can conduct observations while collecting detailed environmental information.

[0067] "Wide-area observation" refers to observational activities that cover a large area and acquire information quickly.

[0068] "Information processing means" refers to functions, including artificial intelligence, used to analyze collected data and extract necessary information.

[0069] A "rescue procedure" is a set of steps for planning the priorities and flow of various activities in rescuing disaster victims.

[0070] "Communication means" refers to functions that include communication technologies for exchanging information and transmitting instructions between various devices.

[0071] "Geometric analysis" is a technology that analyzes images acquired by cameras and sensors to extract useful information.

[0072] "Speech recognition" is a technology that analyzes audio data to identify linguistic information and the source of sound.

[0073] "Environmental conditions" refer to the situation on site, including geographical information of the affected area and the impact of the disaster.

[0074] "Resource status" refers to the current state of available resources such as personnel, supplies, and equipment.

[0075] This invention is an autonomous disaster relief system in which a server, terminals, and users work together. The server acts as the central control unit, aggregating and analyzing data transmitted from aircraft and ground transport vehicles. Specifically, the server uses a high-performance computer to build an information processing system using a generative AI model. This system excels particularly in image recognition and speech recognition technologies, enabling it to pinpoint the location of disaster victims in real time and analyze the situation in disaster areas.

[0076] The aircraft, acting as terminals, are equipped with high-resolution cameras and infrared sensors, enabling rapid observation of wide areas. This allows the aircraft to collect data from the air and immediately transmit the information to the server. Ground carriers, moving across the land, can collect detailed environmental information and are equipped with sensors to avoid obstacles. These terminals operate based on commands from the server, enabling a rapid response to disaster victims.

[0077] Users can monitor the overall progress of the system via an operating terminal and intervene in rescue operations as needed. Users can view terrain information and analysis data in real time and send new commands to the server based on that information. For example, if the disaster area has very complex terrain or if a large number of victims are detected, users can make immediate decisions and readjust rescue policies.

[0078] As a concrete example, consider a scenario where a large-scale earthquake causes part of a city to collapse. Aircraft scan the affected area from above with thermal cameras, and ground transport vehicles navigate the complex terrain to deliver supplies to victims. This entire process is supported by a system that uses server analysis and allows users to monitor the results in real time. An example of a prompt from the generated AI model is, "Using thermal cameras, how can we locate victims trapped under rubble in the disaster area?" This system enables rapid and effective relief for disaster victims.

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The server receives observational data transmitted from aircraft and ground carriers. This input data includes high-resolution images, thermal information from infrared sensors, and information about the local environmental conditions. The server performs a basic validation of this data and prepares it for analysis. Specific actions performed at this stage include data format conversion and priority setting.

[0082] Step 2:

[0083] The server analyzes the received data using a generating AI model. Using the high-resolution images and thermal information received in step 1 as input, it executes image recognition and speech recognition algorithms. The server identifies the location of victims through shape recognition and detects unusual sounds and voices from the audio data. This results in the production of location information and status data for the victims as the output of the analysis.

[0084] Step 3:

[0085] The server develops rescue procedures based on the analysis results. It uses the location and status of victims, as well as the environmental conditions at the site, obtained in Step 2, as input. Based on this, the server determines the priority of rescue operations and plans the supply routes for relief materials. The output generates information regarding rescue procedures and transportation routes.

[0086] Step 4:

[0087] The server transmits specific commands to aircraft and ground transport vehicles based on the established rescue procedures and transport routes. This process includes detailed instructions on how to operate rescue equipment and the delivery schedule for supplies. Terminals receive these commands and begin actual observation and rescue operations.

[0088] Step 5:

[0089] Users monitor ongoing rescue operations using control terminals for real-time monitoring. Using data provided in real-time from the server as input, users can make immediate decisions based on the situation and send new commands to the server. This allows for manual intervention as needed.

[0090] (Application Example 1)

[0091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] In recent years, the frequency of natural disasters has increased, highlighting the growing importance of rapid and efficient disaster response. However, conventional disaster relief systems often suffer from delays in information acquisition and transmission, resulting in insufficient support for disaster victims. Furthermore, dynamic evacuation guidance tailored to the specific conditions of the affected area has been difficult. Therefore, there is a need for a system that can collect accurate information in real time during a disaster and quickly provide rescue operations and evacuation guidance to victims.

[0093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0094] In this invention, the server includes means for observing the environment using aircraft and ground transporters, information processing means including artificial intelligence for analyzing data acquired from the aircraft and ground transporters, means for optimizing resource allocation and formulating rescue procedures based on the location and condition of disaster victims, and means for providing disaster information and evacuation routes using communication means connected to a device that can be carried by citizens. This enables rapid and efficient support for disaster victims and effective evacuation guidance for citizens during disasters.

[0095] An "aircraft" is a means of transportation that can directly observe the Earth's surface from above and rapidly collect data over a wide area.

[0096] A "ground transport device" is a device used to collect detailed environmental data from the ground surface and move through disaster-stricken areas while avoiding obstacles.

[0097] "Information processing means" refers to a system that includes artificial intelligence to analyze data acquired from aircraft and ground transport vehicles and support effective rescue operations.

[0098] "Means for optimizing resource allocation and formulating rescue procedures" refers to methods for planning rapid and appropriate rescue actions by efficiently allocating available resources based on the location and condition of the victims.

[0099] "Communication methods" refer to technologies that provide real-time disaster information and evacuation routes to disaster victims and the general public using mobile devices that they can access.

[0100] This invention implements an autonomous disaster relief system in which servers, terminals, and users each play their respective roles, enabling rapid and accurate assistance to disaster victims during a disaster.

[0101] The server receives environmental data transmitted from aircraft and ground transport vehicles and performs real-time analysis using information processing tools. Specifically, it utilizes artificial intelligence to perform image and voice recognition to identify the location and condition of disaster victims. The server uses the Google® Maps API to update map information of the disaster area and formulate effective rescue procedures. Furthermore, it transmits optimized instructions to aircraft and ground transport vehicles to deploy the necessary actions.

[0102] The aircraft, acting as the terminal, is equipped with high-resolution cameras and infrared sensors to conduct wide-area observations from above. This allows for a detailed understanding of the collapsed areas that are difficult for ground carriers to access. The ground carriers are designed to collect detailed information on the ground and move safely through the disaster area, operating reactively based on instructions from the server.

[0103] Users can receive the latest notifications regarding disaster information and evacuation routes from the server using their smartphones or other mobile devices. Important information is immediately conveyed via push notifications, and detailed information about affected areas is also provided through analysis using generative AI models. Based on this information, users can take swift evacuation action.

[0104] As a concrete example, in the event of a large-scale earthquake in an urban area, the server creates a detailed map of the affected area and analyzes information on the safety of victims. Citizens receive this information via their mobile devices and are guided to safe evacuation routes. An example of a prompt message would be a response to a request such as, "Based on the current evacuation information, please provide the optimal evacuation route."

[0105] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0106] Step 1:

[0107] The server receives real-time sensor data transmitted from aircraft and ground carriers. This data includes image data, infrared analysis data, and audio data. The received data is first temporarily stored in a database and then pre-processed by an artificial intelligence module.

[0108] Step 2:

[0109] The server uses a generative AI model to analyze the received image data and identify the location and condition of the victims. This process applies pattern recognition algorithms to detect human shapes and movements from the images. The input is image data, and the output is the location coordinates of the victims.

[0110] Step 3:

[0111] The server simultaneously analyzes audio data and performs speech recognition processing to identify the voices of disaster victims. Here, the audio waveform is processed as input, and location information of potential victims is output. The results of the audio analysis are combined with the results of the image analysis to improve accuracy.

[0112] Step 4:

[0113] The server uses the Google Maps API to generate a map of the disaster area based on the analysis results from the generated AI model, and overlays the locations of the victims. At this stage, the input is the analyzed location data of the victims, and the output is a detailed map of the disaster area.

[0114] Step 5:

[0115] Users receive maps and the latest evacuation information from a server using their smartphones or smart glasses. The app running on the user's device receives real-time information via push notifications. The input information is command data based on the analysis results of a generated AI model, and the output is evacuation route guidance as visual information.

[0116] Step 6:

[0117] The aircraft and ground transport units, acting as terminals, autonomously move based on the location of disaster victims, receiving commands from the server, and initiate rescue operations as needed, according to the situation in the disaster area. Real-time location information is used as input, and actions such as supplying materials and initiating rescue operations are output.

[0118] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0119] This invention aims to further improve the efficiency of disaster relief activities by combining an emotion engine with an environmental observation system using aircraft and ground transport vehicles. This system utilizes artificial intelligence as a data processing means and can understand the emotional state of users and disaster victims through the emotion engine.

[0120] Server Role

[0121] The server acts as a management center for vast amounts of data. After receiving data from terminals, it first analyzes environmental information. Using image analysis algorithms and speech recognition technology, it identifies the location and condition of disaster victims. This data is used to formulate rescue procedures and optimize resource allocation. The emotion engine also analyzes received audio and image data to recognize the user's emotional state and adjusts rescue procedures based on particularly important emotional changes. Emotion recognition allows the server to make more appropriate decisions for rescue operations.

[0122] Terminal role

[0123] As terminals, aircraft and ground carriers perform specific actions in real time based on instructions from the server. Aircraft are responsible for wide-area observation, continuously collecting data with thermal sensors and high-resolution cameras. Meanwhile, ground carriers acquire detailed environmental information and the location of disaster victims, while simultaneously attempting to communicate with them. For example, by using an emotion engine, it is possible to select an appropriate communication method if a disaster victim is experiencing fear. Furthermore, when delivering relief supplies, consideration is given to the emotions of the disaster victims.

[0124] User roles

[0125] Users can monitor rescue operations via their devices and refer to sentiment analysis data to conduct smoother rescue operations. In particular, sentiment data is fed back into the user interface, serving as a decision-making support tool for users. Specifically, it can present users with countermeasures to alleviate the anxiety and panic of disaster victims, enabling more effective rescue operations. This information is reported to the server and reflected in future rescue operations.

[0126] As a result, this system, which incorporates an emotional engine, provides deeper insights that could not be obtained in conventional rescue operations, enabling swift and accurate rescue efforts.

[0127] The following describes the processing flow.

[0128] Step 1:

[0129] The server continuously receives data from aircraft and ground carriers equipped with various sensors. The data includes a wide range of information, such as images, audio, and temperature information of the external environment.

[0130] Step 2:

[0131] The aircraft, acting as the terminal, will photograph a wide area of ​​the disaster zone from above and provide real-time data. The aircraft will fly autonomously to cover the entire disaster area and transmit the observation data to the server.

[0132] Step 3:

[0133] The ground-based transporter, acting as the terminal, avoids obstacles on the ground and transmits acquired audio and proximity sensor data to the server. This allows for high-precision detection of sounds emitted by people trapped under rubble.

[0134] Step 4:

[0135] The server processes incoming data instantly, using AI image analysis and speech recognition to determine the location and condition of victims. Furthermore, an emotion engine analyzes the emotions from the victims' voices to identify their emotional state.

[0136] Step 5:

[0137] The server develops rescue procedures based on the physical location and psychological state of the victims. If the emotional state is urgent, this is reflected in the rescue plan, and the procedures are adjusted to ensure a rapid response.

[0138] Step 6:

[0139] Based on the formulated plan, the server sends specific instructions to the terminals. Aircraft are instructed on the location and order of dropping relief supplies, and ground transport aircraft are instructed on the delivery of supplies to victims and appropriate communication methods.

[0140] Step 7:

[0141] Users have the ability to review sentiment analysis data sent from the server and adjust rescue priorities in real time. They also communicate with discovered victims sequentially and provide support according to suggestions from the sentiment engine.

[0142] Step 8:

[0143] Users can send feedback to the server as needed regarding ongoing rescue operations, providing insights to improve the quality of future operations. This feedback is recorded and analyzed by the system and used to improve the efficiency of future rescue operations.

[0144] (Example 2)

[0145] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0146] Conventional disaster relief systems were somewhat effective in locating the position and condition of victims. However, they lacked the ability to adaptively change rescue procedures based on changes in the emotional state of victims, and the effective allocation of rescue resources was insufficient. As a result, rapid and accurate rescue of victims was difficult.

[0147] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0148] In this invention, the server includes means for analyzing data acquired from aircraft and ground transport vehicles to understand the emotional state of disaster victims, means for optimizing the allocation of rescue resources based on the emotional state, and means for formulating rescue procedures and transmitting instructions for them. This enables flexible adjustment of rescue procedures according to the emotional state of disaster victims and more efficient allocation of rescue resources.

[0149] An "aircraft" is a means of transportation used for observation and data collection by flying through the air.

[0150] A "ground transport device" is a means of transport used to collect and transmit environmental data and disaster victim information while moving along the ground surface.

[0151] "Means of observing the environment" refers to devices, including sensors and cameras, that are mounted on aircraft or ground transport vehicles and are necessary for detailed analysis of the surrounding environment.

[0152] A "data processing system" is a system that includes machine learning algorithms for analyzing acquired data and making decisions based on that information.

[0153] "Analyzing emotional states" is the process of identifying and understanding the emotional responses of disaster victims from audio and image data.

[0154] "Optimizing resource allocation" is the process of effectively utilizing limited rescue resources and quickly distributing them to the places and situations where they are most needed.

[0155] "Means for formulating rescue procedures" refers to the function of formulating a specific rescue action plan based on the analysis results and building a strategy for putting it into action.

[0156] This invention is designed to enhance the effectiveness of disaster relief operations based on an environmental observation system utilizing aircraft and ground carriers. This system uses aircraft and ground carriers equipped with various sensors as hardware, and machine learning algorithms running on a server.

[0157] Server Role

[0158] The server plays a central role in receiving and analyzing data transmitted from terminals. Specifically, it uses machine learning software to process image and audio data collected from aircraft and ground transport vehicles. Widely used machine learning libraries are utilized for image processing to identify the location and condition of victims. For audio data, sentiment analysis techniques are used to determine the emotional state of victims.

[0159] Terminal role

[0160] The aircraft, acting as the terminal, is equipped with high-resolution cameras and thermal sensors to conduct wide-area environmental observations. The aircraft collects environmental information over a wide area and transmits this data to a server. Ground carriers are responsible for approaching disaster victims and acquiring detailed environmental and audio information. This enables direct communication on-site and allows for adaptive responses based on emotion analysis.

[0161] User roles

[0162] Users refer to data from the server and manage rescue operations based on the displayed situation. The user interface reflects the emotional state and location information of victims in real time, enabling rapid decision-making based on this information. This information is also fed back after the rescue operation is completed and used to improve future responses.

[0163] Specific examples and prompt statements

[0164] One specific example is the use of this system to conduct rapid rescue operations in areas where flooding is anticipated. In this case, an aircraft would take extensive aerial photographs of the affected area, and a server would use these to identify the location and emotional state of the victims. Ground carriers would then send reassuring messages to the identified victims.

[0165] Examples of prompts to input into a generative AI model:

[0166] "Explain how to identify victims in flood-affected areas using sentiment-based data and rapidly allocate appropriate relief resources."

[0167] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0168] Step 1:

[0169] The terminal uses a high-resolution camera and thermal sensors mounted on the aircraft to collect environmental information for a designated area. The input consists of image data and thermal data acquired from the air, and the output is collected environmental data. This data provides fundamental information necessary to understand the overall situation of the region. Specifically, the aircraft follows a designated flight path and periodically records data.

[0170] Step 2:

[0171] The terminal collects detailed information about the vicinity of disaster victims via a ground transport device. Inputs include voice data and more detailed environmental data, and output is the result of on-site communication. The ground transport device understands the emotional state of disaster victims by directly communicating with them via voice. Specific actions include the ground transport device moving closer to the disaster victims and attempting to communicate using microphones and speakers.

[0172] Step 3:

[0173] The server analyzes all data transmitted from aircraft and ground transport vehicles. Inputs include collected image data, thermal data, and audio data, while outputs include the location information and emotional state analysis results of victims. The server uses machine learning algorithms to analyze images, identify the location of victims, and recognize emotional states based on audio data. Specific operations include training a model using the dataset and making predictions through real-time processing.

[0174] Step 4:

[0175] The server develops rescue procedures and creates an optimal resource allocation plan based on the analysis results. Inputs are location information and emotional state, while outputs are specific rescue procedures and resource allocation plans. The developed procedures are flexibly adjusted to reflect the emotional state of the victims, ensuring appropriate actions are taken according to the situation. Specific operations include prompt generation using a generative AI model and scenario simulation.

[0176] Step 5:

[0177] Users monitor rescue procedures generated by the server and make modifications as needed. Inputs are detailed rescue procedures and real-time feedback from the field, while outputs are the final execution plan. Users refer to the plan and make necessary decisions through the interface. Specific actions include sending instructions to the rescue team and checking and updating field data.

[0178] (Application Example 2)

[0179] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0180] Current disaster relief systems can grasp the physical condition and location of victims, but they struggle to monitor their emotional state in real time and provide appropriate support accordingly. Furthermore, in elderly care support, the lack of consideration for emotional states makes it difficult for caregivers to quickly and accurately determine the necessary support.

[0181] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0182] In this invention, the server includes means for observing the environment using an aircraft and a ground transporter, data processing means including artificial intelligence for analyzing data acquired from the aircraft and ground transporter, and processing means for analyzing emotional information and determining appropriate responses and support based on the emotional state of disaster victims. This enables prompt and accurate support and care that takes into account the emotional state of disaster victims and the elderly.

[0183] An "aircraft" is a flying device used for aerial observation and data collection.

[0184] A "ground transport vehicle" is a vehicle used for movement near the Earth's surface, transporting materials, and conducting detailed observations.

[0185] "Means of observing the environment" refers to methods of acquiring surrounding environmental data using aircraft or ground-based transport vehicles.

[0186] "Data processing means" refers to methods that use artificial intelligence to analyze data acquired from aircraft and ground transport vehicles.

[0187] "Artificial intelligence" is a program that analyzes data and automatically makes judgments and decisions using specific algorithms.

[0188] "Emotional information" refers to data on the emotional state of disaster victims in real time, as well as the results of its analysis.

[0189] An "emotion engine" is a technology that analyzes the voice and image data of disaster victims to recognize their emotional state.

[0190] "Means for optimizing resource allocation" refers to methods for determining rescue procedures and support based on acquired data and emotional information.

[0191] "Rescue procedures" are the procedures established to carry out appropriate rescue operations according to the circumstances of the disaster.

[0192] "Care support" refers to support activities aimed at providing appropriate assistance according to the emotional state of elderly individuals.

[0193] This invention is a system that uses aircraft and ground transport vehicles to observe the environment and employs artificial intelligence as a data processing tool to grasp the emotional state of disaster victims and the elderly in real time, thereby enabling optimal support activities.

[0194] The server receives environmental data transmitted from aircraft and ground transport vehicles and analyzes it using artificial intelligence. This allows it to understand the location and physical condition of disaster victims, and recognize their emotional state using an emotion engine. This process utilizes image analysis libraries and speech recognition technologies (e.g., OpenCV and TENSORFLOW®). Emotional information is used to formulate rescue procedures and contribute to the optimization of support activities. Based on the accumulated data, the server improves the efficiency of future rescue operations.

[0195] The aircraft and ground carriers, acting as terminals, are responsible for wide-area observation and detailed data acquisition. The aircraft primarily collects environmental information from the air, while the ground carriers are configured to capture detailed ground information and emotional changes. The ground carriers communicate with disaster victims and adjust appropriate support activities according to their emotional state. In this process, emotional recognition is used to select responses that will make disaster victims feel safe.

[0196] Users can view aggregated data using a dedicated interface and check the emotional state of disaster victims and the elderly in real time. This supports rapid decision-making on-site and enables the sending of appropriate instructions to the terminal. In this process, the emotional data and recommended actions presented to the user are updated in real time by a generative AI model.

[0197] As a concrete example, when this system is applied to a nursing home, it can monitor the emotional state of elderly residents and automatically provide relaxing music content when it detects that they are feeling lonely. This can reduce the psychological burden on the elderly and create a more comfortable living environment.

[0198] An example of a prompt for a generative AI model is: "Recommend appropriate music content to alleviate feelings of loneliness in the elderly. If the emotional state is determined to be 'lonely,' please provide specific song titles and the reasons for their selection."

[0199] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0200] Step 1:

[0201] The server receives real-time environmental data transmitted from aircraft and ground transport vehicles. This includes high-resolution image and audio data. Based on this input data, artificial intelligence begins initial data processing.

[0202] Step 2:

[0203] The server uses the received image data to identify the locations of disaster victims and elderly individuals using an image analysis library (e.g., OpenCV). Additionally, it utilizes speech recognition technology (e.g., TensorFlow) to extract meaning and emotional information from the audio data. This process generates detailed data regarding the location and condition of disaster victims.

[0204] Step 3:

[0205] The ground transporter, acting as a terminal, collects detailed ground information and utilizes an emotion engine to recognize the detailed emotional state of disaster victims. The analyzed emotional information is sent to a server, and an appropriate method of communication with the victims is selected. In this process, data based on changes in emotion is generated.

[0206] Step 4:

[0207] The server integrates all analyzed data and optimizes rescue procedures and support methods using a generative AI model. It provides users with support suggestions and action guidelines based on their emotional state. This process generates prompts and determines the specific details of the support.

[0208] Step 5:

[0209] Based on data provided by the server, the user selects an appropriate action and sends instructions to the device. For example, if loneliness in an elderly person is detected, a specific instruction is created to provide relaxing music. The device then carries out the support activity according to the user's instructions.

[0210] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0211] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0212] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0213] [Second Embodiment]

[0214] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0215] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0216] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0217] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0218] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0219] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0220] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0221] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0222] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0223] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0224] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0225] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0226] The following describes a specific process for implementing the autonomous disaster relief system of the present invention.

[0227] Server Role

[0228] The server is a central hub for managing the vast amount of data transmitted from aircraft and ground transport vehicles during disasters. Specifically, the server rapidly analyzes the received real-time data and utilizes artificial intelligence modules to identify victims and create maps of the affected areas. In the analysis, image recognition technology is used to pinpoint the location of victims, and voice recognition technology is used to detect their voices and movements. This enables more accurate rescue operations. Furthermore, the server formulates efficient rescue procedures and transmits this information as commands to aircraft and ground transport vehicles.

[0229] Terminal role

[0230] Aircraft and ground carriers, acting as terminals, utilize their different capabilities to cover disaster areas. Aircraft, with their ability to move quickly over wide areas, primarily conduct aerial observations. High-resolution cameras and infrared sensors are used for data collection, and this collected information is immediately transmitted to the server. Ground carriers acquire detailed environmental data and move through the disaster area while avoiding obstacles. In this way, terminals utilize sensors and communication functions to collect information accurately and quickly, and carry out rescue operations with the support of the server.

[0231] User roles

[0232] Users can monitor the progress of rescue operations through their control terminals and intervene manually as needed. In particular, in areas with complex terrain or where obstacles arise, users can make quick decisions to improve the efficiency of life-saving efforts. Users can also adjust resource allocation using analysis results from the server, allowing for a rapid response to high-priority rescue missions.

[0233] Specific example

[0234] For example, if a large-scale earthquake causes part of a city to collapse, the server creates a map of the affected area and identifies the locations of victims. When an aircraft flies over the collapsed buildings and uses thermal cameras to locate victims under the rubble, ground transport aircraft head to that location to deliver relief supplies. Users can monitor this data in real time and send new commands to the server as needed. This system makes it possible to provide rapid and effective support to victims.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] The server receives sensor data and image data transmitted from terminal aircraft and ground transport vehicles. This data is important as basic information for understanding the current situation in the disaster-stricken areas.

[0238] Step 2:

[0239] The aircraft, acting as the terminal, flies within a designated patrol area, using its onboard cameras and infrared sensors to collect real-time video and thermal information. The collected data is immediately transmitted to a server.

[0240] Step 3:

[0241] The ground transporter, acting as the terminal, moves across the ground while avoiding obstacles, and uses voice and heat sensors to locate potential locations of victims buried under rubble, detecting their voices and body temperature. This information is also immediately transmitted to the server.

[0242] Step 4:

[0243] The server uses image recognition algorithms based on the received data to identify individuals and locations that may be victims of the disaster. This process analyzes the obtained thermal and audio data to prioritize the identification of high-risk areas.

[0244] Step 5:

[0245] The server develops rescue procedures based on the location information of identified victims. These procedures include determining the type and quantity of relief supplies needed and optimizing resource allocation, such as which terminals should use which resources.

[0246] Step 6:

[0247] Based on the established rescue procedures, the server transmits specific action instructions to the aircraft and ground transport vehicles, which act as terminals. The aircraft airdrops the relief supplies to the designated locations, and the ground transport vehicles move to the detailed locations to distribute the supplies.

[0248] Step 7:

[0249] Users use dedicated control terminals to monitor information from the server and track the progress of rescue operations. They can also send new commands to the server as needed to further adjust the rescue efforts.

[0250] Step 8:

[0251] Users monitor the progress of rescue operations and provide feedback on the procedures developed by the server. This feedback will be used to determine future resource allocation and prioritization.

[0252] (Example 1)

[0253] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0254] When a disaster strikes, it is crucial to quickly and accurately ascertain the location and condition of victims and to conduct effective rescue operations. Current systems lack the accuracy of real-time information analysis and the immediacy of rescue instructions, resulting in reduced efficiency in rescue efforts. Furthermore, a system is needed that can flexibly respond to widespread changes in the situation and formulate optimal rescue procedures. Therefore, key challenges in this technological field are improving the accuracy of information analysis and achieving optimal resource allocation according to the disaster situation.

[0255] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0256] In this invention, the server includes means for conducting wide-area observations using aircraft and ground transporters, information processing means including artificial intelligence for analyzing information collected from the aircraft and ground transporters in real time, and means for formulating rescue procedures and setting priorities for rescue operations based on the location and condition of victims and the situation in the affected area. This enables effective rescue operations through the rapid identification of victims and appropriate allocation of resources.

[0257] An "aircraft" is a means of air transport, such as airplanes and drones, that can fly through the air and conduct observations over a wide area.

[0258] A "ground transport vehicle" is a mechanical device that moves along the ground and can conduct observations while collecting detailed environmental information.

[0259] "Wide-area observation" refers to observational activities that cover a large area and acquire information quickly.

[0260] "Information processing means" refers to functions, including artificial intelligence, used to analyze collected data and extract necessary information.

[0261] A "rescue procedure" is a set of steps for planning the priorities and flow of various activities in rescuing disaster victims.

[0262] "Communication means" refers to functions that include communication technologies for exchanging information and transmitting instructions between various devices.

[0263] "Geometric analysis" is a technology that analyzes images acquired by cameras and sensors to extract useful information.

[0264] "Speech recognition" is a technology that analyzes audio data to identify linguistic information and the source of sound.

[0265] "Environmental conditions" refer to the situation on site, including geographical information of the affected area and the impact of the disaster.

[0266] "Resource status" refers to the current state of available resources such as personnel, supplies, and equipment.

[0267] This invention is an autonomous disaster relief system in which a server, terminals, and users work together. The server acts as the central control unit, aggregating and analyzing data transmitted from aircraft and ground transport vehicles. Specifically, the server uses a high-performance computer to build an information processing system using a generative AI model. This system excels particularly in image recognition and speech recognition technologies, enabling it to pinpoint the location of disaster victims in real time and analyze the situation in disaster areas.

[0268] The aircraft, acting as terminals, are equipped with high-resolution cameras and infrared sensors, enabling rapid observation of wide areas. This allows the aircraft to collect data from the air and immediately transmit the information to the server. Ground carriers, moving across the land, can collect detailed environmental information and are equipped with sensors to avoid obstacles. These terminals operate based on commands from the server, enabling a rapid response to disaster victims.

[0269] Users can monitor the overall progress of the system via an operating terminal and intervene in rescue operations as needed. Users can view terrain information and analysis data in real time and send new commands to the server based on that information. For example, if the disaster area has very complex terrain or if a large number of victims are detected, users can make immediate decisions and readjust rescue policies.

[0270] As a concrete example, consider a scenario where a large-scale earthquake causes part of a city to collapse. Aircraft scan the affected area from above with thermal cameras, and ground transport vehicles navigate the complex terrain to deliver supplies to victims. This entire process is supported by a system that uses server analysis and allows users to monitor the results in real time. An example of a prompt from the generated AI model is, "Using thermal cameras, how can we locate victims trapped under rubble in the disaster area?" This system enables rapid and effective relief for disaster victims.

[0271] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0272] Step 1:

[0273] The server receives observational data transmitted from aircraft and ground carriers. This input data includes high-resolution images, thermal information from infrared sensors, and information about the local environmental conditions. The server performs a basic validation of this data and prepares it for analysis. Specific actions performed at this stage include data format conversion and priority setting.

[0274] Step 2:

[0275] The server analyzes the received data using a generating AI model. Using the high-resolution images and thermal information received in step 1 as input, it executes image recognition and speech recognition algorithms. The server identifies the location of victims through shape recognition and detects unusual sounds and voices from the audio data. This results in the production of location information and status data for the victims as the output of the analysis.

[0276] Step 3:

[0277] The server develops rescue procedures based on the analysis results. It uses the location and status of victims, as well as the environmental conditions at the site, obtained in Step 2, as input. Based on this, the server determines the priority of rescue operations and plans the supply routes for relief materials. The output generates information regarding rescue procedures and transportation routes.

[0278] Step 4:

[0279] The server transmits specific commands to aircraft and ground transport vehicles based on the established rescue procedures and transport routes. This process includes detailed instructions on how to operate rescue equipment and the delivery schedule for supplies. Terminals receive these commands and begin actual observation and rescue operations.

[0280] Step 5:

[0281] Users monitor ongoing rescue operations using control terminals for real-time monitoring. Using data provided in real-time from the server as input, users can make immediate decisions based on the situation and send new commands to the server. This allows for manual intervention as needed.

[0282] (Application Example 1)

[0283] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0284] In recent years, the frequency of natural disasters has been increasing, and along with this, the importance of prompt and efficient disaster response has been growing. However, conventional disaster support systems often suffer from delays in information acquisition and transmission, and insufficient support for disaster victims. Additionally, it has been difficult to provide dynamic evacuation guidance according to the situation in the disaster area. Therefore, there is a need for a system that can collect accurate information in real time during disasters and quickly conduct rescue operations and evacuation guidance for disaster victims.

[0285] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following respective means.

[0286] In this invention, the server includes means for observing the environment using an aircraft and a ground vehicle, information processing means including an artificial intelligence for analyzing the data acquired from the aircraft and the ground vehicle, means for optimizing the allocation of resources and formulating rescue procedures based on the position and condition of the disaster victims, and means for providing disaster information and evacuation routes using communication means connected to a device that can be carried by citizens. Thereby, prompt and efficient support for disaster victims during disasters and effective evacuation guidance for citizens become possible.

[0287] An "aircraft" is a means of movement that can directly observe the ground surface from the air and quickly collect data over a wide area.

[0288] A "ground vehicle" is a device for collecting detailed environmental data on the ground surface and moving through the disaster area while avoiding obstacles.

[0289] "Information processing means" is a system including an artificial intelligence for analyzing the data acquired from an aircraft and a ground vehicle and supporting effective rescue operations.

[0290] "Means for optimizing the allocation of resources and formulating rescue procedures" is a method for planning prompt and appropriate rescue actions by efficiently allocating available resources based on the position and condition of the disaster victims.

[0291] "Communication methods" refer to technologies that provide real-time disaster information and evacuation routes to disaster victims and the general public using mobile devices that they can access.

[0292] This invention implements an autonomous disaster relief system in which servers, terminals, and users each play their respective roles, enabling rapid and accurate assistance to disaster victims during a disaster.

[0293] The server receives environmental data transmitted from aircraft and ground transport vehicles and analyzes it in real time using information processing tools. Specifically, it uses artificial intelligence to perform image and voice recognition to identify the location and condition of disaster victims. The server uses the Google Maps API to update map information of the disaster area and formulate effective rescue procedures. Furthermore, it transmits optimized instructions to aircraft and ground transport vehicles to deploy the necessary actions.

[0294] The aircraft, acting as the terminal, is equipped with high-resolution cameras and infrared sensors to conduct wide-area observations from above. This allows for a detailed understanding of the collapsed areas that are difficult for ground carriers to access. The ground carriers are designed to collect detailed information on the ground and move safely through the disaster area, operating reactively based on instructions from the server.

[0295] Users can receive the latest notifications regarding disaster information and evacuation routes from the server using their smartphones or other mobile devices. Important information is immediately conveyed via push notifications, and detailed information about affected areas is also provided through analysis using generative AI models. Based on this information, users can take swift evacuation action.

[0296] As a concrete example, in the event of a large-scale earthquake in an urban area, the server creates a detailed map of the affected area and analyzes information on the safety of victims. Citizens receive this information via their mobile devices and are guided to safe evacuation routes. An example of a prompt message would be a response to a request such as, "Based on the current evacuation information, please provide the optimal evacuation route."

[0297] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0298] Step 1:

[0299] The server receives real-time sensor data transmitted from aircraft and ground carriers. This data includes image data, infrared analysis data, and audio data. The received data is first temporarily stored in a database and then pre-processed by an artificial intelligence module.

[0300] Step 2:

[0301] The server uses a generative AI model to analyze the received image data and identify the location and condition of the victims. This process applies pattern recognition algorithms to detect human shapes and movements from the images. The input is image data, and the output is the location coordinates of the victims.

[0302] Step 3:

[0303] The server simultaneously analyzes audio data and performs speech recognition processing to identify the voices of disaster victims. Here, the audio waveform is processed as input, and location information of potential victims is output. The results of the audio analysis are combined with the results of the image analysis to improve accuracy.

[0304] Step 4:

[0305] The server uses the Google Maps API to generate a map of the disaster area based on the analysis results from the generated AI model, and overlays the locations of the victims. At this stage, the input is the analyzed location data of the victims, and the output is a detailed map of the disaster area.

[0306] Step 5:

[0307] The user receives the map and the latest evacuation information from the server using a smartphone or smart glasses. The app running on the user device receives real-time information through the push notification function. The input information is command data based on the analysis results of the generation AI model, and the output is the evacuation route guidance as visual information.

[0308] Step 6:

[0309] The aircraft and ground transportation vehicles, which are terminals, perform autonomous movement based on the commands from the server and the position of the disaster victims, and start rescue activities according to the situation in the disaster area as needed. Here, real-time position information is used as the input, and actions such as supplying materials and starting rescue activities are output.

[0310] Furthermore, an emotion engine for estimating the user's emotions may be combined. That is, the specific processing unit 290 may estimate the user's emotions using the emotion identification model 59 and perform specific processing using the user's emotions.

[0311] The present invention is for further improving the efficiency of disaster relief activities by combining an emotion engine with an environmental observation system using aircraft and ground transportation vehicles. In this system, artificial intelligence is utilized as the data processing means, and it is possible to grasp the emotional states of users and disaster victims through the emotion engine.

[0312] The role of the server

[0313] The server is a management center for a vast amount of data. After receiving data from the terminals, it first analyzes the environmental information. Using image analysis algorithms and voice recognition technologies, it identifies the positions and states of the disaster victims. This data is used for formulating rescue procedures and optimizing the allocation of resources. Also, the emotion engine analyzes the received voice data and image data to recognize the user's emotional state, and adjusts the rescue procedures based on particularly important emotional changes. Through emotion recognition, the server can make more appropriate judgments for conducting rescue activities.

[0314] Terminal role

[0315] As terminals, aircraft and ground carriers perform specific actions in real time based on instructions from the server. Aircraft are responsible for wide-area observation, continuously collecting data with thermal sensors and high-resolution cameras. Meanwhile, ground carriers acquire detailed environmental information and the location of disaster victims, while simultaneously attempting to communicate with them. For example, by using an emotion engine, it is possible to select an appropriate communication method if a disaster victim is experiencing fear. Furthermore, when delivering relief supplies, consideration is given to the emotions of the disaster victims.

[0316] User roles

[0317] Users can monitor rescue operations via their devices and refer to sentiment analysis data to conduct smoother rescue operations. In particular, sentiment data is fed back into the user interface, serving as a decision-making support tool for users. Specifically, it can present users with countermeasures to alleviate the anxiety and panic of disaster victims, enabling more effective rescue operations. This information is reported to the server and reflected in future rescue operations.

[0318] As a result, this system, which incorporates an emotional engine, provides deeper insights that could not be obtained in conventional rescue operations, enabling swift and accurate rescue efforts.

[0319] The following describes the processing flow.

[0320] Step 1:

[0321] The server continuously receives data from aircraft and ground carriers equipped with various sensors. The data includes a wide range of information, such as images, audio, and temperature information of the external environment.

[0322] Step 2:

[0323] The aircraft, acting as the terminal, will photograph a wide area of ​​the disaster zone from above and provide real-time data. The aircraft will fly autonomously to cover the entire disaster area and transmit the observation data to the server.

[0324] Step 3:

[0325] The ground-based transporter, acting as the terminal, avoids obstacles on the ground and transmits acquired audio and proximity sensor data to the server. This allows for high-precision detection of sounds emitted by people trapped under rubble.

[0326] Step 4:

[0327] The server processes incoming data instantly, using AI image analysis and speech recognition to determine the location and condition of victims. Furthermore, an emotion engine analyzes the emotions from the victims' voices to identify their emotional state.

[0328] Step 5:

[0329] The server develops rescue procedures based on the physical location and psychological state of the victims. If the emotional state is urgent, this is reflected in the rescue plan, and the procedures are adjusted to ensure a rapid response.

[0330] Step 6:

[0331] Based on the formulated plan, the server sends specific instructions to the terminals. Aircraft are instructed on the location and order of dropping relief supplies, and ground transport aircraft are instructed on the delivery of supplies to victims and appropriate communication methods.

[0332] Step 7:

[0333] Users have the ability to review sentiment analysis data sent from the server and adjust rescue priorities in real time. They also communicate with discovered victims sequentially and provide support according to suggestions from the sentiment engine.

[0334] Step 8:

[0335] Users can send feedback to the server as needed regarding ongoing rescue operations, providing insights to improve the quality of future operations. This feedback is recorded and analyzed by the system and used to improve the efficiency of future rescue operations.

[0336] (Example 2)

[0337] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0338] Conventional disaster relief systems were somewhat effective in locating the position and condition of victims. However, they lacked the ability to adaptively change rescue procedures based on changes in the emotional state of victims, and the effective allocation of rescue resources was insufficient. As a result, rapid and accurate rescue of victims was difficult.

[0339] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0340] In this invention, the server includes means for analyzing data acquired from aircraft and ground transport vehicles to understand the emotional state of disaster victims, means for optimizing the allocation of rescue resources based on the emotional state, and means for formulating rescue procedures and transmitting instructions for them. This enables flexible adjustment of rescue procedures according to the emotional state of disaster victims and more efficient allocation of rescue resources.

[0341] An "aircraft" is a means of transportation used for observation and data collection by flying through the air.

[0342] A "ground transport device" is a means of transport used to collect and transmit environmental data and disaster victim information while moving along the ground surface.

[0343] "Means of observing the environment" refers to devices, including sensors and cameras, that are mounted on aircraft or ground transport vehicles and are necessary for detailed analysis of the surrounding environment.

[0344] A "data processing system" is a system that includes machine learning algorithms for analyzing acquired data and making decisions based on that information.

[0345] "Analyzing emotional states" is the process of identifying and understanding the emotional responses of disaster victims from audio and image data.

[0346] "Optimizing resource allocation" is the process of effectively utilizing limited rescue resources and quickly distributing them to the places and situations where they are most needed.

[0347] "Means for formulating rescue procedures" refers to the function of formulating a specific rescue action plan based on the analysis results and building a strategy for putting it into action.

[0348] This invention is designed to enhance the effectiveness of disaster relief operations based on an environmental observation system utilizing aircraft and ground carriers. This system uses aircraft and ground carriers equipped with various sensors as hardware, and machine learning algorithms running on a server.

[0349] Server Role

[0350] The server plays a central role in receiving and analyzing data transmitted from terminals. Specifically, it uses machine learning software to process image and audio data collected from aircraft and ground transport vehicles. Widely used machine learning libraries are utilized for image processing to identify the location and condition of victims. For audio data, sentiment analysis techniques are used to determine the emotional state of victims.

[0351] Terminal role

[0352] The aircraft, acting as the terminal, is equipped with high-resolution cameras and thermal sensors to conduct wide-area environmental observations. The aircraft collects environmental information over a wide area and transmits this data to a server. Ground carriers are responsible for approaching disaster victims and acquiring detailed environmental and audio information. This enables direct communication on-site and allows for adaptive responses based on emotion analysis.

[0353] User roles

[0354] Users refer to data from the server and manage rescue operations based on the displayed situation. The user interface reflects the emotional state and location information of victims in real time, enabling rapid decision-making based on this information. This information is also fed back after the rescue operation is completed and used to improve future responses.

[0355] Specific examples and prompt statements

[0356] One specific example is the use of this system to conduct rapid rescue operations in areas where flooding is anticipated. In this case, an aircraft would take extensive aerial photographs of the affected area, and a server would use these to identify the location and emotional state of the victims. Ground carriers would then send reassuring messages to the identified victims.

[0357] Examples of prompts to input into a generative AI model:

[0358] "Explain how to identify victims in flood-affected areas using sentiment-based data and rapidly allocate appropriate relief resources."

[0359] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0360] Step 1:

[0361] The terminal uses a high-resolution camera and thermal sensors mounted on the aircraft to collect environmental information for a designated area. The input consists of image data and thermal data acquired from the air, and the output is collected environmental data. This data provides fundamental information necessary to understand the overall situation of the region. Specifically, the aircraft follows a designated flight path and periodically records data.

[0362] Step 2:

[0363] The terminal collects detailed information about the vicinity of disaster victims via a ground transport device. Inputs include voice data and more detailed environmental data, and output is the result of on-site communication. The ground transport device understands the emotional state of disaster victims by directly communicating with them via voice. Specific actions include the ground transport device moving closer to the disaster victims and attempting to communicate using microphones and speakers.

[0364] Step 3:

[0365] The server analyzes all data transmitted from aircraft and ground transport vehicles. Inputs include collected image data, thermal data, and audio data, while outputs include the location information and emotional state analysis results of victims. The server uses machine learning algorithms to analyze images, identify the location of victims, and recognize emotional states based on audio data. Specific operations include training a model using the dataset and making predictions through real-time processing.

[0366] Step 4:

[0367] The server develops rescue procedures and creates an optimal resource allocation plan based on the analysis results. Inputs are location information and emotional state, while outputs are specific rescue procedures and resource allocation plans. The developed procedures are flexibly adjusted to reflect the emotional state of the victims, ensuring appropriate actions are taken according to the situation. Specific operations include prompt generation using a generative AI model and scenario simulation.

[0368] Step 5:

[0369] Users monitor rescue procedures generated by the server and make modifications as needed. Inputs are detailed rescue procedures and real-time feedback from the field, while outputs are the final execution plan. Users refer to the plan and make necessary decisions through the interface. Specific actions include sending instructions to the rescue team and checking and updating field data.

[0370] (Application Example 2)

[0371] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0372] Current disaster relief systems can grasp the physical condition and location of victims, but they struggle to monitor their emotional state in real time and provide appropriate support accordingly. Furthermore, in elderly care support, the lack of consideration for emotional states makes it difficult for caregivers to quickly and accurately determine the necessary support.

[0373] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0374] In this invention, the server includes means for observing the environment using an aircraft and a ground transporter, data processing means including artificial intelligence for analyzing data acquired from the aircraft and ground transporter, and processing means for analyzing emotional information and determining appropriate responses and support based on the emotional state of disaster victims. This enables prompt and accurate support and care that takes into account the emotional state of disaster victims and the elderly.

[0375] An "aircraft" is a flying device used for aerial observation and data collection.

[0376] A "ground transport vehicle" is a vehicle used for movement near the Earth's surface, transporting materials, and conducting detailed observations.

[0377] "Means of observing the environment" refers to methods of acquiring surrounding environmental data using aircraft or ground-based transport vehicles.

[0378] "Data processing means" refers to methods that use artificial intelligence to analyze data acquired from aircraft and ground transport vehicles.

[0379] "Artificial intelligence" is a program that analyzes data and automatically makes judgments and decisions using specific algorithms.

[0380] "Emotional information" refers to data on the emotional state of disaster victims in real time, as well as the results of its analysis.

[0381] An "emotion engine" is a technology that analyzes the voice and image data of disaster victims to recognize their emotional state.

[0382] "Means for optimizing resource allocation" refers to methods for determining rescue procedures and support based on acquired data and emotional information.

[0383] "Rescue procedures" are the procedures established to carry out appropriate rescue operations according to the circumstances of the disaster.

[0384] "Care support" refers to support activities aimed at providing appropriate assistance according to the emotional state of elderly individuals.

[0385] This invention is a system that uses aircraft and ground transport vehicles to observe the environment and employs artificial intelligence as a data processing tool to grasp the emotional state of disaster victims and the elderly in real time, thereby enabling optimal support activities.

[0386] The server receives environmental data transmitted from aircraft and ground transport vehicles and analyzes it using artificial intelligence. This allows it to understand the location and physical condition of disaster victims, and recognize their emotional state using an emotion engine. This process utilizes image analysis libraries and speech recognition technologies (e.g., OpenCV and TensorFlow). Emotional information is used to formulate rescue procedures and contribute to the optimization of support activities. Based on the accumulated data, the server improves the efficiency of future rescue operations.

[0387] The aircraft and ground carriers, acting as terminals, are responsible for wide-area observation and detailed data acquisition. The aircraft primarily collects environmental information from the air, while the ground carriers are configured to capture detailed ground information and emotional changes. The ground carriers communicate with disaster victims and adjust appropriate support activities according to their emotional state. In this process, emotional recognition is used to select responses that will make disaster victims feel safe.

[0388] Users can view aggregated data using a dedicated interface and check the emotional state of disaster victims and the elderly in real time. This supports rapid decision-making on-site and enables the sending of appropriate instructions to the terminal. In this process, the emotional data and recommended actions presented to the user are updated in real time by a generative AI model.

[0389] As a concrete example, when this system is applied to a nursing home, it can monitor the emotional state of elderly residents and automatically provide relaxing music content when it detects that they are feeling lonely. This can reduce the psychological burden on the elderly and create a more comfortable living environment.

[0390] An example of a prompt for a generative AI model is: "Recommend appropriate music content to alleviate feelings of loneliness in the elderly. If the emotional state is determined to be 'lonely,' please provide specific song titles and the reasons for their selection."

[0391] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0392] Step 1:

[0393] The server receives real-time environmental data transmitted from aircraft and ground transport vehicles. This includes high-resolution image and audio data. Based on this input data, artificial intelligence begins initial data processing.

[0394] Step 2:

[0395] The server uses the received image data to identify the locations of disaster victims and elderly individuals using an image analysis library (e.g., OpenCV). Additionally, it utilizes speech recognition technology (e.g., TensorFlow) to extract meaning and emotional information from the audio data. This process generates detailed data regarding the location and condition of disaster victims.

[0396] Step 3:

[0397] The ground transporter, acting as a terminal, collects detailed ground information and utilizes an emotion engine to recognize the detailed emotional state of disaster victims. The analyzed emotional information is sent to a server, and an appropriate method of communication with the victims is selected. In this process, data based on changes in emotion is generated.

[0398] Step 4:

[0399] The server integrates all analyzed data and optimizes rescue procedures and support methods using a generative AI model. It provides users with support suggestions and action guidelines based on their emotional state. This process generates prompts and determines the specific details of the support.

[0400] Step 5:

[0401] Based on data provided by the server, the user selects an appropriate action and sends instructions to the device. For example, if loneliness in an elderly person is detected, a specific instruction is created to provide relaxing music. The device then carries out the support activity according to the user's instructions.

[0402] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0403] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0404] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0405] [Third Embodiment]

[0406] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0407] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0408] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0409] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0410] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0411] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0412] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0413] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0414] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0415] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0416] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0417] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0418] The following describes a specific process for implementing the autonomous disaster relief system of the present invention.

[0419] Server Role

[0420] The server is a central hub for managing the vast amount of data transmitted from aircraft and ground transport vehicles during disasters. Specifically, the server rapidly analyzes the received real-time data and utilizes artificial intelligence modules to identify victims and create maps of the affected areas. In the analysis, image recognition technology is used to pinpoint the location of victims, and voice recognition technology is used to detect their voices and movements. This enables more accurate rescue operations. Furthermore, the server formulates efficient rescue procedures and transmits this information as commands to aircraft and ground transport vehicles.

[0421] Terminal role

[0422] Aircraft and ground carriers, acting as terminals, utilize their different capabilities to cover disaster areas. Aircraft, with their ability to move quickly over wide areas, primarily conduct aerial observations. High-resolution cameras and infrared sensors are used for data collection, and this collected information is immediately transmitted to the server. Ground carriers acquire detailed environmental data and move through the disaster area while avoiding obstacles. In this way, terminals utilize sensors and communication functions to collect information accurately and quickly, and carry out rescue operations with the support of the server.

[0423] User roles

[0424] Users can monitor the progress of rescue operations through their control terminals and intervene manually as needed. In particular, in areas with complex terrain or where obstacles arise, users can make quick decisions to improve the efficiency of life-saving efforts. Users can also adjust resource allocation using analysis results from the server, allowing for a rapid response to high-priority rescue missions.

[0425] Specific example

[0426] For example, if a large-scale earthquake causes part of a city to collapse, the server creates a map of the affected area and identifies the locations of victims. When an aircraft flies over the collapsed buildings and uses thermal cameras to locate victims under the rubble, ground transport aircraft head to that location to deliver relief supplies. Users can monitor this data in real time and send new commands to the server as needed. This system makes it possible to provide rapid and effective support to victims.

[0427] The following describes the processing flow.

[0428] Step 1:

[0429] The server receives sensor data and image data transmitted from terminal aircraft and ground transport vehicles. This data is important as basic information for understanding the current situation in the disaster-stricken areas.

[0430] Step 2:

[0431] The aircraft, acting as the terminal, flies within a designated patrol area, using its onboard cameras and infrared sensors to collect real-time video and thermal information. The collected data is immediately transmitted to a server.

[0432] Step 3:

[0433] The ground transporter, acting as the terminal, moves across the ground while avoiding obstacles, and uses voice and heat sensors to locate potential locations of victims buried under rubble, detecting their voices and body temperature. This information is also immediately transmitted to the server.

[0434] Step 4:

[0435] The server uses image recognition algorithms based on the received data to identify individuals and locations that may be victims of the disaster. This process analyzes the obtained thermal and audio data to prioritize the identification of high-risk areas.

[0436] Step 5:

[0437] The server develops rescue procedures based on the location information of identified victims. These procedures include determining the type and quantity of relief supplies needed and optimizing resource allocation, such as which terminals should use which resources.

[0438] Step 6:

[0439] Based on the established rescue procedures, the server transmits specific action instructions to the aircraft and ground transport vehicles, which act as terminals. The aircraft airdrops the relief supplies to the designated locations, and the ground transport vehicles move to the detailed locations to distribute the supplies.

[0440] Step 7:

[0441] Users use dedicated control terminals to monitor information from the server and track the progress of rescue operations. They can also send new commands to the server as needed to further adjust the rescue efforts.

[0442] Step 8:

[0443] Users monitor the progress of rescue operations and provide feedback on the procedures developed by the server. This feedback will be used to determine future resource allocation and prioritization.

[0444] (Example 1)

[0445] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0446] When a disaster strikes, it is crucial to quickly and accurately ascertain the location and condition of victims and to conduct effective rescue operations. Current systems lack the accuracy of real-time information analysis and the immediacy of rescue instructions, resulting in reduced efficiency in rescue efforts. Furthermore, a system is needed that can flexibly respond to widespread changes in the situation and formulate optimal rescue procedures. Therefore, key challenges in this technological field are improving the accuracy of information analysis and achieving optimal resource allocation according to the disaster situation.

[0447] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0448] In this invention, the server includes means for conducting wide-area observations using aircraft and ground transporters, information processing means including artificial intelligence for analyzing information collected from the aircraft and ground transporters in real time, and means for formulating rescue procedures and setting priorities for rescue operations based on the location and condition of victims and the situation in the affected area. This enables effective rescue operations through the rapid identification of victims and appropriate allocation of resources.

[0449] An "aircraft" is a means of air transport, such as airplanes and drones, that can fly through the air and conduct observations over a wide area.

[0450] A "ground transport vehicle" is a mechanical device that moves along the ground and can conduct observations while collecting detailed environmental information.

[0451] "Wide-area observation" refers to observational activities that cover a large area and acquire information quickly.

[0452] "Information processing means" refers to functions, including artificial intelligence, used to analyze collected data and extract necessary information.

[0453] A "rescue procedure" is a set of steps for planning the priorities and flow of various activities in rescuing disaster victims.

[0454] "Communication means" refers to functions that include communication technologies for exchanging information and transmitting instructions between various devices.

[0455] "Geometric analysis" is a technology that analyzes images acquired by cameras and sensors to extract useful information.

[0456] "Speech recognition" is a technology that analyzes audio data to identify linguistic information and the source of sound.

[0457] "Environmental conditions" refer to the situation on site, including geographical information of the affected area and the impact of the disaster.

[0458] "Resource status" refers to the current state of available resources such as personnel, supplies, and equipment.

[0459] This invention is an autonomous disaster relief system in which a server, terminals, and users work together. The server acts as the central control unit, aggregating and analyzing data transmitted from aircraft and ground transport vehicles. Specifically, the server uses a high-performance computer to build an information processing system using a generative AI model. This system excels particularly in image recognition and speech recognition technologies, enabling it to pinpoint the location of disaster victims in real time and analyze the situation in disaster areas.

[0460] The aircraft, acting as terminals, are equipped with high-resolution cameras and infrared sensors, enabling rapid observation of wide areas. This allows the aircraft to collect data from the air and immediately transmit the information to the server. Ground carriers, moving across the land, can collect detailed environmental information and are equipped with sensors to avoid obstacles. These terminals operate based on commands from the server, enabling a rapid response to disaster victims.

[0461] Users can monitor the overall progress of the system via an operating terminal and intervene in rescue operations as needed. Users can view terrain information and analysis data in real time and send new commands to the server based on that information. For example, if the disaster area has very complex terrain or if a large number of victims are detected, users can make immediate decisions and readjust rescue policies.

[0462] As a concrete example, consider a scenario where a large-scale earthquake causes part of a city to collapse. Aircraft scan the affected area from above with thermal cameras, and ground transport vehicles navigate the complex terrain to deliver supplies to victims. This entire process is supported by a system that uses server analysis and allows users to monitor the results in real time. An example of a prompt from the generated AI model is, "Using thermal cameras, how can we locate victims trapped under rubble in the disaster area?" This system enables rapid and effective relief for disaster victims.

[0463] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0464] Step 1:

[0465] The server receives observational data transmitted from aircraft and ground carriers. This input data includes high-resolution images, thermal information from infrared sensors, and information about the local environmental conditions. The server performs a basic validation of this data and prepares it for analysis. Specific actions performed at this stage include data format conversion and priority setting.

[0466] Step 2:

[0467] The server analyzes the received data using a generating AI model. Using the high-resolution images and thermal information received in step 1 as input, it executes image recognition and speech recognition algorithms. The server identifies the location of victims through shape recognition and detects unusual sounds and voices from the audio data. This results in the production of location information and status data for the victims as the output of the analysis.

[0468] Step 3:

[0469] The server develops rescue procedures based on the analysis results. It uses the location and status of victims, as well as the environmental conditions at the site, obtained in Step 2, as input. Based on this, the server determines the priority of rescue operations and plans the supply routes for relief materials. The output generates information regarding rescue procedures and transportation routes.

[0470] Step 4:

[0471] The server transmits specific commands to aircraft and ground transport vehicles based on the established rescue procedures and transport routes. This process includes detailed instructions on how to operate rescue equipment and the delivery schedule for supplies. Terminals receive these commands and begin actual observation and rescue operations.

[0472] Step 5:

[0473] Users monitor ongoing rescue operations using control terminals for real-time monitoring. Using data provided in real-time from the server as input, users can make immediate decisions based on the situation and send new commands to the server. This allows for manual intervention as needed.

[0474] (Application Example 1)

[0475] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0476] In recent years, the frequency of natural disasters has increased, highlighting the growing importance of rapid and efficient disaster response. However, conventional disaster relief systems often suffer from delays in information acquisition and transmission, resulting in insufficient support for disaster victims. Furthermore, dynamic evacuation guidance tailored to the specific conditions of the affected area has been difficult. Therefore, there is a need for a system that can collect accurate information in real time during a disaster and quickly provide rescue operations and evacuation guidance to victims.

[0477] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0478] In this invention, the server includes means for observing the environment using aircraft and ground transporters, information processing means including artificial intelligence for analyzing data acquired from the aircraft and ground transporters, means for optimizing resource allocation and formulating rescue procedures based on the location and condition of disaster victims, and means for providing disaster information and evacuation routes using communication means connected to a device that can be carried by citizens. This enables rapid and efficient support for disaster victims and effective evacuation guidance for citizens during disasters.

[0479] An "aircraft" is a means of transportation that can directly observe the Earth's surface from above and rapidly collect data over a wide area.

[0480] A "ground transport device" is a device used to collect detailed environmental data from the ground surface and move through disaster-stricken areas while avoiding obstacles.

[0481] "Information processing means" refers to a system that includes artificial intelligence to analyze data acquired from aircraft and ground transport vehicles and support effective rescue operations.

[0482] "Means for optimizing resource allocation and formulating rescue procedures" refers to methods for planning rapid and appropriate rescue actions by efficiently allocating available resources based on the location and condition of the victims.

[0483] "Communication methods" refer to technologies that provide real-time disaster information and evacuation routes to disaster victims and the general public using mobile devices that they can access.

[0484] This invention implements an autonomous disaster relief system in which servers, terminals, and users each play their respective roles, enabling rapid and accurate assistance to disaster victims during a disaster.

[0485] The server receives environmental data transmitted from aircraft and ground transport vehicles and analyzes it in real time using information processing tools. Specifically, it uses artificial intelligence to perform image and voice recognition to identify the location and condition of disaster victims. The server uses the Google Maps API to update map information of the disaster area and formulate effective rescue procedures. Furthermore, it transmits optimized instructions to aircraft and ground transport vehicles to deploy the necessary actions.

[0486] The aircraft, acting as the terminal, is equipped with high-resolution cameras and infrared sensors to conduct wide-area observations from above. This allows for a detailed understanding of the collapsed areas that are difficult for ground carriers to access. The ground carriers are designed to collect detailed information on the ground and move safely through the disaster area, operating reactively based on instructions from the server.

[0487] Users can receive the latest notifications regarding disaster information and evacuation routes from the server using their smartphones or other mobile devices. Important information is immediately conveyed via push notifications, and detailed information about affected areas is also provided through analysis using generative AI models. Based on this information, users can take swift evacuation action.

[0488] As a concrete example, in the event of a large-scale earthquake in an urban area, the server creates a detailed map of the affected area and analyzes information on the safety of victims. Citizens receive this information via their mobile devices and are guided to safe evacuation routes. An example of a prompt message would be a response to a request such as, "Based on the current evacuation information, please provide the optimal evacuation route."

[0489] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0490] Step 1:

[0491] The server receives real-time sensor data transmitted from aircraft and ground carriers. This data includes image data, infrared analysis data, and audio data. The received data is first temporarily stored in a database and then pre-processed by an artificial intelligence module.

[0492] Step 2:

[0493] The server uses a generative AI model to analyze the received image data and identify the location and condition of the victims. This process applies pattern recognition algorithms to detect human shapes and movements from the images. The input is image data, and the output is the location coordinates of the victims.

[0494] Step 3:

[0495] The server simultaneously analyzes audio data and performs speech recognition processing to identify the voices of disaster victims. Here, the audio waveform is processed as input, and location information of potential victims is output. The results of the audio analysis are combined with the results of the image analysis to improve accuracy.

[0496] Step 4:

[0497] The server uses the Google Maps API to generate a map of the disaster area based on the analysis results from the generated AI model, and overlays the locations of the victims. At this stage, the input is the analyzed location data of the victims, and the output is a detailed map of the disaster area.

[0498] Step 5:

[0499] Users receive maps and the latest evacuation information from a server using their smartphones or smart glasses. The app running on the user's device receives real-time information via push notifications. The input information is command data based on the analysis results of a generated AI model, and the output is evacuation route guidance as visual information.

[0500] Step 6:

[0501] The aircraft and ground transport units, acting as terminals, autonomously move based on the location of disaster victims, receiving commands from the server, and initiate rescue operations as needed, according to the situation in the disaster area. Real-time location information is used as input, and actions such as supplying materials and initiating rescue operations are output.

[0502] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0503] This invention aims to further improve the efficiency of disaster relief activities by combining an emotion engine with an environmental observation system using aircraft and ground transport vehicles. This system utilizes artificial intelligence as a data processing means and can understand the emotional state of users and disaster victims through the emotion engine.

[0504] Server Role

[0505] The server acts as a management center for vast amounts of data. After receiving data from terminals, it first analyzes environmental information. Using image analysis algorithms and speech recognition technology, it identifies the location and condition of disaster victims. This data is used to formulate rescue procedures and optimize resource allocation. The emotion engine also analyzes received audio and image data to recognize the user's emotional state and adjusts rescue procedures based on particularly important emotional changes. Emotion recognition allows the server to make more appropriate decisions for rescue operations.

[0506] Terminal role

[0507] As terminals, aircraft and ground carriers perform specific actions in real time based on instructions from the server. Aircraft are responsible for wide-area observation, continuously collecting data with thermal sensors and high-resolution cameras. Meanwhile, ground carriers acquire detailed environmental information and the location of disaster victims, while simultaneously attempting to communicate with them. For example, by using an emotion engine, it is possible to select an appropriate communication method if a disaster victim is experiencing fear. Furthermore, when delivering relief supplies, consideration is given to the emotions of the disaster victims.

[0508] User roles

[0509] Users can monitor rescue operations via their devices and refer to sentiment analysis data to conduct smoother rescue operations. In particular, sentiment data is fed back into the user interface, serving as a decision-making support tool for users. Specifically, it can present users with countermeasures to alleviate the anxiety and panic of disaster victims, enabling more effective rescue operations. This information is reported to the server and reflected in future rescue operations.

[0510] As a result, this system, which incorporates an emotional engine, provides deeper insights that could not be obtained in conventional rescue operations, enabling swift and accurate rescue efforts.

[0511] The following describes the processing flow.

[0512] Step 1:

[0513] The server continuously receives data from aircraft and ground carriers equipped with various sensors. The data includes a wide range of information, such as images, audio, and temperature information of the external environment.

[0514] Step 2:

[0515] The aircraft, acting as the terminal, will photograph a wide area of ​​the disaster zone from above and provide real-time data. The aircraft will fly autonomously to cover the entire disaster area and transmit the observation data to the server.

[0516] Step 3:

[0517] The ground-based transporter, acting as the terminal, avoids obstacles on the ground and transmits acquired audio and proximity sensor data to the server. This allows for high-precision detection of sounds emitted by people trapped under rubble.

[0518] Step 4:

[0519] The server processes incoming data instantly, using AI image analysis and speech recognition to determine the location and condition of victims. Furthermore, an emotion engine analyzes the emotions from the victims' voices to identify their emotional state.

[0520] Step 5:

[0521] The server develops rescue procedures based on the physical location and psychological state of the victims. If the emotional state is urgent, this is reflected in the rescue plan, and the procedures are adjusted to ensure a rapid response.

[0522] Step 6:

[0523] Based on the formulated plan, the server sends specific instructions to the terminals. Aircraft are instructed on the location and order of dropping relief supplies, and ground transport aircraft are instructed on the delivery of supplies to victims and appropriate communication methods.

[0524] Step 7:

[0525] Users have the ability to review sentiment analysis data sent from the server and adjust rescue priorities in real time. They also communicate with discovered victims sequentially and provide support according to suggestions from the sentiment engine.

[0526] Step 8:

[0527] Users can send feedback to the server as needed regarding ongoing rescue operations, providing insights to improve the quality of future operations. This feedback is recorded and analyzed by the system and used to improve the efficiency of future rescue operations.

[0528] (Example 2)

[0529] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0530] Conventional disaster relief systems were somewhat effective in locating the position and condition of victims. However, they lacked the ability to adaptively change rescue procedures based on changes in the emotional state of victims, and the effective allocation of rescue resources was insufficient. As a result, rapid and accurate rescue of victims was difficult.

[0531] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0532] In this invention, the server includes means for analyzing data acquired from aircraft and ground transport vehicles to understand the emotional state of disaster victims, means for optimizing the allocation of rescue resources based on the emotional state, and means for formulating rescue procedures and transmitting instructions for them. This enables flexible adjustment of rescue procedures according to the emotional state of disaster victims and more efficient allocation of rescue resources.

[0533] An "aircraft" is a means of transportation used for observation and data collection by flying through the air.

[0534] A "ground transport device" is a means of transport used to collect and transmit environmental data and disaster victim information while moving along the ground surface.

[0535] "Means of observing the environment" refers to devices, including sensors and cameras, that are mounted on aircraft or ground transport vehicles and are necessary for detailed analysis of the surrounding environment.

[0536] A "data processing system" is a system that includes machine learning algorithms for analyzing acquired data and making decisions based on that information.

[0537] "Analyzing emotional states" is the process of identifying and understanding the emotional responses of disaster victims from audio and image data.

[0538] "Optimizing resource allocation" is the process of effectively utilizing limited rescue resources and quickly distributing them to the places and situations where they are most needed.

[0539] "Means for formulating rescue procedures" refers to the function of formulating a specific rescue action plan based on the analysis results and building a strategy for putting it into action.

[0540] This invention is designed to enhance the effectiveness of disaster relief operations based on an environmental observation system utilizing aircraft and ground carriers. This system uses aircraft and ground carriers equipped with various sensors as hardware, and machine learning algorithms running on a server.

[0541] Server Role

[0542] The server plays a central role in receiving and analyzing data transmitted from terminals. Specifically, it uses machine learning software to process image and audio data collected from aircraft and ground transport vehicles. Widely used machine learning libraries are utilized for image processing to identify the location and condition of victims. For audio data, sentiment analysis techniques are used to determine the emotional state of victims.

[0543] Terminal role

[0544] The aircraft, acting as the terminal, is equipped with high-resolution cameras and thermal sensors to conduct wide-area environmental observations. The aircraft collects environmental information over a wide area and transmits this data to a server. Ground carriers are responsible for approaching disaster victims and acquiring detailed environmental and audio information. This enables direct communication on-site and allows for adaptive responses based on emotion analysis.

[0545] User roles

[0546] Users refer to data from the server and manage rescue operations based on the displayed situation. The user interface reflects the emotional state and location information of victims in real time, enabling rapid decision-making based on this information. This information is also fed back after the rescue operation is completed and used to improve future responses.

[0547] Specific examples and prompt statements

[0548] One specific example is the use of this system to conduct rapid rescue operations in areas where flooding is anticipated. In this case, an aircraft would take extensive aerial photographs of the affected area, and a server would use these to identify the location and emotional state of the victims. Ground carriers would then send reassuring messages to the identified victims.

[0549] Examples of prompts to input into a generative AI model:

[0550] "Explain how to identify victims in flood-affected areas using sentiment-based data and rapidly allocate appropriate relief resources."

[0551] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0552] Step 1:

[0553] The terminal uses a high-resolution camera and thermal sensors mounted on the aircraft to collect environmental information for a designated area. The input consists of image data and thermal data acquired from the air, and the output is collected environmental data. This data provides fundamental information necessary to understand the overall situation of the region. Specifically, the aircraft follows a designated flight path and periodically records data.

[0554] Step 2:

[0555] The terminal collects detailed information about the vicinity of disaster victims via a ground transport device. Inputs include voice data and more detailed environmental data, and output is the result of on-site communication. The ground transport device understands the emotional state of disaster victims by directly communicating with them via voice. Specific actions include the ground transport device moving closer to the disaster victims and attempting to communicate using microphones and speakers.

[0556] Step 3:

[0557] The server analyzes all data transmitted from aircraft and ground transport vehicles. Inputs include collected image data, thermal data, and audio data, while outputs include the location information and emotional state analysis results of victims. The server uses machine learning algorithms to analyze images, identify the location of victims, and recognize emotional states based on audio data. Specific operations include training a model using the dataset and making predictions through real-time processing.

[0558] Step 4:

[0559] The server develops rescue procedures and creates an optimal resource allocation plan based on the analysis results. Inputs are location information and emotional state, while outputs are specific rescue procedures and resource allocation plans. The developed procedures are flexibly adjusted to reflect the emotional state of the victims, ensuring appropriate actions are taken according to the situation. Specific operations include prompt generation using a generative AI model and scenario simulation.

[0560] Step 5:

[0561] Users monitor rescue procedures generated by the server and make modifications as needed. Inputs are detailed rescue procedures and real-time feedback from the field, while outputs are the final execution plan. Users refer to the plan and make necessary decisions through the interface. Specific actions include sending instructions to the rescue team and checking and updating field data.

[0562] (Application Example 2)

[0563] Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0564] Current disaster relief systems can grasp the physical condition and location of victims, but they struggle to monitor their emotional state in real time and provide appropriate support accordingly. Furthermore, in elderly care support, the lack of consideration for emotional states makes it difficult for caregivers to quickly and accurately determine the necessary support.

[0565] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0566] In this invention, the server includes means for observing the environment using an aircraft and a ground transporter, data processing means including artificial intelligence for analyzing data acquired from the aircraft and ground transporter, and processing means for analyzing emotional information and determining appropriate responses and support based on the emotional state of disaster victims. This enables prompt and accurate support and care that takes into account the emotional state of disaster victims and the elderly.

[0567] An "aircraft" is a flying device used for aerial observation and data collection.

[0568] A "ground transport vehicle" is a vehicle used for movement near the Earth's surface, transporting materials, and conducting detailed observations.

[0569] "Means of observing the environment" refers to methods of acquiring surrounding environmental data using aircraft or ground-based transport vehicles.

[0570] "Data processing means" refers to methods that use artificial intelligence to analyze data acquired from aircraft and ground transport vehicles.

[0571] "Artificial intelligence" is a program that analyzes data and automatically makes judgments and decisions using specific algorithms.

[0572] "Emotional information" refers to data on the emotional state of disaster victims in real time, as well as the results of its analysis.

[0573] An "emotion engine" is a technology that analyzes the voice and image data of disaster victims to recognize their emotional state.

[0574] "Means for optimizing resource allocation" refers to methods for determining rescue procedures and support content based on acquired data and emotional information.

[0575] "Rescue procedures" are the procedures established to carry out appropriate rescue operations according to the circumstances of the disaster.

[0576] "Care support" refers to support activities aimed at providing appropriate assistance according to the emotional state of elderly individuals.

[0577] This invention is a system that uses aircraft and ground transport vehicles to observe the environment and employs artificial intelligence as a data processing tool to grasp the emotional state of disaster victims and the elderly in real time, thereby enabling optimal support activities.

[0578] The server receives environmental data transmitted from aircraft and ground transport vehicles and analyzes it using artificial intelligence. This allows it to understand the location and physical condition of disaster victims, and recognize their emotional state using an emotion engine. This process utilizes image analysis libraries and speech recognition technologies (e.g., OpenCV and TensorFlow). Emotional information is used to formulate rescue procedures and contribute to the optimization of support activities. Based on the accumulated data, the server improves the efficiency of future rescue operations.

[0579] The aircraft and ground carriers, acting as terminals, are responsible for wide-area observation and detailed data acquisition. The aircraft primarily collects environmental information from the air, while the ground carriers are configured to capture detailed ground information and emotional changes. The ground carriers communicate with disaster victims and adjust appropriate support activities according to their emotional state. In this process, emotional recognition is used to select responses that will make disaster victims feel safe.

[0580] Users can view aggregated data using a dedicated interface and check the emotional state of disaster victims and the elderly in real time. This supports rapid decision-making on-site and enables the sending of appropriate instructions to the terminal. In this process, the emotional data and recommended actions presented to the user are updated in real time by a generative AI model.

[0581] As a concrete example, when this system is applied to a nursing home, it can monitor the emotional state of elderly residents and automatically provide relaxing music content when it detects that they are feeling lonely. This can reduce the psychological burden on the elderly and create a more comfortable living environment.

[0582] An example of a prompt for a generative AI model is: "Recommend appropriate music content to alleviate feelings of loneliness in the elderly. If the emotional state is determined to be 'lonely,' please provide specific song titles and the reasons for their selection."

[0583] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0584] Step 1:

[0585] The server receives real-time environmental data transmitted from aircraft and ground transport vehicles. This includes high-resolution image and audio data. Based on this input data, artificial intelligence begins initial data processing.

[0586] Step 2:

[0587] The server uses the received image data to identify the locations of disaster victims and elderly individuals using an image analysis library (e.g., OpenCV). Additionally, it utilizes speech recognition technology (e.g., TensorFlow) to extract meaning and emotional information from the audio data. This process generates detailed data regarding the location and condition of disaster victims.

[0588] Step 3:

[0589] The ground transporter, acting as a terminal, collects detailed ground information and utilizes an emotion engine to recognize the detailed emotional state of disaster victims. The analyzed emotional information is sent to a server, and an appropriate method of communication with the victims is selected. In this process, data based on changes in emotion is generated.

[0590] Step 4:

[0591] The server integrates all analyzed data and optimizes rescue procedures and support methods using a generative AI model. It provides users with support suggestions and action guidelines based on their emotional state. This process generates prompts and determines the specific details of the support.

[0592] Step 5:

[0593] Based on data provided by the server, the user selects an appropriate action and sends instructions to the device. For example, if loneliness in an elderly person is detected, a specific instruction is created to provide relaxing music. The device then carries out the support activity according to the user's instructions.

[0594] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0595] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0596] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0597] [Fourth Embodiment]

[0598] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0599] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0600] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0601] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0602] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0603] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0604] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0605] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0606] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0607] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0608] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0609] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0610] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0611] The following describes a specific process for implementing the autonomous disaster relief system of the present invention.

[0612] Server Role

[0613] The server is a central hub for managing the vast amount of data transmitted from aircraft and ground transport vehicles during disasters. Specifically, the server rapidly analyzes the received real-time data and utilizes artificial intelligence modules to identify victims and create maps of the affected areas. In the analysis, image recognition technology is used to pinpoint the location of victims, and voice recognition technology is used to detect their voices and movements. This enables more accurate rescue operations. Furthermore, the server formulates efficient rescue procedures and transmits this information as commands to aircraft and ground transport vehicles.

[0614] Terminal role

[0615] Aircraft and ground carriers, acting as terminals, utilize their different capabilities to cover disaster areas. Aircraft, with their ability to move quickly over wide areas, primarily conduct aerial observations. High-resolution cameras and infrared sensors are used for data collection, and this collected information is immediately transmitted to the server. Ground carriers acquire detailed environmental data and move through the disaster area while avoiding obstacles. In this way, terminals utilize sensors and communication functions to collect information accurately and quickly, and carry out rescue operations with the support of the server.

[0616] User roles

[0617] Users can monitor the progress of rescue operations through their control terminals and intervene manually as needed. In particular, in areas with complex terrain or where obstacles arise, users can make quick decisions to improve the efficiency of life-saving efforts. Users can also adjust resource allocation using analysis results from the server, allowing for a rapid response to high-priority rescue missions.

[0618] Specific example

[0619] For example, if a large-scale earthquake causes part of a city to collapse, the server creates a map of the affected area and identifies the locations of victims. When an aircraft flies over the collapsed buildings and uses thermal cameras to locate victims under the rubble, ground transport aircraft head to that location to deliver relief supplies. Users can monitor this data in real time and send new commands to the server as needed. This system makes it possible to provide rapid and effective support to victims.

[0620] The following describes the processing flow.

[0621] Step 1:

[0622] The server receives sensor data and image data transmitted from terminal aircraft and ground transport vehicles. This data is important as basic information for understanding the current situation in the disaster-stricken areas.

[0623] Step 2:

[0624] The aircraft, acting as the terminal, flies within a designated patrol area, using its onboard cameras and infrared sensors to collect real-time video and thermal information. The collected data is immediately transmitted to a server.

[0625] Step 3:

[0626] The ground transporter, acting as the terminal, moves across the ground while avoiding obstacles, and uses voice and heat sensors to locate potential locations of victims buried under rubble, detecting their voices and body temperature. This information is also immediately transmitted to the server.

[0627] Step 4:

[0628] The server uses image recognition algorithms based on the received data to identify individuals and locations that may be victims of the disaster. This process analyzes the obtained thermal and audio data to prioritize the identification of high-risk areas.

[0629] Step 5:

[0630] The server develops rescue procedures based on the location information of identified victims. These procedures include determining the type and quantity of relief supplies needed and optimizing resource allocation, such as which terminals should use which resources.

[0631] Step 6:

[0632] Based on the established rescue procedures, the server transmits specific action instructions to the aircraft and ground transport vehicles, which act as terminals. The aircraft airdrops the relief supplies to the designated locations, and the ground transport vehicles move to the detailed locations to distribute the supplies.

[0633] Step 7:

[0634] Users use dedicated control terminals to monitor information from the server and track the progress of rescue operations. They can also send new commands to the server as needed to further adjust the rescue efforts.

[0635] Step 8:

[0636] Users monitor the progress of rescue operations and provide feedback on the procedures developed by the server. This feedback will be used to determine future resource allocation and prioritization.

[0637] (Example 1)

[0638] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0639] When a disaster strikes, it is crucial to quickly and accurately ascertain the location and condition of victims and to conduct effective rescue operations. Current systems lack the accuracy of real-time information analysis and the immediacy of rescue instructions, resulting in reduced efficiency in rescue efforts. Furthermore, a system is needed that can flexibly respond to widespread changes in the situation and formulate optimal rescue procedures. Therefore, key challenges in this technological field are improving the accuracy of information analysis and achieving optimal resource allocation according to the disaster situation.

[0640] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0641] In this invention, the server includes means for conducting wide-area observations using aircraft and ground transporters, information processing means including artificial intelligence for analyzing information collected from the aircraft and ground transporters in real time, and means for formulating rescue procedures and setting priorities for rescue operations based on the location and condition of victims and the situation in the affected area. This enables effective rescue operations through the rapid identification of victims and appropriate allocation of resources.

[0642] An "aircraft" is a means of air transport, such as airplanes and drones, that can fly through the air and conduct observations over a wide area.

[0643] A "ground transport vehicle" is a mechanical device that moves along the ground and can conduct observations while collecting detailed environmental information.

[0644] "Wide-area observation" refers to observational activities that cover a large area and acquire information quickly.

[0645] "Information processing means" refers to functions, including artificial intelligence, used to analyze collected data and extract necessary information.

[0646] A "rescue procedure" is a set of steps for planning the priorities and flow of various activities in rescuing disaster victims.

[0647] "Communication means" refers to functions that include communication technologies for exchanging information and transmitting instructions between various devices.

[0648] "Geometric analysis" is a technology that analyzes images acquired by cameras and sensors to extract useful information.

[0649] "Speech recognition" is a technology that analyzes audio data to identify linguistic information and the source of sound.

[0650] "Environmental conditions" refer to the situation on site, including geographical information of the affected area and the impact of the disaster.

[0651] "Resource status" refers to the current state of available resources such as personnel, supplies, and equipment.

[0652] This invention is an autonomous disaster relief system in which a server, terminals, and users work together. The server acts as the central control unit, aggregating and analyzing data transmitted from aircraft and ground transport vehicles. Specifically, the server uses a high-performance computer to build an information processing system using a generative AI model. This system excels particularly in image recognition and speech recognition technologies, enabling it to pinpoint the location of disaster victims in real time and analyze the situation in disaster areas.

[0653] The aircraft, acting as terminals, are equipped with high-resolution cameras and infrared sensors, enabling rapid observation of wide areas. This allows the aircraft to collect data from the air and immediately transmit the information to the server. Ground carriers, moving across the land, can collect detailed environmental information and are equipped with sensors to avoid obstacles. These terminals operate based on commands from the server, enabling a rapid response to disaster victims.

[0654] Users can monitor the overall progress of the system via an operating terminal and intervene in rescue operations as needed. Users can view terrain information and analysis data in real time and send new commands to the server based on that information. For example, if the disaster area has very complex terrain or if a large number of victims are detected, users can make immediate decisions and readjust rescue policies.

[0655] As a concrete example, consider a scenario where a large-scale earthquake causes part of a city to collapse. Aircraft scan the affected area from above with thermal cameras, and ground transport vehicles navigate the complex terrain to deliver supplies to victims. This entire process is supported by a system that uses server analysis and allows users to monitor the results in real time. An example of a prompt from the generated AI model is, "Using thermal cameras, how can we locate victims trapped under rubble in the disaster area?" This system enables rapid and effective relief for disaster victims.

[0656] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0657] Step 1:

[0658] The server receives observational data transmitted from aircraft and ground carriers. This input data includes high-resolution images, thermal information from infrared sensors, and information about the local environmental conditions. The server performs a basic validation of this data and prepares it for analysis. Specific actions performed at this stage include data format conversion and priority setting.

[0659] Step 2:

[0660] The server analyzes the received data using a generating AI model. Using the high-resolution images and thermal information received in step 1 as input, it executes image recognition and speech recognition algorithms. The server identifies the location of victims through shape recognition and detects unusual sounds and voices from the audio data. This results in the production of location information and status data for the victims as the output of the analysis.

[0661] Step 3:

[0662] The server develops rescue procedures based on the analysis results. It uses the location and status of victims, as well as the environmental conditions at the site, obtained in Step 2, as input. Based on this, the server determines the priority of rescue operations and plans the supply routes for relief materials. The output generates information regarding rescue procedures and transportation routes.

[0663] Step 4:

[0664] The server transmits specific commands to aircraft and ground transport vehicles based on the established rescue procedures and transport routes. This process includes detailed instructions on how to operate rescue equipment and the delivery schedule for supplies. Terminals receive these commands and begin actual observation and rescue operations.

[0665] Step 5:

[0666] Users monitor ongoing rescue operations using control terminals for real-time monitoring. Using data provided in real-time from the server as input, users can make immediate decisions based on the situation and send new commands to the server. This allows for manual intervention as needed.

[0667] (Application Example 1)

[0668] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0669] In recent years, the frequency of natural disasters has increased, highlighting the growing importance of rapid and efficient disaster response. However, conventional disaster relief systems often suffer from delays in information acquisition and transmission, resulting in insufficient support for disaster victims. Furthermore, dynamic evacuation guidance tailored to the specific conditions of the affected area has been difficult. Therefore, there is a need for a system that can collect accurate information in real time during a disaster and quickly provide rescue operations and evacuation guidance to victims.

[0670] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0671] In this invention, the server includes means for observing the environment using aircraft and ground transporters, information processing means including artificial intelligence for analyzing data acquired from the aircraft and ground transporters, means for optimizing resource allocation and formulating rescue procedures based on the location and condition of disaster victims, and means for providing disaster information and evacuation routes using communication means connected to a device that can be carried by citizens. This enables rapid and efficient support for disaster victims and effective evacuation guidance for citizens during disasters.

[0672] An "aircraft" is a means of transportation that can directly observe the Earth's surface from above and rapidly collect data over a wide area.

[0673] A "ground transport device" is a device used to collect detailed environmental data from the ground surface and move through disaster-stricken areas while avoiding obstacles.

[0674] "Information processing means" refers to a system that includes artificial intelligence to analyze data acquired from aircraft and ground transport vehicles and support effective rescue operations.

[0675] "Means for optimizing resource allocation and formulating rescue procedures" refers to methods for planning rapid and appropriate rescue actions by efficiently allocating available resources based on the location and condition of the victims.

[0676] "Communication methods" refer to technologies that provide real-time disaster information and evacuation routes to disaster victims and the general public using mobile devices that they can access.

[0677] This invention implements an autonomous disaster relief system in which servers, terminals, and users each play their respective roles, enabling rapid and accurate assistance to disaster victims during a disaster.

[0678] The server receives environmental data transmitted from aircraft and ground transport vehicles and analyzes it in real time using information processing tools. Specifically, it uses artificial intelligence to perform image and voice recognition to identify the location and condition of disaster victims. The server uses the Google Maps API to update map information of the disaster area and formulate effective rescue procedures. Furthermore, it transmits optimized instructions to aircraft and ground transport vehicles to deploy the necessary actions.

[0679] The aircraft, acting as the terminal, is equipped with high-resolution cameras and infrared sensors to conduct wide-area observations from above. This allows for a detailed understanding of the collapsed areas that are difficult for ground carriers to access. The ground carriers are designed to collect detailed information on the ground and move safely through the disaster area, operating reactively based on instructions from the server.

[0680] Users can receive the latest notifications regarding disaster information and evacuation routes from the server using their smartphones or other mobile devices. Important information is immediately conveyed via push notifications, and detailed information about affected areas is also provided through analysis using generative AI models. Based on this information, users can take swift evacuation action.

[0681] As a concrete example, in the event of a large-scale earthquake in an urban area, the server creates a detailed map of the affected area and analyzes information on the safety of victims. Citizens receive this information via their mobile devices and are guided to safe evacuation routes. An example of a prompt message would be a response to a request such as, "Based on the current evacuation information, please provide the optimal evacuation route."

[0682] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0683] Step 1:

[0684] The server receives real-time sensor data transmitted from aircraft and ground carriers. This data includes image data, infrared analysis data, and audio data. The received data is first temporarily stored in a database and then pre-processed by an artificial intelligence module.

[0685] Step 2:

[0686] The server uses a generative AI model to analyze the received image data and identify the location and condition of the victims. This process applies pattern recognition algorithms to detect human shapes and movements from the images. The input is image data, and the output is the location coordinates of the victims.

[0687] Step 3:

[0688] The server simultaneously analyzes audio data and performs speech recognition processing to identify the voices of disaster victims. Here, the audio waveform is processed as input, and location information of potential victims is output. The results of the audio analysis are combined with the results of the image analysis to improve accuracy.

[0689] Step 4:

[0690] The server uses the Google Maps API to generate a map of the disaster area based on the analysis results from the generated AI model, and overlays the locations of the victims. At this stage, the input is the analyzed location data of the victims, and the output is a detailed map of the disaster area.

[0691] Step 5:

[0692] Users receive maps and the latest evacuation information from a server using their smartphones or smart glasses. The app running on the user's device receives real-time information via push notifications. The input information is command data based on the analysis results of a generated AI model, and the output is evacuation route guidance as visual information.

[0693] Step 6:

[0694] The aircraft and ground transport units, acting as terminals, autonomously move based on the location of disaster victims, receiving commands from the server, and initiate rescue operations as needed, according to the situation in the disaster area. Real-time location information is used as input, and actions such as supplying materials and initiating rescue operations are output.

[0695] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0696] This invention aims to further improve the efficiency of disaster relief activities by combining an emotion engine with an environmental observation system using aircraft and ground transport vehicles. This system utilizes artificial intelligence as a data processing means and can understand the emotional state of users and disaster victims through the emotion engine.

[0697] Server Role

[0698] The server acts as a management center for vast amounts of data. After receiving data from terminals, it first analyzes environmental information. Using image analysis algorithms and speech recognition technology, it identifies the location and condition of disaster victims. This data is used to formulate rescue procedures and optimize resource allocation. The emotion engine also analyzes received audio and image data to recognize the user's emotional state and adjusts rescue procedures based on particularly important emotional changes. Emotion recognition allows the server to make more appropriate decisions for rescue operations.

[0699] Terminal role

[0700] As terminals, aircraft and ground carriers perform specific actions in real time based on instructions from the server. Aircraft are responsible for wide-area observation, continuously collecting data with thermal sensors and high-resolution cameras. Meanwhile, ground carriers acquire detailed environmental information and the location of disaster victims, while simultaneously attempting to communicate with them. For example, by using an emotion engine, it is possible to select an appropriate communication method if a disaster victim is experiencing fear. Furthermore, when delivering relief supplies, consideration is given to the emotions of the disaster victims.

[0701] User roles

[0702] Users can monitor rescue operations via their devices and refer to sentiment analysis data to conduct smoother rescue operations. In particular, sentiment data is fed back into the user interface, serving as a decision-making support tool for users. Specifically, it can present users with countermeasures to alleviate the anxiety and panic of disaster victims, enabling more effective rescue operations. This information is reported to the server and reflected in future rescue operations.

[0703] As a result, this system, which incorporates an emotional engine, provides deeper insights that could not be obtained in conventional rescue operations, enabling swift and accurate rescue efforts.

[0704] The following describes the processing flow.

[0705] Step 1:

[0706] The server continuously receives data from aircraft and ground carriers equipped with various sensors. The data includes a wide range of information, such as images, audio, and temperature information of the external environment.

[0707] Step 2:

[0708] The aircraft, acting as the terminal, will photograph a wide area of ​​the disaster zone from above and provide real-time data. The aircraft will fly autonomously to cover the entire disaster area and transmit the observation data to the server.

[0709] Step 3:

[0710] The ground-based transporter, acting as the terminal, avoids obstacles on the ground and transmits acquired audio and proximity sensor data to the server. This allows for high-precision detection of sounds emitted by people trapped under rubble.

[0711] Step 4:

[0712] The server processes incoming data instantly, using AI image analysis and speech recognition to determine the location and condition of victims. Furthermore, an emotion engine analyzes the emotions from the victims' voices to identify their emotional state.

[0713] Step 5:

[0714] The server develops rescue procedures based on the physical location and psychological state of the victims. If the emotional state is urgent, this is reflected in the rescue plan, and the procedures are adjusted to ensure a rapid response.

[0715] Step 6:

[0716] Based on the formulated plan, the server sends specific instructions to the terminals. Aircraft are instructed on the location and order of dropping relief supplies, and ground transport aircraft are instructed on the delivery of supplies to victims and appropriate communication methods.

[0717] Step 7:

[0718] Users have the ability to review sentiment analysis data sent from the server and adjust rescue priorities in real time. They also communicate with discovered victims sequentially and provide support according to suggestions from the sentiment engine.

[0719] Step 8:

[0720] Users can send feedback to the server as needed regarding ongoing rescue operations, providing insights to improve the quality of future operations. This feedback is recorded and analyzed by the system and used to improve the efficiency of future rescue operations.

[0721] (Example 2)

[0722] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0723] Conventional disaster relief systems were somewhat effective in locating the position and condition of victims. However, they lacked the ability to adaptively change rescue procedures based on changes in the emotional state of victims, and the effective allocation of rescue resources was insufficient. As a result, rapid and accurate rescue of victims was difficult.

[0724] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0725] In this invention, the server includes means for analyzing data acquired from aircraft and ground transport vehicles to understand the emotional state of disaster victims, means for optimizing the allocation of rescue resources based on the emotional state, and means for formulating rescue procedures and transmitting instructions for them. This enables flexible adjustment of rescue procedures according to the emotional state of disaster victims and more efficient allocation of rescue resources.

[0726] An "aircraft" is a means of transportation used for observation and data collection by flying through the air.

[0727] A "ground transport device" is a means of transport used to collect and transmit environmental data and disaster victim information while moving along the ground surface.

[0728] "Means of observing the environment" refers to devices, including sensors and cameras, that are mounted on aircraft or ground transport vehicles and are necessary for detailed analysis of the surrounding environment.

[0729] A "data processing system" is a system that includes machine learning algorithms for analyzing acquired data and making decisions based on that information.

[0730] "Analyzing emotional states" is the process of identifying and understanding the emotional responses of disaster victims from audio and image data.

[0731] "Optimizing resource allocation" is the process of effectively utilizing limited rescue resources and quickly distributing them to the places and situations where they are most needed.

[0732] "Means for formulating rescue procedures" refers to the function of formulating a specific rescue action plan based on the analysis results and building a strategy for putting it into action.

[0733] This invention is designed to enhance the effectiveness of disaster relief operations based on an environmental observation system utilizing aircraft and ground carriers. This system uses aircraft and ground carriers equipped with various sensors as hardware, and machine learning algorithms running on a server.

[0734] Server Role

[0735] The server plays a central role in receiving and analyzing data transmitted from terminals. Specifically, it uses machine learning software to process image and audio data collected from aircraft and ground transport vehicles. Widely used machine learning libraries are utilized for image processing to identify the location and condition of victims. For audio data, sentiment analysis techniques are used to determine the emotional state of victims.

[0736] Terminal role

[0737] The aircraft, acting as the terminal, is equipped with high-resolution cameras and thermal sensors to conduct wide-area environmental observations. The aircraft collects environmental information over a wide area and transmits this data to a server. Ground carriers are responsible for approaching disaster victims and acquiring detailed environmental and audio information. This enables direct communication on-site and allows for adaptive responses based on emotion analysis.

[0738] User roles

[0739] Users refer to data from the server and manage rescue operations based on the displayed situation. The user interface reflects the emotional state and location information of victims in real time, enabling rapid decision-making based on this information. This information is also fed back after the rescue operation is completed and used to improve future responses.

[0740] Specific examples and prompt statements

[0741] One specific example is the use of this system to conduct rapid rescue operations in areas where flooding is anticipated. In this case, an aircraft would take extensive aerial photographs of the affected area, and a server would use these to identify the location and emotional state of the victims. Ground carriers would then send reassuring messages to the identified victims.

[0742] Examples of prompts to input into a generative AI model:

[0743] "Explain how to identify victims in flood-affected areas using sentiment-based data and rapidly allocate appropriate relief resources."

[0744] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0745] Step 1:

[0746] The terminal uses a high-resolution camera and thermal sensors mounted on the aircraft to collect environmental information for a designated area. The input consists of image data and thermal data acquired from the air, and the output is collected environmental data. This data provides fundamental information necessary to understand the overall situation of the region. Specifically, the aircraft follows a designated flight path and periodically records data.

[0747] Step 2:

[0748] The terminal collects detailed information about the vicinity of disaster victims via a ground transport device. Inputs include voice data and more detailed environmental data, and output is the result of on-site communication. The ground transport device understands the emotional state of disaster victims by directly communicating with them via voice. Specific actions include the ground transport device moving closer to the disaster victims and attempting to communicate using microphones and speakers.

[0749] Step 3:

[0750] The server analyzes all data transmitted from aircraft and ground transport vehicles. Inputs include collected image data, thermal data, and audio data, while outputs include the location information and emotional state analysis results of victims. The server uses machine learning algorithms to analyze images, identify the location of victims, and recognize emotional states based on audio data. Specific operations include training a model using the dataset and making predictions through real-time processing.

[0751] Step 4:

[0752] The server develops rescue procedures and creates an optimal resource allocation plan based on the analysis results. Inputs are location information and emotional state, while outputs are specific rescue procedures and resource allocation plans. The developed procedures are flexibly adjusted to reflect the emotional state of the victims, ensuring appropriate actions are taken according to the situation. Specific operations include prompt generation using a generative AI model and scenario simulation.

[0753] Step 5:

[0754] Users monitor rescue procedures generated by the server and make modifications as needed. Inputs are detailed rescue procedures and real-time feedback from the field, while outputs are the final execution plan. Users refer to the plan and make necessary decisions through the interface. Specific actions include sending instructions to the rescue team and checking and updating field data.

[0755] (Application Example 2)

[0756] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0757] Current disaster relief systems can grasp the physical condition and location of victims, but they struggle to monitor their emotional state in real time and provide appropriate support accordingly. Furthermore, in elderly care support, the lack of consideration for emotional states makes it difficult for caregivers to quickly and accurately determine the necessary support.

[0758] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0759] In this invention, the server includes means for observing the environment using an aircraft and a ground transporter, data processing means including artificial intelligence for analyzing data acquired from the aircraft and ground transporter, and processing means for analyzing emotional information and determining appropriate responses and support based on the emotional state of disaster victims. This enables prompt and accurate support and care that takes into account the emotional state of disaster victims and the elderly.

[0760] An "aircraft" is a flying device used for aerial observation and data collection.

[0761] A "ground transport vehicle" is a vehicle used for movement near the Earth's surface, transporting materials, and conducting detailed observations.

[0762] "Means of observing the environment" refers to methods of acquiring surrounding environmental data using aircraft or ground-based transport vehicles.

[0763] "Data processing means" refers to methods that use artificial intelligence to analyze data acquired from aircraft and ground transport vehicles.

[0764] "Artificial intelligence" is a program that analyzes data and automatically makes judgments and decisions using specific algorithms.

[0765] "Emotional information" refers to data on the emotional state of disaster victims in real time, as well as the results of its analysis.

[0766] An "emotion engine" is a technology that analyzes the voice and image data of disaster victims to recognize their emotional state.

[0767] "Means for optimizing resource allocation" refers to methods for determining rescue procedures and support content based on acquired data and emotional information.

[0768] "Rescue procedures" are the procedures established to carry out appropriate rescue operations according to the circumstances of the disaster.

[0769] "Care support" refers to support activities aimed at providing appropriate assistance according to the emotional state of elderly individuals.

[0770] This invention is a system that uses aircraft and ground transport vehicles to observe the environment and employs artificial intelligence as a data processing tool to grasp the emotional state of disaster victims and the elderly in real time, thereby enabling optimal support activities.

[0771] The server receives environmental data transmitted from aircraft and ground transport vehicles and analyzes it using artificial intelligence. This allows it to understand the location and physical condition of disaster victims, and recognize their emotional state using an emotion engine. This process utilizes image analysis libraries and speech recognition technologies (e.g., OpenCV and TensorFlow). Emotional information is used to formulate rescue procedures and contribute to the optimization of support activities. Based on the accumulated data, the server improves the efficiency of future rescue operations.

[0772] The aircraft and ground carriers, acting as terminals, are responsible for wide-area observation and detailed data acquisition. The aircraft primarily collects environmental information from the air, while the ground carriers are configured to capture detailed ground information and emotional changes. The ground carriers communicate with disaster victims and adjust appropriate support activities according to their emotional state. In this process, emotional recognition is used to select responses that will make disaster victims feel safe.

[0773] Users can view aggregated data using a dedicated interface and check the emotional state of disaster victims and the elderly in real time. This supports rapid decision-making on-site and enables the sending of appropriate instructions to the terminal. In this process, the emotional data and recommended actions presented to the user are updated in real time by a generative AI model.

[0774] As a concrete example, when this system is applied to a nursing home, it can monitor the emotional state of elderly residents and automatically provide relaxing music content when it detects that they are feeling lonely. This can reduce the psychological burden on the elderly and create a more comfortable living environment.

[0775] An example of a prompt for a generative AI model is: "Recommend appropriate music content to alleviate feelings of loneliness in the elderly. If the emotional state is determined to be 'lonely,' please provide specific song titles and the reasons for their selection."

[0776] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0777] Step 1:

[0778] The server receives real-time environmental data transmitted from aircraft and ground transport vehicles. This includes high-resolution image and audio data. Based on this input data, artificial intelligence begins initial data processing.

[0779] Step 2:

[0780] The server uses the received image data to identify the locations of disaster victims and elderly individuals using an image analysis library (e.g., OpenCV). Additionally, it utilizes speech recognition technology (e.g., TensorFlow) to extract meaning and emotional information from the audio data. This process generates detailed data regarding the location and condition of disaster victims.

[0781] Step 3:

[0782] The ground transporter, acting as a terminal, collects detailed ground information and utilizes an emotion engine to recognize the detailed emotional state of disaster victims. The analyzed emotional information is sent to a server, and an appropriate method of communication with the victims is selected. In this process, data based on changes in emotion is generated.

[0783] Step 4:

[0784] The server integrates all analyzed data and optimizes rescue procedures and support methods using a generative AI model. It provides users with support suggestions and action guidelines based on their emotional state. This process generates prompts and determines the specific details of the support.

[0785] Step 5:

[0786] Based on data provided by the server, the user selects an appropriate action and sends instructions to the device. For example, if loneliness in an elderly person is detected, a specific instruction is created to provide relaxing music. The device then carries out the support activity according to the user's instructions.

[0787] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0788] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0789] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0790] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0791] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0792] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0793] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0794] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0795] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0796] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0797] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0798] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0799] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0800] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0801] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0802] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0803] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0804] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0805] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0806] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0807] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0808] The following is further disclosed regarding the embodiments described above.

[0809] (Claim 1)

[0810] Means of observing the environment using aircraft and ground-based carriers,

[0811] Data processing means including artificial intelligence for analyzing data acquired from the aforementioned aircraft and ground transport vehicles,

[0812] A means of optimizing resource allocation and formulating rescue procedures based on the location and condition of disaster victims,

[0813] Means for transmitting instructions to the aircraft and ground transport aircraft for supplying relief supplies based on the rescue procedure,

[0814] A system that includes this.

[0815] (Claim 2)

[0816] The system according to claim 1, wherein the data processing means includes an algorithm for identifying disaster victims by image analysis and speech recognition.

[0817] (Claim 3)

[0818] The system according to claim 1, wherein the means for optimizing the allocation of the aforementioned resources is configured to determine the priority of rescue resources based on environmental information at the time of discovery of the disaster victim.

[0819] "Example 1"

[0820] (Claim 1)

[0821] A means of conducting wide-area observations using aircraft and ground-based carriers,

[0822] Information processing means including artificial intelligence for analyzing information collected from the aforementioned aircraft and ground transport vehicles in real time,

[0823] A means of formulating rescue procedures and setting priorities for rescue operations based on the location and condition of the victims, as well as the situation in the affected area.

[0824] A means of communication for transmitting instructions to rescue equipment and supplying materials based on the aforementioned rescue procedure,

[0825] A system that includes this.

[0826] (Claim 2)

[0827] The system according to claim 1, wherein the information processing means includes an algorithm that identifies victims and determines environmental conditions by graphic analysis and speech recognition.

[0828] (Claim 3)

[0829] The system according to claim 1, wherein the means for formulating the rescue procedure is configured to determine the highest priority of rescue resources based on topographical conditions and resource availability at the time of discovery of the victim.

[0830] "Application Example 1"

[0831] (Claim 1)

[0832] Means of observing the environment using aircraft and ground-based carriers,

[0833] Information processing means including artificial intelligence for analyzing data acquired from the aforementioned aircraft and ground transport vehicles,

[0834] A means of optimizing resource allocation and formulating rescue procedures based on the location and condition of disaster victims,

[0835] Means for transmitting instructions to the aircraft and ground transport aircraft for supplying relief supplies based on the rescue procedure,

[0836] A means of providing disaster information and evacuation routes to citizens using communication means connected to portable devices,

[0837] A system that includes this.

[0838] (Claim 2)

[0839] The system according to claim 1, wherein the information processing means includes an algorithm for identifying disaster victims by image analysis and speech recognition.

[0840] (Claim 3)

[0841] The system according to claim 1, wherein the means for optimizing the allocation of the resources is configured to determine the priority of rescue resources based on environmental information at the time of discovery of the victims, and the communication means transmits commands to provide evacuation route information in real time.

[0842] "Example 2 of combining an emotion engine"

[0843] (Claim 1)

[0844] Means of observing the environment using aircraft and ground-based carriers,

[0845] Data processing means including machine learning for analyzing data acquired from the aforementioned aircraft and ground transport vehicles,

[0846] A means for analyzing emotional states based on audio and image data, optimizing resource allocation based on the location and condition of victims, and formulating rescue procedures,

[0847] Means for transmitting instructions to the aircraft and ground transport aircraft for supplying relief supplies based on the rescue procedure,

[0848] A system that includes this.

[0849] (Claim 2)

[0850] The system according to claim 1, wherein the data processing means includes a function to identify victims and understand their emotional state through image analysis and speech recognition.

[0851] (Claim 3)

[0852] The system according to claim 1, wherein the means for optimizing the allocation of the aforementioned resources is configured to determine the priority of rescue resources based on environmental information and emotional state of the victims at the time of discovery.

[0853] "Application example 2 when combining with an emotional engine"

[0854] (Claim 1)

[0855] Means of observing the environment using aircraft and ground-based carriers,

[0856] Data processing means including artificial intelligence for analyzing data acquired from the aforementioned aircraft and ground transport vehicles,

[0857] A means of optimizing resource allocation and formulating rescue procedures based on the location and condition of disaster victims,

[0858] Means for transmitting instructions to the aircraft and ground transport aircraft for supplying relief supplies based on the rescue procedure,

[0859] A processing method for analyzing emotional information and determining appropriate responses and support based on the emotional state of disaster victims,

[0860] A system that includes this.

[0861] (Claim 2)

[0862] The system according to claim 1, wherein the data processing means includes an algorithm that identifies victims by image analysis and speech recognition, and further recognizes their emotional state using an emotion engine.

[0863] (Claim 3)

[0864] The system according to claim 1, wherein the means for optimizing the allocation of the aforementioned resources is configured to determine the priority of rescue resources based on environmental information and emotional information at the time of discovery of the victim. [Explanation of Symbols]

[0865] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means of observing the environment using aircraft and ground-based carriers, Information processing means including artificial intelligence for analyzing data acquired from the aforementioned aircraft and ground transport vehicles, A means of optimizing resource allocation and formulating rescue procedures based on the location and condition of disaster victims, Means for transmitting instructions to the aircraft and ground transport aircraft for supplying relief supplies based on the rescue procedure, A means of providing disaster information and evacuation routes to citizens using communication means connected to portable devices, A system that includes this.

2. The system according to claim 1, wherein the information processing means includes an algorithm for identifying disaster victims by image analysis and speech recognition.

3. The system according to claim 1, wherein the means for optimizing the allocation of the aforementioned resources is configured to determine the priority of rescue resources based on environmental information at the time of discovery of the victims, and the communication means transmits commands for providing evacuation route information in real time.

Images