Information processing system

Abstract

The invention provides an information processing system comprising: an operable flight device; an operable photographing device; the artificial intelligence processing device is used for analyzing and analyzing the generated data in real time; the processing device is used for calculating an optimal rescue route based on the analysis and analysis results; a device for carrying materials on the flying device and transporting the materials; a device used for deploying equipment in a communication interruption area and collecting field state data; the device is used for analyzing the collected data and generating emergency response information; and the device is used for linking the emergency response information with an external rescue mechanism.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech.

[0003] Existing disaster emergency response systems suffer from slow and inaccurate on-site information collection after a disaster, and lack intelligent analysis in the formulation of rescue routes and material transportation paths. This results in low rescue efficiency, an inability to provide effective emergency support to areas with communication disruptions, and an inability to promptly transmit emergency information to relevant rescue agencies, seriously affecting the timeliness and effectiveness of disaster relief work. Summary of the Invention

[0004] This invention provides a system comprising an operable flight device, an operable shooting device, a real-time artificial intelligence data processing device, an optimal rescue route calculation device, a material loading and transportation device, a communication interruption zone equipment deployment and on-site data collection device, and an emergency response information generation and linkage device. By collecting and intelligently analyzing information from the disaster site in real time, it automatically calculates the optimal rescue and transportation routes and can collect and feedback information from communication interruption zones. Furthermore, it can quickly transmit emergency response information to relevant rescue organizations, thereby achieving high efficiency and intelligence in disaster emergency response.

[0005] "Operable flight device" refers to a drone or similar flight device that can fly autonomously or remotely according to instructions and can carry mission equipment or supplies.

[0006] "Operable shooting device" refers to a photographic or video recording device that can be mounted and controlled to acquire on-site images, videos, and audio data.

[0007] "Artificial intelligence processing device" refers to a computing device or system that has the function of real-time analysis and interpretation of collected data using artificial intelligence algorithms.

[0008] The "optimal rescue route calculation device" refers to a processing device that can automatically calculate the most suitable rescue route and plan at a disaster site based on data analysis results.

[0009] "Materials loading and transport equipment" refers to the relevant machinery and control units that can load the required materials onto the aircraft and transport them safely to the designated destination.

[0010] "Equipment deployment and field data collection device for communication interruption areas" refers to the hardware and related systems used to deploy auxiliary equipment in disaster areas where communication is interrupted and to realize field data collection and return.

[0011] "Emergency response information generation and linkage device" refers to a device or system that can generate emergency rescue-related information from collected and analyzed data and achieve information sharing and linkage with external rescue organizations.

[0012] "External assistance organizations" refers to external assistance units such as hospitals, government agencies, fire brigades, and non-governmental organizations that participate in rescue activities during disaster emergency response. Attached Figure Description

[0013] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0014] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0015] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0016] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0017] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0018] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0019] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0020] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0021] Figure 9 This represents an emotion map that maps multiple emotions.

[0022] Figure 10 This represents an emotion map that maps multiple emotions.

[0023] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0024] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0025] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0026] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0027] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0028] First, let me explain the terminology used in the following instructions.

[0029] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0030] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0031] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0032] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0033] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0034] First Implementation Method

[0035] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0036] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0037] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0038] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0039] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0040] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0041] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0042] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0043] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0044] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0045] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0046] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0047] Example 1

[0048] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0049] In the event of a disaster, how to efficiently and accurately acquire multi-source information from the disaster site, how to achieve real-time collection and comprehensive analysis of on-site data in environments with communication disruptions or limited information, and how to formulate optimal rescue routes and material transportation plans based on complex and dynamically changing multi-source data, while simultaneously linking emergency response information to external rescue organizations in a timely manner, have always been key technical challenges that urgently need to be addressed in the field of emergency rescue. Existing technologies often lack integrated data acquisition, intelligent analysis, and automated decision-making processes, resulting in slow initial response speeds, unreasonable resource allocation, and difficulty in guaranteeing the overall efficiency and accuracy of rescue operations.

[0050] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0051] In this invention, the server includes: a device for controlling a movable aerial mobile device; a device for controlling a movable image information acquisition device; an information analysis device for real-time analysis of various types of data using an information processing device; a path selection device for automatically calculating the optimal rescue path based on the analysis results; a transportation control device for automatically formulating and executing material transportation plans according to priority; a field information acquisition device for configuring equipment to acquire field information in special environments such as communication interruptions; a response information generation device for generating emergency response information; an information linkage device for linking emergency response information to external support organizations; an artificial intelligence analysis device for intelligently parsing and prioritizing multi-source disaster data using a generative artificial intelligence model; a prompt statement generation device for automatically generating and inputting disaster response prompt statements into the artificial intelligence model; a data integration device for automatically integrating and sorting heterogeneous data collected by multiple types of sensors; and a processing integration device for automatically and integratedly operating all disaster response processes, including the above steps, in an automated manner. This enables efficient and comprehensive collection and intelligent analysis of disaster site information in complex disaster environments, rapid generation of scientific rescue and material dispatch decisions, and ensures that various emergency response measures can be linked to relevant rescue agencies in a timely and accurate manner, significantly improving the efficiency and reliability of emergency response.

[0052] "Airborne mobile devices" refer to mobile devices that can fly autonomously or remotely to perform tasks such as data collection, material delivery, or equipment delivery, such as drones and remote-controlled aircraft.

[0053] "Image information acquisition device" refers to a device that can acquire multimedia information such as still images, dynamic videos or audio from the environment, such as information acquisition modules such as cameras and microphones.

[0054] "Information processing device" refers to a hardware or software system that can store, calculate, process and manage various types of data collected, such as servers and data processing terminals.

[0055] "Information analysis device" refers to a system used for real-time identification, screening, evaluation and preliminary intelligent processing of multi-source and multi-type data, which can realize functions such as data decoding, preliminary statistics and data aggregation.

[0056] "Route selection device" refers to a system that automatically selects the optimal rescue or transportation route based on analysis results and can apply various algorithm models such as shortest path, risk avoidance, and safety optimization.

[0057] "Transportation control device" refers to a system that coordinates and automatically controls the entire process of loading, departure, route tracking, material delivery, and return of aerial mobile devices.

[0058] "Field information acquisition device" refers to a device that collects field information such as environment, location, and images in areas where conventional communication cannot cover, including but not limited to portable recorders and satellite communication equipment.

[0059] "Response Information Generation Device" refers to a system that can automatically generate emergency response suggestions, action plans, and response information based on data collected and analyzed on-site.

[0060] "Information linkage device" refers to a communication and data interaction system that accurately and promptly pushes generated emergency information to external rescue organizations, users, or management platforms.

[0061] "Generative artificial intelligence models" refer to artificial intelligence systems or models that are based on machine learning technology and can understand, reason about, and generate complex output results from input data, descriptions, instructions, etc.

[0062] "Artificial intelligence analysis device" refers to a system that relies on generative artificial intelligence models to perform advanced analysis, pattern recognition, classification, priority determination, and other complex processing on disaster-related multi-source data.

[0063] "Prompt statement generation device" refers to a system that automatically generates instructional text descriptions based on the current disaster scenario to guide artificial intelligence models in carrying out specific analyses or tasks.

[0064] A "data integration device" refers to a system that unifies, automatically aggregates, cleans, and sorts heterogeneous data from different types of sensors or devices.

[0065] "Integrated processing device" refers to a comprehensive processing system that integrates information acquisition, data analysis, route planning, material transportation, and external information linkage into a unified management platform to achieve automatic flow and integrated collaboration.

[0066] To ensure the successful implementation of this invention, the following will provide a detailed description of the overall structure, hardware and software configuration, data processing content, and specific usage methods of this system.

[0067] The system of this invention mainly includes a server, multiple terminals (such as aerial mobile devices, image acquisition devices, etc.), and end users. The server, as the central control unit of the system, coordinates the actions of all related devices, automatically integrates and intelligently analyzes data, and ultimately generates rescue plans and emergency information, achieving integrated and efficient disaster emergency response.

[0068] For the hardware, the aerial mobile device can be a commercial-grade drone; the image acquisition device can be a high-resolution camera and microphone mounted on the drone; in special scenarios, portable recorders and satellite communication equipment can also be configured. The server can be a general-purpose computer server equipped with a high-performance CPU and GPU to support real-time analysis of large amounts of data. The terminal device maintains high-speed and stable data exchange with the server via a wireless communication module (such as a 4G / 5G module or a satellite communication terminal).

[0069] On the software side, the server is equipped with image recognition software (such as OpenCV and YOLO), a geographic information analysis platform (such as a general map API service), and an AI platform loaded with generative artificial intelligence models (such as commercially available GPT-4 or similar large language models). The server also integrates a data integration program that can automatically aggregate and clean heterogeneous data uploaded from multiple terminals, including images, audio, and geographic location information. The AI ​​analysis module is responsible for classifying, prioritizing, and generating action recommendations for all data.

[0070] During system operation, users primarily submit requests through a dedicated client or mobile application, and can view the rescue status in real time or add additional data collection requests. The server automatically generates prompts for the generative artificial intelligence model based on the scenario, ensuring that the AI ​​can provide the most scientific analysis and emergency recommendations tailored to the specific disaster type and on-site conditions. For example:

[0071] "A magnitude 7 earthquake occurred in city A, and some areas have been isolated from the outside world. Based on the following real-time drone video and audio data, please identify which areas require priority rescue, calculate the optimal route for the rescue team, and the types and quantities of supplies needed. All actions must be completed within 72 hours."

[0072] "Current drone footage shows a large area of ​​urban X is fractured. Please use map data to calculate the fastest route for rescue vehicles to reach the stranded area, and list potential hazards along the way."

[0073] In one specific embodiment, after a user receives a disaster response request through a client at the emergency command center, the server immediately analyzes the disaster area via a map API, automatically directs multiple drones to designated data collection points, and transmits multimodal data collected by cameras and microphones to the server in real time. An artificial intelligence model on the server interprets the acquired data, automatically generates a disaster map and a list of priority rescue targets, and then uses a path planning algorithm (such as A* or Dijkstra's algorithm) to automatically generate rescue routes. The drones also automatically load supplies (such as medicine and food) according to instructions from the server and proceed to the appropriate locations to perform supply delivery tasks. When encountering areas with network signal interruptions, the server controls the drones to carry satellite phones and mobile data collection equipment, which are then handed over to on-site rescue personnel to supplement information collection. The drones then return to base and upload the data for analysis.

[0074] Through the above configuration and process, the system can realize automatic multi-source data collection after a disaster, multi-modal AI intelligent deep analysis, automated decision-making on rescue routes and material allocation, and precise linkage of emergency information among rescue agencies at all levels, greatly improving the speed and scientific nature of disaster emergency response.

[0075] use Figure 11 The processing procedure is explained.

[0076] Step 1:

[0077] The server receives disaster notifications from users, government agencies, or automated sensors. Input consists of structured data such as disaster type, location, and time. The server parses the notifications, stores the data in a database, and automatically creates emergency response tasks. Output includes registered disaster event entries and preliminary task instructions.

[0078] Step 2:

[0079] Based on disaster information and geographic data, the server calls the map API to determine the key areas and data collection points that the drones (terminals) need to go to. Inputs include the disaster location, terrain data, and surrounding environmental information. The server compares existing resources, allocates several drones, and generates a task plan that includes target coordinates, data collection type (images, videos, audio), and priority. Output is a list of drone dispatch tasks and instructions.

[0080] Step 3:

[0081] The terminal (drone) receives instructions from the server and automatically takes off to the designated area. Inputs include flight plan, target coordinates, and mission type. The terminal uses GPS and navigation systems for path planning and calibration. Upon reaching the target, it activates its camera and microphone to capture high-definition photos, videos, and on-site audio, achieving real-time data acquisition. Output is the acquired multimodal on-site data stream.

[0082] Step 4:

[0083] The terminal transmits the collected data back to the server in real time via 4G / 5G networks or satellite communication. The input is multimodal raw data (images, videos, audio, and location information). After receiving the data, the server initiates data decoding and preliminary screening, performing frame-level segmentation, noise filtering, and correlation labeling. The output is a structured, processed dataset.

[0084] Step 5:

[0085] The server utilizes an artificial intelligence analysis module (generative AI model) to analyze structured data. Input consists of filtered image, audio, and environmental feature data. The server first generates prompts appropriate for on-site needs, then calls the generative AI model to classify and analyze the data, identifying information such as collapses, trapped personnel, and dangerous areas. Output includes a disaster area status report, a list of priority rescue targets, and detailed annotations.

[0086] Step 6:

[0087] Based on AI analysis results and map data, the server automatically generates optimal rescue routes and material transportation plans using path planning methods such as A* and Dijkstra's algorithm. Inputs include priority rescue target points, traffic and obstacle information, and resource distribution. The server comprehensively calculates the distance, hazard level, and passability of each route, outputting an optimal action plan that includes route details, estimated travel time, and hazard avoidance guidelines.

[0088] Step 7:

[0089] The server analyzes material demand and plans the drones' material loading accordingly. Inputs include a list of material requirements for each target point, warehouse data, and the drone's loading capacity. The server then schedules drones to automatically load supplies such as medicine and food and transport them to specific areas. Outputs the drone loading, flight, and delivery tasks.

[0090] Step 8:

[0091] When performing material transportation tasks, the terminal provides real-time feedback on task progress. Inputs include flight logs, location information, and delivery confirmation (image or video receipt). Upon arrival at the site, the terminal automatically deploys the materials, collects delivery photos, and confirms delivery completion. Outputs include delivery proof received by the server and supplementary on-site data.

[0092] Step 9:

[0093] The server dispatches terminals equipped with satellite phones and on-site data collection equipment to relevant locations in areas experiencing communication outages. Inputs include the distribution of communication blind spots and the data collection requirements. The terminals collaborate with on-site users to acquire supplementary data, then return and upload it. The server receives and analyzes the supplementary data to complete the disaster information. Output is a complete comprehensive disaster database.

[0094] Step 10:

[0095] Based on all analysis results, the server automatically generates a comprehensive rescue report and emergency response recommendations. Inputs include all collected and analyzed data, including multiple rounds of AI interpretation results and user feedback. The server pushes the report, rescue plan, and material dispatch scheme to relevant rescue organizations and users via email, API, or a dedicated platform. Outputs include comprehensive emergency response information and operational recommendations distributed to all parties.

[0096] Application Example 1

[0097] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0098] With the increasing frequency of natural disasters and emergencies, how to quickly and accurately collect on-site information, analyze abnormal situations, rationally allocate rescue resources, effectively deliver emergency supplies, and collect information in areas with communication disruptions has become a prominent challenge in the field of emergency management. Traditional systems cannot achieve intelligent real-time analysis of on-site data and efficient multi-terminal linkage, making it difficult to respond promptly to changes in complex disaster sites, dynamically adjust material transportation and rescue plans, and comprehensively consider the emotional information of disaster victims, thus affecting the efficiency of emergency response and the effectiveness of rescue efforts.

[0099] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0100] In this invention, the server includes: a device for controlling movable bodies to collect on-site information; a device for collecting and uploading image and audio data in real time; an information processing device for inputting the collected data into a generative artificial intelligence model, analyzing and reasoning through prompts, and outputting anomaly judgment results; a path calculation device for dynamically calculating the optimal personnel allocation and material transportation routes based on reasoning and spatial information; a material transportation device for controlling movable bodies to automatically load and deliver supplies; a non-real-time information collection and feedback device for communication-disrupted areas; an automatic generation and distribution device for abnormal and emergency information; a response control and dynamic plan adjustment device; a device for analyzing the emotions of disaster victims using an emotion recognition model; and a device for generating and managing input prompts for the generative artificial intelligence model. This enables intelligent collection, real-time analysis, automatic decision-making, dynamic optimization of rescue and material delivery routes, information retrieval covering communication-disrupted areas, and priority adjustment based on personnel emotional states, thereby significantly improving the speed and accuracy of emergency response and optimizing the allocation of rescue resources and on-site command efficiency.

[0101] "Movable mobile bodies" refer to automated devices such as robots, drones, and unmanned vehicles that can move autonomously in a designated area or environment to complete tasks such as information collection and material transportation.

[0102] "Image information" refers to static images or dynamic video data that reflect the visual state of the scene, acquired through acquisition devices.

[0103] "Audio information" refers to sound signal data such as ambient sound and human voice acquired through audio acquisition devices.

[0104] "Master control device" refers to a server or central control system used to centrally manage, control, and process all collected information and coordinate the execution of tasks in each stage.

[0105] "Generative artificial intelligence models" refer to artificial intelligence systems that are based on deep learning and big data training, capable of accepting input data and prompts, automatically analyzing and reasoning, and generating output results.

[0106] "Prompts" are instructional texts that point to the input of generative artificial intelligence models, including analysis instructions or query requests, and are used to guide the artificial intelligence model to perform specified tasks.

[0107] "Information processing device" refers to a collection of hardware and software used to analyze and process collected data and output results of abnormal status or urgency level judgment.

[0108] "Route calculation device" refers to a hardware and software device that dynamically calculates the optimal rescue and material transportation route and outputs the plan based on analysis results and on-site spatial data.

[0109] "Goods transport device" refers to an automated execution unit used to automatically load materials onto a movable body and transport them to a target location according to a specified path.

[0110] "Non-real-time information acquisition device" refers to a device used to temporarily collect and store information in areas where communication is blocked, and to transmit data to the main control device when conditions permit.

[0111] "Emergency information generation and distribution device" refers to a processing and communication unit that can automatically synthesize abnormal and emergency event information and push it to various terminals in real time.

[0112] "Response control device" refers to a device that dynamically adjusts the execution plan and controls the system's operating status based on user feedback or changes in the field.

[0113] "Emotional analysis device" refers to a hardware and software unit that uses emotion recognition algorithms to judge the emotions of disaster victims based on collected on-site images or audio information.

[0114] The present invention relates to an intelligent emergency response system that utilizes a generative artificial intelligence model and multi-terminal collaborative action. The system mainly consists of a server, movable mobile bodies (such as drones and unmanned vehicles), various sensors, an emergency material loading unit, an emotion analysis module, and user terminals.

[0115] In this invention, the server serves as the core of the system, responsible for overall task scheduling, data analysis, path optimization, information distribution, and user interaction. The server is equipped with high-performance computing hardware (e.g., servers with NVIDIA GPUs and Intel Xeon processors) and deploys a comprehensive platform including deep learning-based generative artificial intelligence models and emotion recognition models, path planning algorithms (such as A* algorithm and Google OR-Tools), large-scale databases, and geographic information systems (such as QGIS).

[0116] In practical implementation, the terminal (such as a drone) is equipped with a high-definition camera (such as an industrial-grade visible light and infrared dual-mode camera), an environmental audio acquisition device (such as a high-sensitivity microphone), a positioning module (such as GPS / BeiDou), and a data transmission module capable of 4G / 5G / satellite communication. Users can report emergency events, browse real-time on-site information, and request and respond to rescue requests through a smartphone application or web interface.

[0117] During the operation, users first report the disaster location and basic information to the server via their smart terminals. The server retrieves emergency databases, geographic information, and weather information to analyze the current situation in the affected area. The server then issues task instructions to the terminal, and mobile devices such as drones automatically take off according to the assigned tasks, collecting video, audio, and spatial data on-site. The above data is uploaded to the cloud server in real time in a preset format.

[0118] The server utilizes generative artificial intelligence models (such as Transformer-based multimodal models or YOLOv8 models) and Whisper audio analysis models to perform deep analysis on the acquired video and audio data. During this process, the server guides the AI ​​model to identify features such as abnormal events, structural damage, and crowd emotions as needed by crafting specific prompts. For example, the server might input the following prompt:

[0119] Please analyze the following camera footage and predict the abnormal activity score:

[0120] Frame: [Image data]

[0121] Please identify the emotional state of the characters in the following audio data:

[0122] Audio: [Audio Data]

[0123] Through the aforementioned generative artificial intelligence model, the server can quickly assess abnormal situations and the emotional state of disaster victims, providing timely information support for decision-makers. Based on the analysis results and geographic information, the server invokes a path planning algorithm to generate the optimal rescue route and material delivery plan, then automatically sends task instructions to mobile devices such as drones. Terminals, following server instructions, automatically load medicines, first-aid kits, food, or communication equipment, completing the material delivery task via the optimal route. If the target area is a communication-disrupted region, drones carrying satellite phones or data recording equipment are used for on-site delivery. Disaster site personnel then use these devices to collect supplementary data, which is transmitted back to the server when conditions permit, further refining the rescue strategy.

[0124] In addition, the server integrates an emotion recognition model for in-depth analysis of the emotions of disaster victims in images and audio. The server dynamically adjusts rescue priorities based on the fear, anxiety, and other emotional states of the victims, enabling rapid response to vulnerable areas. Users can view real-time audio and video streams, anomaly analysis, and rescue routes through the app, and provide feedback on changes on-site. The server further adjusts its dispatch strategy based on user feedback, achieving a continuously self-optimizing dynamic emergency response.

[0125] This system can integrate high-definition video, audio, geolocation, user feedback, and artificial intelligence algorithms to achieve comprehensive dynamic perception, intelligent decision-making, and precise rescue at disaster sites, significantly improving emergency response efficiency and capabilities.

[0126] use Figure 12 The processing procedure is explained.

[0127] Step 1:

[0128] Users send disaster notifications to the server via smartphone apps or web browsers, including the type of disaster and the specific coordinates of the affected location.

[0129] Input: Basic information about the type and location of the disaster entered by the user.

[0130] After receiving the information, the server retrieves relevant background data from geographic information databases, meteorological databases, and rescue resource databases to generate a regional risk analysis report.

[0131] Output: Dispatch data containing the current status and initial information of the disaster area.

[0132] Specific actions: Users fill in and submit "earthquake, address, coordinates" on the App; the server automatically extracts geographical, weather and rescue resource information.

[0133] Step 2:

[0134] Based on the received disaster notifications and analysis results, the server uses path planning algorithms to automatically calculate the optimal flight or movement route for mobile objects (such as drones) and generate inspection or deployment task instructions.

[0135] Input: Dispatch data (disaster type, geographical location, resource information, etc.).

[0136] The server algorithm processes geographic data and real-time road obstacle data to generate the optimal route and inspection task list.

[0137] Output: Mobile task instructions and route information.

[0138] Specific actions: The server uses the A* algorithm / Google OR-Tools to calculate the optimal route and sends flight and data collection tasks to the terminal.

[0139] Step 3:

[0140] After receiving instructions from the server, the terminal (such as a drone) automatically starts, carrying sensors such as a high-definition camera and microphone, and travels to the disaster site along a designated route to collect real-time video, image and audio data.

[0141] Input: Flight route and mission instructions issued by the server.

[0142] The terminal collects data through various sensors and performs preliminary formatting processing.

[0143] Output: Acquired audio and video data and on-site environmental parameters.

[0144] Specific actions: The drone takes off and flies to the designated coordinates, continuously recording live footage and audio during flight, and marking the collection time and GPS location.

[0145] Step 4:

[0146] The terminal uploads the collected image and audio data to the server in real time via wireless communication (4G / 5G / satellite).

[0147] Input: Audio and video data collected by the terminal.

[0148] Data is encoded and transmitted at the terminal to ensure data integrity and real-time performance.

[0149] Output: The data stream is uploaded to the specified interface on the server backend.

[0150] Specific actions: The data transmission process is automatically segmented and packaged, and pushed to the server in real time via a secure connection.

[0151] Step 5:

[0152] After receiving the audio and video data uploaded by the client, the server calls the generative artificial intelligence model and combines it with specific prompts to perform intelligent analysis on the video and audio content, including anomaly detection, event classification, and emotion recognition.

[0153] Input: Audio and video data and corresponding prompts.

[0154] The server uses OpenCV to process video frames, YOLOv8 to identify targets such as collapses and fires, Whisper to transcribe audio, and a generative AI model to perform deep reasoning with prompts such as "Please analyze the following camera footage and predict the score of abnormal activity: Frame: [Image Data]" as input.

[0155] Outputs include: abnormal event reports, risk classifications, and sentiment analysis results.

[0156] Specific actions: The server inputs each captured frame and audio into the AI ​​model to obtain the judgment result and confidence score.

[0157] Step 6:

[0158] Based on AI analysis results, geographic information, road conditions, and other factors, the server calculates the optimal routes for material delivery and rescue team dispatch, and generates action recommendations.

[0159] Input: AI analysis results, geographic and road dynamic data.

[0160] The server uses an optimization algorithm to calculate the optimal resource matching and material delivery path.

[0161] Outputs: Action plan, rescue team route, and material allocation plan.

[0162] Specific actions: The server generates messages such as "It is recommended to take a detour via the North Road, prioritize support for Area A, and carry medical kits to point X" and pushes them to relevant terminals or apps.

[0163] Step 7:

[0164] The terminal receives new instructions from the server for material delivery or inspection, automatically loads, transports, and delivers emergency supplies according to the planned route, and reports the execution progress back to the server.

[0165] Input: Action plan and material loading instructions.

[0166] The terminal loads supplies according to instructions, moves along the optimal route, and provides real-time feedback on the delivery status.

[0167] Output: Confirmation of resource deployment, current location, and remaining task status.

[0168] Specific actions: The drone retrieves the required supplies from the warehouse, flies to the site along the recommended route and drops them, and automatically reports the delivery status upon return.

[0169] Step 8:

[0170] The server integrates user feedback, new changes on-site, and subsequent information to dynamically revise emergency measures and continuously optimize response plans through a data closed loop.

[0171] Input: Real-time user feedback and new on-site data.

[0172] The server performs information integration and self-optimization of decision-making.

[0173] Output: New round of task instructions and optimized contingency plan.

[0174] Specific actions: The server automatically recalculates the allocation and scheduling of rescue resources and notifies users and all relevant terminals of the new decisions.

[0175] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0176] Example 2

[0177] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0178] During disasters, delayed or inaccurate on-site information collection makes it difficult to effectively adjust rescue routes and material distribution plans according to real-time conditions. Furthermore, the emotional state of affected individuals is often overlooked, leading to irrational allocation of rescue resources and psychological support, thus impacting rescue efficiency and the physical and mental recovery of victims. In addition, traditional systems have limited data collection and on-site analysis capabilities in areas with communication disruptions, lacking automated, intelligent, and comprehensive solutions for handling complex disasters.

[0179] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0180] In this invention, the server includes: a device for controlling a movable aerial vehicle; a device for operating a movable imaging device; a device for acquiring and structuring on-site information; a device for collecting and integrating various types of information from multiple external information sources; a device for performing on-site analysis and emotional state analysis based on multi-source data and generative artificial intelligence models; a device for dynamically optimizing movement paths and material delivery paths based on analysis results; a device for deploying equipment and collecting data in areas with communication disruptions; a device for automatically generating emergency judgment information and activity plans and connecting them with external organizations; and a device for parsing user prompts and autonomously controlling the system. This enables the automatic acquisition of multi-source real-time data at disaster sites, AI-driven data analysis and prioritization, and intelligent generation and dynamic adjustment of rescue and material delivery plans, while fully considering the emotional state of disaster victims and the special needs of special areas, thereby improving overall rescue efficiency and the health recovery of disaster victims.

[0181] "Airborne mobile body" refers to a flying device that has autonomous flight capability and can perform data collection or material transportation tasks in the air according to a predetermined route.

[0182] "Mobile imaging device" refers to a camera device that can capture static or dynamic images of a scene as it moves with an object in the air, including but not limited to cameras, video cameras, and other similar devices.

[0183] "Field information" refers to multi-dimensional data obtained at disaster or emergency sites through sensors, images, audio, and other means, including information on the disaster situation, personnel distribution, and environmental conditions.

[0184] "External information sources" refers to various databases or information platforms other than this system that provide information related to disaster areas, such as maps, weather, and shelters.

[0185] "Generative artificial intelligence models" refer to deep learning algorithm systems that can automatically generate text, labels, or decision results based on input data, such as natural language generation models or multimodal analysis models.

[0186] "Multi-source data" refers to a composite collection of information from different types of devices or channels, including images, audio, sensors, third-party databases, etc.

[0187] "Emotional state analysis" refers to the method of identifying the current psychological or emotional state (such as fear, anxiety, or peace of mind) of disaster victims by analyzing features in voice, text, or video data.

[0188] "Dynamic optimization" refers to the process by which the system can automatically adjust rescue routes, material allocation plans, or resource scheduling strategies based on real-time changes received.

[0189] "Communication interruption area" refers to an area where traditional communication methods cannot cover the area due to infrastructure damage or special conditions, requiring special equipment to collect and transmit data.

[0190] "Emergency assessment information" refers to comprehensive reference information about hazard levels, priority handling matters, and response measures that is automatically generated by the system after analyzing multi-source data from the site and external sources.

[0191] "Activity plan" refers to a specific on-site rescue, material transportation and resource allocation plan automatically generated by the system based on data analysis results and general rescue procedures.

[0192] "User prompts" refer to text information that users input in natural language to describe their needs or instructions, which the system then parses and automatically generates corresponding response tasks.

[0193] This invention relates to an intelligent data acquisition and rescue dispatch system for emergency response at disaster sites, specifically including a server, terminals (such as drones and other aerial mobile devices), and users accessing the terminals. The system integrates various sensors, communication hardware, and generative artificial intelligence models, enabling it to automatically complete disaster site information collection, data analysis, route planning, material distribution, and emotional state recognition. Furthermore, it can achieve intelligent responses to specific emergency scenarios based on natural and concise prompts from users.

[0194] The server is powered by high-performance computing equipment and runs on a Python environment. It integrates mainstream deep learning platforms such as TensorFlow and PyTorch, and is equipped with a multi-threaded task scheduling system. The server also exchanges data in real time with external map platforms, weather platforms, and evacuation information databases via APIs. The server can be deployed on cloud platforms or local data centers and is pre-configured with hardware and software environments for image recognition (such as YOLOv5 and ResNet), audio recognition (such as automatic speech recognition APIs), and natural language processing (such as generative AI models like BERT and GPT).

[0195] Terminals typically utilize drones with autonomous navigation capabilities as their primary execution units. These drones are equipped with high-resolution cameras, microphones, GPS modules, high-speed wireless communication modules, and delivery systems. The terminal can communicate with the server in real-time via various protocols such as 5G, 4G, and satellite communication, ensuring maximum data transmission across diverse infrastructure environments.

[0196] Users can interact with the server through various methods, including web and mobile devices, by inputting prompts describing their on-site needs or rescue intentions. The server uses a generative artificial intelligence model to automatically parse the prompts and, combined with real-time on-site data, generates specific and executable rescue or delivery instructions. Through this fully automated process, users can achieve efficient command without complex operations.

[0197] During system implementation, the server first dynamically acquires target area information from external map databases, meteorological databases, and shelter information databases via API, forming comprehensive on-site environmental data. Then, the server determines the drone's flight path, shooting locations, and data collection tasks. For example, based on real-time weather data and on-site road conditions, the server allocates the optimal flight path for the drone, designating collapsed buildings and severely flooded areas as key image acquisition targets.

[0198] After receiving the task from the server, the terminal (drone) activates its camera and microphone to collect high-definition images, videos, and on-site audio data at designated times and locations, and uploads them to the server in real time using its installed high-speed wireless communication module. In areas where communication is interrupted, the terminal can also deploy special equipment such as satellite phones and digital cameras, allowing on-site rescue personnel to transmit important information back.

[0199] The server uses deep learning models trained on platforms such as TensorFlow (e.g., YOLOv5, ResNet) to analyze the received image data, identifying information such as collapsed buildings, isolated individuals, and flood locations in the disaster area. Audio data is converted to text via an automatic speech recognition API (e.g., a general ASR system), and then used sentiment analysis AI models (e.g., BERT, GPT) to identify the emotional states of disaster victims, such as fear, anxiety, or relief.

[0200] Based on these analysis results, the server uses path optimization algorithms such as A* and Dijkstra's algorithm to determine the optimal rescue and material delivery routes and dynamically adjust resource allocation and priorities. For example, in areas where disaster victims exhibit high levels of panic, the system prioritizes the delivery of psychological support supplies and professional personnel. All rescue and transport plans automatically generate reports and are pushed to external rescue agencies via API, SMS, or email.

[0201] The following are examples of prompts that users can input during specific operations:

[0202] "A magnitude 7 earthquake has occurred, and parts of the city are completely isolated. Please dispatch drones to quickly collect on-site information from the disaster area, calculate rescue routes, and formulate a material transportation plan. At the same time, analyze the audio and video data of the affected people and dynamically adjust the rescue priority."

[0203] In summary, this invention provides a system that combines hardware and software, is highly automated, and can perceive, analyze intelligently, and respond flexibly to various emergency scenarios in real time, providing strong technical support for improving disaster relief efficiency and the overall health recovery of disaster victims.

[0204] use Figure 13 The processing procedure is explained.

[0205] Step 1:

[0206] The server receives disaster notifications. Input consists of disaster event notification data from external monitoring platforms or government agencies. The server triggers an emergency response process based on the notification type. The server automatically retrieves map data, meteorological information, and shelter information for the affected area via API, outputting an integrated preliminary on-site environmental data package. During this process, the server performs data retrieval and data format standardization, and stores the data in a dedicated database.

[0207] Step 2:

[0208] The server integrates the acquired data to generate the drone's flight path, camera coordinates, and data acquisition command. Inputs include environmental data packets and information such as disaster type and affected area. The server utilizes Geographic Information System (GIS) analysis functions (such as spatial data processing algorithms) to combine these into the optimal drone mission plan, outputting a drone mission command package. This step includes path calculation, camera point selection, and automatic determination of the data type to be collected.

[0209] Step 3:

[0210] After receiving the mission command, the terminal (drone) initiates the automatic flight and data collection process. The input is the mission command packet sent by the server. Based on the flight route and target point, the terminal automatically navigates to the designated location, controls the camera to capture images and videos, and collects ambient sound data via the microphone. The output is the collected multi-source raw data (images, audio, flight logs, etc.). Specific actions include GPS calibration, camera autofocus, and microphone auto gain control.

[0211] Step 4:

[0212] The terminal uploads raw data collected in real time to the server via 5G / 4G or satellite communication. The input consists of all raw data collected by the terminal. After initial data compression and encryption, the terminal sends the data to the server using an efficient and secure protocol. The output is the structured data packet received by the server. The terminal also performs a network status self-check; if the signal is poor, it can automatically switch to a delayed upload mode.

[0213] Step 5:

[0214] The server performs deep learning processing on received image, audio, and sensor data. The input is structured multi-source data packets uploaded by the drone. The server calls generative AI models trained on platforms such as TensorFlow and PyTorch to perform object detection and scene segmentation on images, transcribe audio data using an automatic speech recognition API, and analyze human emotions using a sentiment analysis model. After feature extraction and pattern recognition, the output includes disaster scene assessment results, sentiment status labels, and the distribution of high-risk points requiring priority handling. Specific actions include automatic reasoning, batch feature calculation, and sentiment natural language analysis.

[0215] Step 6:

[0216] Based on the analysis results, the server uses either the A* algorithm or Dijkstra's algorithm to plan the optimal rescue and supply transportation routes. The inputs are disaster assessment results, geographic and road network information. The server integrates information on damaged buildings, isolated individuals, dangerous areas, and sentiment, outputting rescue route plans and supply deployment arrangements. The server simultaneously considers the level of urgency and adjusts rescue priorities accordingly. Specific actions include dynamically updating road accessibility assessments and simulating supply delivery.

[0217] Step 7:

[0218] The server notifies relevant rescue organizations and supply providers of the generated rescue information and logistics plans via API, SMS, or email. Inputs include rescue routes and supply arrangements, emergency priorities, and sentiment analysis tags. The server aggregates these into an emergency response report and outputs it to external communication interfaces, ensuring information coverage of all key response units. This step includes automated document generation and multi-channel distribution.

[0219] Step 8:

[0220] Users can input prompts via the terminal, such as "Please prioritize rescuing those exhibiting extreme panic." The input is the user's natural language command text. The server uses a generative artificial intelligence model to perform semantic analysis on the statement, translating user needs into system operation procedures and dynamically updating drone missions and priority assignments. The output is the system's new adaptive action instructions. Operational actions include natural language parsing, process adjustment, and command delivery.

[0221] Step 9:

[0222] In areas with communication disruptions, the server allocates drones equipped with specialized devices (such as satellite phones and portable cameras) to deliver the equipment to the site as needed and arranges for data to be uploaded back. The input consists of a list of communication disruption areas and new task requirements. The endpoints deploy equipment, confirm the site, and retrieve data according to server instructions. The output is supplementary data uploaded back to the site. Specific actions include area positioning, physical equipment delivery, and return route planning.

[0223] Application Example 2

[0224] The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0225] During disasters, unclear on-site conditions, damaged communication networks, and the inability to effectively collect and analyze the emotional state of affected users lead to inefficient allocation of rescue resources and transportation of supplies, hindering timely and accurate support to disaster-stricken areas. Current technologies lack a system that can comprehensively utilize unmanned aerial vehicles, artificial intelligence algorithms, and emotional data to achieve efficient disaster information collection, real-time analysis, dynamic priority adjustment, and optimal rescue route planning.

[0226] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0227] In this invention, the server includes a device for remotely or autonomously controlling an operable flight device; a device for acquiring on-site images and audio data through an imaging device; an artificial intelligence processing device for real-time parsing and analysis of various types of information; a computing device for integrating various geographical and environmental data and calculating the optimal rescue route based on algorithms; a device for transporting goods; a device for collecting on-site information and reporting data from areas with communication interruptions; a device for using artificial intelligence to determine the urgency of information and automatically generate emergency response information; a device for contacting external rescue organizations; a device for collecting and parsing user emotional data; a control device for dynamically adjusting rescue priorities; a management device for comprehensively managing various disaster response data; and a generation support device for inputting generative artificial intelligence model instructions to obtain emergency support information. This enables efficient acquisition of on-site information, intelligent adjustment of rescue priorities, optimization of support responses based on user emotional states, and automatic planning of the optimal rescue route based on real-time multi-source data, significantly improving the accuracy and efficiency of disaster emergency response.

[0228] "Operable flight device" refers to machine equipment that can be remotely controlled or autonomously perform flight missions, including but not limited to drones and unmanned aerial vehicles.

[0229] "Imaging device" refers to a device used to collect static or dynamic visual information on site, including hardware such as cameras and video cameras that can acquire images or videos.

[0230] "Information processing device" refers to computing or communication equipment capable of collecting, transmitting, analyzing and storing multi-source data, including but not limited to servers and cloud platforms.

[0231] "Artificial intelligence processing device" refers to a hardware and software combination system that uses artificial intelligence algorithms or models to analyze and interpret various types of information, such as collected image data and audio data, in real time.

[0232] "Computing device" refers to the hardware and software equipment used to perform algorithm-based computational processing on various integrated data information and output decision information such as optimal path and scheduling scheme.

[0233] "Transportation device" refers to equipment capable of carrying and transporting goods to a designated area, including but not limited to the goods-carrying mechanism on unmanned aerial vehicles.

[0234] "Field information acquisition device" refers to various sensing and communication terminals that can collect, aggregate, and transmit field data information to back-end information processing devices in environments where communication is interrupted or in adverse conditions.

[0235] "Generation device" refers to a device that can determine the urgency and importance of collected information based on artificial intelligence and automatically generate emergency response information.

[0236] "Communication device" refers to a device used to communicate or notify external rescue organizations of generated emergency response information through communication networks or other means.

[0237] "Electronic information terminal" refers to electronic interactive devices that can collect, input, and transmit user location information and emotional data, including but not limited to smartphones and tablets.

[0238] "Sentiment analysis algorithms" refer to a system of algorithms that use computer models to identify and analyze the emotional state in user text, audio, or video data.

[0239] "Control device" refers to equipment or system that dynamically adjusts and controls the priority of rescue activities based on user emotion analysis results and on-site conditions.

[0240] "Integrated information management device" refers to a hardware and software system that can integrate, store, analyze, and visualize multi-source data to optimize disaster emergency response processes.

[0241] "Generative support device" refers to a device or system used to input instructions into a generative artificial intelligence model and obtain disaster response support information.

[0242] This invention provides a disaster emergency response system based on an operable flight device, an artificial intelligence processing device, a computing device, a field information acquisition device, a user terminal, and a generation support device. This system can automatically, in real-time, and efficiently collect, analyze, and comprehensively utilize multi-source data during a disaster to achieve intelligent rescue dispatch and priority delivery of supplies.

[0243] In this invention, the server serves as an information processing and command center, equipped with a high-performance processor and large-capacity storage devices, and integrates mainstream software tools such as Python, TensorFlow, PyTorch, NetworkX, and QGIS to achieve multi-dimensional real-time processing of images, audio, and geographic information. The server can interact with terminals via 4G / 5G, satellite, and other communication methods, and can interconnect with the information platform of rescue organizations.

[0244] The terminal is primarily an unmanned aerial vehicle (UAV) device, which can utilize common drone platforms and is equipped with a high-resolution camera (such as the IMX series image sensor), a high-sensitivity microphone, a material delivery module, and a multi-mode wireless communication module. Based on task instructions issued by the server, the terminal can automatically navigate to the designated disaster area and sequentially collect on-site environmental images, videos, and audio, uploading the collected data (JPEG images, H.264 video, WAV audio, etc.) to the server in real time. For areas with communication disruptions, the terminal can also autonomously deploy a portable satellite connection module to ensure effective data transmission.

[0245] Users can upload their location information (automatically obtained by the built-in GPS module) and current emotional state via a companion smartphone app (Android or iOS). Emotional data can be entered via text or voice recording. The app integrates mainstream emotion recognition components such as EmotionRecognizer, uploading the collected data along with tags such as time and location to the server.

[0246] After receiving multimedia information from the terminal, the server will invoke generative artificial intelligence models (such as YOLOv8, BERT, Vision Transformer, and other components implemented in PyTorch) to perform contextual and sentiment analysis on the on-site data. The server's official software modules can perform target recognition and segmentation on images, extract keywords and detect emergency calls in audio files, and, through integrated graph algorithm libraries such as NetworkX, perform segmented modeling of the road network to automatically deduce the optimal rescue route.

[0247] The server can also dynamically adjust the rescue priority of each area based on various factors such as user sentiment heatmap distribution, road conditions, and geographical environment, and issue specific instructions (including rescue routes, material types, and target locations) to the terminal. The terminal carries the materials and autonomously completes actions such as transportation, airdrop, on-site photo documentation, and timely feedback on mission completion status.

[0248] The server can connect to external rescue organization platforms and automatically and promptly push the analyzed emergency information, optimal rescue routes, user sentiment data, etc., to relevant organizations and command personnel, enabling cross-regional multi-party collaborative response.

[0249] This system also supports the Prompt operation of generative artificial intelligence models. Users, rescue commanders, and others can input prompts to quickly generate auxiliary decision-making content such as the best rescue suggestions, route recommendations, and emotional counseling plans in the corresponding scenarios.

[0250] Specific examples: For instance, during the 2024 Great East Japan Earthquake, the server used the YOLOv8 model to detect broken roads and dangerous buildings in the footage uploaded by drones, identified the extreme fear state of users in the Shinjuku area using the EmotionRecognizer, quickly calculated safe evacuation routes using the A* algorithm, and dispatched supply drones to provide support.

[0251] Example of a prompt statement:

[0252] User's current location: Shinjuku, Tokyo. Uploaded emotional message: "Extremely terrified, many people around me are crying." Please identify the current emotional level and recommend the safest escape route.

[0253] This data was collected from drones, including photos and audio files from the disaster area. Please automatically detect and mark all potentially dangerous areas and output suggested rescue routes.

[0254] Through the above-mentioned combined structure and technical solution, the present invention can significantly improve the intelligence, informatization and automation level of disaster emergency response, and realize efficient support and dispatch guided by user needs and actual on-site conditions.

[0255] use Figure 14 The processing procedure is explained.

[0256] Step 1:

[0257] Upon receiving initial disaster information, the server automatically invokes a geographic information interface to retrieve the latest maps, weather data, and shelter information for the disaster area. Inputs include disaster notifications and basic geographic environmental data. The server standardizes and integrates the data from the data sources, and outputs task instructions specific to the disaster area, including flight paths, shooting locations, and data collection types.

[0258] Step 2:

[0259] The terminal (unmanned aerial vehicle) autonomously takes off and flies to the designated disaster area according to the mission instructions issued by the server. The input is the mission instructions from the server. The terminal uses its onboard camera and microphone to periodically collect high-resolution image, video, and audio data, while also collecting location information and timestamps. The output is a raw field data packet containing multimodal information.

[0260] Step 3:

[0261] The terminal collects on-site images, videos, and audio data, compresses them in real time, and uploads them to the server via a wireless communication module (such as 4G / 5G or satellite link). The input is the collected multimedia data. The terminal preprocesses the data (such as video / image compression and audio noise reduction), and the output is a compressed data stream transmitted to the server.

[0262] Step 4:

[0263] The server receives uploaded data packets and uses artificial intelligence processing models (such as YOLOv8 image recognition and BERT) for real-time analysis. Inputs include high-quality image and audio data. The server performs target detection on images and video streams (e.g., identifying road interruptions, fires, etc.), extracts keywords from audio files (e.g., detecting the word "help"), and integrates geographic data. Outputs include on-site hazard area markings, a list of obstacle points, and a digital disaster map.

[0264] Step 5:

[0265] Users report their personal location information (GPS data) and current emotional state (text or voice) via a smart terminal app. The input consists of the user's location information and emotional content. The app automatically packages the user data, encrypts it, and uploads it to the server. The output is a user emotional data package with spatiotemporal tags.

[0266] Step 6:

[0267] The server receives user sentiment data and processes it using sentiment analysis algorithms such as EmotionRecognizer. The input includes user location information and raw sentiment data. The server performs natural language analysis on the text or speech content to determine the current user's sentiment level. The output includes the user's sentiment state (e.g., fear or anxiety level) and spatial heatmap data.

[0268] Step 7:

[0269] The server integrates multi-source data, including disaster information, dangerous area distribution, and user sentiment heatmaps, to comprehensively calculate rescue priorities and recommend optimal rescue routes. Inputs include a digital disaster map, dangerous area coordinates, and user heatmap distribution. The server uses path planning algorithms (such as A* and Dijkstra's algorithm) to search for the optimal route and dynamically adjusts the rescue order based on sentiment heatmaps. Outputs prioritized rescue instructions, the optimal route, and target coordinates.

[0270] Step 8:

[0271] After receiving rescue and material transport instructions from the server, the terminal (such as a drone) automatically selects and loads the required materials and autonomously flies to the target point along the optimal route. The input is a detailed rescue and transportation plan. After completing the designated task, the terminal takes photos for on-site evidence and reports the task completion status. The output is material delivery confirmation information and on-site verification data.

[0272] Step 9:

[0273] The server continuously aggregates all collected data, including rescue progress, emotional changes, and on-site images, and presents it to the rescue command center through a visualization system. Inputs include end-user task feedback and the latest user emotional data. The server integrates and dynamically displays the information, outputting decision support visualization panels and reports for subsequent dispatching and decision support.

[0274] Step 10:

[0275] The server interacts in real-time with the human-machine interface of external rescue organizations or managers, supporting rescuers to input prompts from the generative AI model to quickly obtain scenario-appropriate rescue plans. Input consists of human prompts and the latest multi-source data. The server sends instructions to the generative AI model, producing customized suggestion text. Output includes action guidance and decision support information tailored to the specified scenario.

[0276] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0277] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0278] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0279] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0280] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0281] Second Implementation Method

[0282] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0283] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0284] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0285] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0286] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0287] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0288] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0289] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0290] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0291] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0292] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0293] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0294] Example 1

[0295] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0296] Application Example 1

[0297] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0298] Example 2

[0299] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0300] Application Example 2

[0301] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0302] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0303] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0304] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0305] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0306] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0307] Third Implementation Method

[0308] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0309] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0310] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0311] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0312] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0313] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0314] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0315] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0316] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0317] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0318] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0319] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0320] Example 1

[0321] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0322] Application Example 1

[0323] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0324] Example 2

[0325] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0326] Application Example 2

[0327] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0328] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0329] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 includes prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0330] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0331] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0332] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0333] Fourth Implementation Method

[0334] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0335] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0336] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0337] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0338] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0339] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0340] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0341] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0342] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0343] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0344] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0345] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0346] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0347] Example 1

[0348] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0349] Application Example 1

[0350] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0351] Example 2

[0352] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0353] Application Example 2

[0354] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0355] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0356] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0357] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0358] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0359] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0360] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0361] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0362] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0363] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0364] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0365] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0366] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0367] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0368] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0369] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0370] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0371] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0372] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0373] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0374] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0375] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0376] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0377] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0378] In addition, the following notes are provided in response to the above explanation.

[0379] Example 1

[0380] (Note 1)

[0381] An information processing system includes: a device for controlling a mobile aerial device; a device for controlling a mobile image acquisition device; an information analysis device for real-time analysis of various types of acquired data using the information processing device; a path selection device for calculating the optimal path based on the analysis results of the information analysis device; a transport control device for controlling the mobile aerial device to carry and transport objects; a field information acquisition device for configuring recording devices and other equipment in geographical areas where communication methods are unavailable to acquire field information; a response information generation device for generating emergency response information using data obtained by the field information acquisition device; an information linkage device for linking the emergency response information generated by the response information generation device to external support organizations; an artificial intelligence analysis device for analyzing, classifying, and prioritizing various disaster-related information using a generative artificial intelligence model; a prompt statement generation device for generating instruction statements for disaster response and inputting them into the generative artificial intelligence model; a data integration device for integrating and automatically prioritizing heterogeneous data acquired by various types of sensors; and an integrated processing device for automatically processing a series of processes within the rescue target area, including information acquisition, analysis, path planning, material transportation, and information linkage.

[0382] (Note 2)

[0383] According to the information processing system described in Appendix 1, the airborne mobile device is automatically controlled based on the analysis results of the information analysis device and the artificial intelligence analysis device.

[0384] (Note 3)

[0385] According to the information processing system described in Appendix 1, the transportation control device automatically selects the optimal transportation plan and transportation route based on the analysis results and priority information of the generative artificial intelligence model.

[0386] Application Example 1

[0387] (Note 1)

[0388] An information processing system includes: a device for controlling a movable body to acquire remote on-site information; a device for generating data from acquired image and audio information and transmitting it to a main control device via a communication device; an information processing device for inputting received data into a generative artificial intelligence model and performing reasoning processing using prompts to determine abnormal states or urgency levels; a path calculation device for calculating the optimal action path and outputting personnel allocation and material transportation plans based on the reasoning results and spatial information; a material transportation device for automatically loading materials onto the movable body and transporting the materials to the disaster area or required location according to the optimal path; and a data acquisition device for deploying data acquisition equipment in areas where external communication is blocked, and for on-site... Non-real-time information collection devices that temporarily collect or save information and subsequently retrieve and transmit it to the main control device; emergency information generation and distribution devices that automatically generate anomaly judgments or emergency information and distribute it to remote terminals or command devices in real time; response control devices that, through a response system linked to the main control device, reflect user input or feedback information to information processing devices and path calculation devices, and dynamically adjust action plans and material delivery instructions; emotion analysis devices that, based on acquired image or audio information, analyze the on-site situation or personnel emotions through emotion recognition models and use them for rescue priority and support strategies; and prompt generation devices that generate or manage instruction statements or query statements as input to generative artificial intelligence models.

[0389] (Note 2)

[0390] According to the information processing system described in Appendix 1, the movable body can receive motion control commands based on the analysis and reasoning results of the information processing device or emotion analysis device of the generative artificial intelligence model, and autonomously collect supplementary on-site information or adjust the transportation path according to the content.

[0391] (Note 3)

[0392] According to the information processing system described in Appendix 1, the goods transportation device can comprehensively judge the urgency of the rescue target, the type of goods required, and the obstacle information of the transportation route based on the generated data and the information of the emotion analysis device, select or automatically generate the optimal transportation route, time and loading plan, and execute it.

[0393] Example 2

[0394] (Note 1)

[0395] An information processing system includes: a device for controlling a movable aerial body; a device for operating a movable imaging device; a device for acquiring on-site information and structuring it into data; a device for collecting and integrating spatial information, environmental information, refuge point information, etc., from multiple external information sources; a device for analyzing and interpreting the scene using a generative artificial intelligence model based on features extracted from multiple sensor data, images, and audio; a device for calculating and determining the optimal movement path and material delivery path based on the on-site analysis results, external information, and machine learning techniques; a device for analyzing emotional states based on data extracted from on-site audio and video and dynamically adjusting activity priorities; a device for deploying equipment in communication interruption areas and collecting and transmitting on-site data through terminals; a device for analyzing various types of on-site data and generating emergency judgment information and activity plans; a device for automatically connecting the generated emergency information and activity plans to external organizations; and a device for using a generative artificial intelligence model to analyze user-input prompts and using them as processing instructions to autonomously control the entire system.

[0396] (Note 2)

[0397] The information processing system according to Appendix 1 is characterized in that it performs dynamic optimization control on airborne mobile bodies and material delivery based on the on-site analysis results and emotional state analysis results.

[0398] (Note 3)

[0399] According to the information processing system described in Appendix 1, the material delivery route and delivery content are automatically selected and corrected based on the on-site condition analysis data and activity priority information.

[0400] Application Example 2

[0401] (Note 1)

[0402] An information processing system includes: a device for remotely or autonomously controlling an operable flight device; a device for acquiring on-site information through an operable imaging device; an artificial intelligence processing device for real-time analysis and interpretation of various information, including on-site image data and audio data, through the information processing device; a computing device for integrating multiple geographic information, meteorological information, shelter information, etc., based on the analysis and interpretation results, and calculating the optimal rescue route using a path search algorithm; a transportation device for carrying items on the flight device and transporting them to the destination area; an on-site information acquisition device for deploying communication-enabled devices in communication-disrupted areas to acquire on-site information and transmit the collected data to the information processing device; and a system for utilizing... A device for determining the urgency and importance of collected information using artificial intelligence and automatically generating emergency response information; a communication device for communicating and notifying external rescue organizations through communication networks; a device for collecting user location information and emotional data through electronic information terminals and analyzing specific emotional states using emotional analysis algorithms; a control device for dynamically adjusting the priority of rescue activities based on user emotional analysis results and on-site conditions; a comprehensive information management device for optimizing disaster response activities by integrating rescue routes, rescue resources, material transportation, on-site data, and emotional data; and a generation support device for inputting command statements into generative artificial intelligence models to generate adaptive disaster response support information.

[0403] (Note 2)

[0404] According to the information processing system described in Appendix 1, the operable flight device can automatically or semi-automatically perform operations that are helpful to disaster response activities based on the analysis and interpretation results, on-site information and sentiment analysis results.

[0405] (Note 3)

[0406] According to the information processing system described in Appendix 1, the transportation device can dynamically determine the disaster situation, the on-site environment, and the user status based on the generated data, thereby determining the optimal transportation route and the transportation object.

Claims

1. An information processing system, characterized in that, include: Operable flight device; Operable shooting device; Artificial intelligence processing device for real-time parsing and analysis of generated data; A processing device for calculating the optimal rescue route based on the analysis and interpretation results; Device for mounting and transporting supplies on the flight device; A device for deploying equipment and collecting field status data in areas where communication is interrupted; A device for analyzing the collected data and generating emergency response information; A device for linking the emergency response information with external rescue agencies.

2. The information processing system according to claim 1, characterized in that, The operable flight device is configured to perform relevant operations based on the analysis and interpretation results.

3. The information processing system according to claim 1, characterized in that, The device for loading and transporting goods can select the optimal transportation route based on the generated data.

Images