system

Abstract

We provide the system. [Solution] Equipped with a sensor device to detect vibrations, A means for automatically activating the edge computing unit when seismic motion exceeding a specific threshold is detected, A means for collecting and analyzing audio data to detect audio information that suggests the need for rescue, A means of automatically requesting rescue from an external communication network or to people in the vicinity based on voice information, A means of providing users with psychological support and first aid guidance via audio, A means of sharing a user's location information with other users or devices, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including: receiving a user utterance; adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot; encoding the prompt; and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a specific region that is a major earthquake country, the problems are that it is impossible to make a prompt and appropriate rescue request in the event of a disaster, and there is a lack of cooperation among local residents under the situation of a severe shortage of rescue teams. In addition, when the communication network becomes unavailable during a disaster, it becomes difficult to make a rescue request and share information, which is an even more serious problem. Under these circumstances, it is serious that there is a lack of information for disaster victims to effectively receive support and take safe actions.

Means for Solving the Problems

[0005] This invention provides a means for detecting voice information indicating the need for rescue by activating an edge computing unit when an earthquake motion is detected above a certain threshold, and by collecting and analyzing voice data. It also has a means for automatically requesting rescue from an external communication network or nearby people based on the voice information. Furthermore, it has a function to provide the user with psychological support and first aid guidance via voice, and a means for sharing the user's location information with other users and devices. This makes it possible to exchange information with other terminals using short-range wireless communication technology even in environments where communication is restricted, thereby promoting cooperation within the region. In addition, it solves these problems by providing a function to send messages to registered contacts to provide psychological support.

[0006] A "vibration-detecting sensor device" is an electronic component used to detect physical vibrations caused by earthquakes or other events, or a device that has such a function.

[0007] "Edge computing" refers to technology that performs data processing within the user's terminal or device, and is a distributed system located outside the communication network.

[0008] "Collecting and analyzing audio data" refers to the process of acquiring audio using devices such as microphones and analyzing its content as digital data.

[0009] A "rescue request" is the act of asking for help from rescue teams, relevant organizations, or people in the vicinity in order to receive support during a disaster or emergency.

[0010] "External communication network" refers to communication infrastructure that exists outside of the user's terminal, such as the internet or mobile phone network.

[0011] "Psychological support" is the process of providing a sense of security and emotional reassurance to disaster victims and people experiencing stress.

[0012] A "first aid manual" is a set of instructions and guidance on how to treat injured or ill people in emergencies without the need for medical assistance.

[0013] "Short-range wireless communication technology" refers to communication technologies such as Bluetooth and Wi-Fi Direct that enable direct data transmission between devices in close proximity.

[0014] "Information exchange" refers to the act or process of sending and receiving data or messages with a specific intention between multiple devices.

[0015] "Sending a message" means sending text, audio, or other information to another recipient using electronic means of communication. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be described.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention is a system implemented within a smart device to provide rapid and effective support during disasters. The system aims to ensure the safety and peace of mind of users in the event of disasters such as earthquakes. The program's functions and specific examples are described below in natural language.

[0038] Sensor detection and edge AI activation

[0039] The device is equipped with a sensor that detects earthquake vibrations, and when the sensor detects earthquake vibrations exceeding a certain threshold, the edge AI is automatically activated. Because this edge AI processes within the device, it can operate independently of network communication conditions.

[0040] Collection of audio data and rescue request

[0041] The device constantly monitors the user's voice and detects unusual sounds or voice information indicating a need for assistance. For example, if a specific keyword such as "help" is detected, an appropriate rescue request is sent. If a communication environment is available, the rescue request is automatically sent to the server. Even if communication is impossible, the device will request assistance from those around it via voice or short-range wireless communication.

[0042] Psychological support and first aid provision

[0043] Edge AI provides users with voice-based psychological support and first-aid guidance tailored to their current situation. For example, it might give specific instructions such as "Try applying pressure to stop the bleeding" to a user who is bleeding, encouraging appropriate action on the spot and ensuring safety.

[0044] Location information sharing and regional collaboration

[0045] By utilizing short-range wireless communication technology, the terminal will connect with other nearby devices to share the location and status of disaster victims. This will create an environment where users can support each other and promote mutual assistance.

[0046] Last message sent

[0047] The device also includes a feature that allows users to send pre-set messages to family and friends when a network connection is available. This helps users ensure their safety and allows them to receive emotional support from their family.

[0048] As a concrete example, consider a situation where a user is trapped and unable to move during an earthquake. In this situation, the device immediately detects the user's body movement, and the edge AI is activated. It picks up the user's cry for help, automatically sends an appropriate rescue signal, and shares location information with other nearby devices. This process allows the user to receive rescue quickly.

[0049] Thus, this system provides multifaceted support for disaster victims and contributes to increasing the rescue rate.

[0050] The following describes the processing flow.

[0051] Step 1:

[0052] The device uses sensors to detect earthquake vibrations and identifies vibrations exceeding a specific threshold. If this threshold is exceeded, the device automatically activates its built-in edge AI.

[0053] Step 2:

[0054] The edge AI is activated and begins collecting audio data on the device. It uses the microphone to capture the user's voice and ambient sounds, filtering important audio information in real time.

[0055] Step 3:

[0056] The device analyzes the user's voice in real time to determine if rescue is needed. Specifically, it uses an algorithm to identify phrases such as "help" and "I'm in pain" to determine the necessity of rescue.

[0057] Step 4:

[0058] If communication is possible, the device will automatically send a distress signal to the server. This signal will include the device's current location and voice information. If communication is not possible, the device will use short-range radio to call for assistance from nearby devices.

[0059] Step 5:

[0060] Edge AI provides voice-based psychological support and first-aid instructions tailored to the user's situation. For example, if bleeding occurs, it will offer specific advice such as, "Apply pressure with a cloth to stop the bleeding."

[0061] Step 6:

[0062] The device exchanges location information with nearby devices using Bluetooth or other short-range communication technologies. This information sharing aims to promote cooperation and mutual assistance activities within the community.

[0063] Step 7:

[0064] The system selects messages that the user has registered in advance and sends them to family and trusted contacts as network conditions permit, thereby communicating the user's situation and providing emotional support.

[0065] (Example 1)

[0066] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0067] Responding quickly and effectively during disasters is difficult, and in situations where communication methods are limited, the inability to share information appropriately and request rescue is a major challenge. Furthermore, there is a lack of systems that can immediately provide psychological support and first-aid guidance in situations where these are needed. To solve these problems, a system is needed that provides real-time information detection and sharing, rapid rescue requests, and appropriate psychological and medical support.

[0068] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0069] In this invention, the server includes sensing means for detecting vibrations, means for automatically activating a distributed processing unit when motion exceeding a specific threshold is detected, means for collecting and analyzing voice information and detecting voice data that is estimated to indicate the need for assistance, and means for automatically requesting rescue from an external network or nearby individuals based on the voice data. This enables rapid information acquisition and rescue activities without communication constraints even during disasters, and makes it possible to provide multifaceted support to disaster victims.

[0070] "Means of detecting vibrations" refers to devices and technologies for detecting earthquakes and other movements, and includes, for example, acceleration sensors and gyroscopes.

[0071] "Movement exceeding a specific threshold" refers to vibrations or movements of an intensity that exceeds a pre-set standard value by the system, and indicates the occurrence of a disaster.

[0072] "Means for automatically activating a distributed processing unit" refers to technology that starts computer processing based on data detected by sensing means, without requiring human intervention.

[0073] "Means for collecting and analyzing voice information and detecting voice data that is estimated to require assistance" refers to technology that picks up the voice emitted by the user and analyzes it to identify situations in which assistance is needed.

[0074] "Means of requesting rescue from an external network or nearby individuals" refers to a function that transmits a rescue signal using the internet or short-range communication technology based on voice data and detection results.

[0075] "Means of providing psychological support and first aid guidance via audio" refers to technologies that use audio to guide disaster victims with calming words and first aid procedures.

[0076] "Means of sharing a user's location information with other users or devices" refers to a system that communicates a user's current location to other devices or users.

[0077] An environment with restricted communication refers to a situation where network connectivity is unstable or nonexistent, and efficient communication methods are required in such situations.

[0078] "Short-range wireless communication technology" refers to technologies such as Bluetooth and Zigbee that enable data exchange over short distances.

[0079] The "function of providing psychological support" refers to the function of providing support to disaster-affected users, such as offering encouragement and promoting a stable mental state.

[0080] This invention is a system designed to provide rapid and effective assistance during disasters. It is primarily implemented in smart devices and aims to ensure user safety and peace of mind during earthquakes and other disasters.

[0081] The device is equipped with vibration detection mechanisms, constantly monitoring vibrations using accelerometers and gyroscopes. When motion exceeding a threshold is detected, the device automatically activates a distributed processing unit, and the edge AI begins operation. This edge AI uses a lightweight machine learning library such as TENSORFLOW® Lite, and performs the necessary data analysis and decision-making on the device.

[0082] The device utilizes speech recognition technology to collect and analyze user voice information. For example, by using Google's (registered trademark) speech recognition API, the device analyzes user speech in real time and detects keywords indicating a need for assistance, such as "help." If the device determines that the user is in a critical situation, it automatically requests rescue.

[0083] The server checks whether external communication is possible and, if so, sends a rescue signal over the network. If communication is restricted, the device uses short-range wireless communication technologies such as Bluetooth or Zigbee to notify other nearby devices of the rescue request. The device also provides the user with voice guidance on psychological support and first aid. This allows the user to respond quickly and appropriately at the scene.

[0084] A concrete example would be a scenario where, upon detecting an earthquake, the device senses the user's cry for help, immediately shares location information with nearby devices, and requests rescue.

[0085] An example of a prompt message would be, "Please explain how to provide rapid support using smart devices during a disaster."

[0086] Thus, the terminal provides diverse support in disaster situations, contributing to the rescue and safety of users.

[0087] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0088] Step 1:

[0089] The device continuously collects data using sensing means that detect ambient vibrations. The input is vibration data acquired from an accelerometer, and the output is a determination of vibrations exceeding a specific threshold. The real-time vibration data acquired by the device is recorded in internal memory and compared with the threshold. If the threshold is exceeded, the following processing is performed.

[0090] Step 2:

[0091] When the device detects vibrations exceeding a threshold, it activates the distributed processing unit. The input is the output of step 1, i.e., the vibration detection, and the output is the activation signal for the edge AI. The device activates the distributed processing unit, and the edge AI begins operation. At this time, a lightweight model such as TensorFlow Lite is loaded on the device, and it is ready to perform analysis.

[0092] Step 3:

[0093] The device collects the user's voice and analyzes it through speech recognition. The input is voice data from the microphone, and the output is the detection result of keywords indicating the need for assistance. The voice is analyzed in real time using a speech recognition API, and when phrases indicating the need for assistance, such as "help," are identified, the system proceeds to process the rescue request.

[0094] Step 4:

[0095] Based on the results of voice analysis, the terminal sends a rescue request to an external network via the server. The input is the keyword detection result, which is the output of step 3, and the output is the transmission of the rescue request signal. If a network connection is available, the terminal sends an emergency signal via the server. If communication is unstable, short-range wireless communication technology is used to notify other nearby devices of the request.

[0096] Step 5:

[0097] The device uses edge AI to generate voice guidance for the user, providing psychological and medical support. The input is the data processing results from steps 1 to 4, and the output is voice guidance for the user. The device immediately provides the user with specific first-aid instructions via voice through its speaker, such as "Try applying pressure to stop the bleeding."

[0098] (Application Example 1)

[0099] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0100] In modern society, it is difficult for individuals to receive prompt and effective rescue during disasters. Furthermore, there are challenges in sharing information in environments with limited communication, and in ensuring safety and security among individuals.

[0101] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0102] In this invention, the server includes a sensor device for detecting vibrations and means for automatically activating the edge computing unit when it detects seismic motion exceeding a certain threshold; means for automatically requesting rescue from an external communication network or nearby individuals when it recognizes a specific keyword; and means for sharing the user's location information with other devices using short-range wireless communication technology. This enables reliable and rapid rescue requests and information sharing even in the event of a disaster.

[0103] A "vibration detection sensor device" is a device that senses ambient vibrations and notifies the user if they exceed a certain threshold.

[0104] The "Edge Computing Department" is a computer division that processes data within the terminal to enable quick responses.

[0105] "Means for collecting and analyzing audio data" refers to technologies for recording ambient sounds and analyzing their content.

[0106] "Recognizing specific keywords" refers to the function of identifying predefined important words from audio data.

[0107] "Means of requesting rescue via an external communication network" refers to a method of requesting assistance through an external communication network in an emergency.

[0108] "Short-range wireless communication technology" refers to technology that enables data transmission between devices over short distances.

[0109] "Means of sharing user location information" refers to technologies for exchanging a user's geographical location with other devices or systems.

[0110] The system program for realizing this invention utilizes a vibration-detecting sensor device, an edge computing unit, voice data collection and analysis means, and short-range wireless communication technology, and is designed to enable users to receive rapid and effective assistance during disasters.

[0111] The devices used are smartphones and other mobile information terminals, which are equipped with hardware including accelerometers, microphones, and GPS. The software utilizes edge AI based on TensorFlow Lite, performing real-time speech recognition and anomaly detection. When the accelerometer detects shaking exceeding a specific threshold during an earthquake, the edge AI immediately initiates the necessary processing.

[0112] The edge AI continuously collects voice data and, upon recognizing "help" or other specific keywords, sends a rescue request to the surrounding area via an external communication network or short-range wireless communication. Furthermore, by sharing location information with nearby devices using short-range wireless communication technology, it facilitates communication among people in disaster areas.

[0113] The server also has a function that automatically sends messages to pre-registered contacts to provide emotional support when a stable communication environment is established.

[0114] As a concrete example, if an earthquake occurs in a park and a user is injured and cries out for help, the device will detect the shaking and the sound and immediately send out a rescue signal. This signal will be received by surrounding devices, and support will be provided quickly from there.

[0115] An example of a prompt message is, "My smartphone has an AI-powered emergency response app that automatically activates during earthquakes and accidents, recognizing cries for help and sending out rescue requests. How do you usually think about emergency preparedness?" This allows users to be more mindful of disaster preparedness on a daily basis.

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] The device constantly monitors ambient vibrations using an accelerometer. Its input is acceleration data related to seismic motion. When it detects vibrations exceeding a specific threshold, it generates a signal to activate the edge computing unit. The operation involves setting a certain threshold value, and activating the on-device AI when that value is exceeded.

[0119] Step 2:

[0120] The edge computing unit collects and analyzes audio data in real time. The input data is ambient noise, and the output is a determination of whether or not a rescue request is necessary. Specifically, it uses a generative AI model to analyze the audio and activates the rescue request process when it detects a specific keyword (e.g., "help").

[0121] Step 3:

[0122] The terminal transmits a rescue request via an external communication network or short-range wireless communication. The input is keyword detection results from the edge computing unit, and the output is the generation of an emergency signal. Short-range communication utilizes technologies such as Bluetooth and Wi-Fi Direct.

[0123] Step 4:

[0124] The server sends a message to pre-registered contacts when communication conditions are met. The input is a set of rescue request and location information, and the output is a reassuring message automatically sent to family and close friends. In practice, a regular automatic message sending schedule is set up.

[0125] Step 5:

[0126] This system allows users to share their location and status with other devices via short-range wireless communication. Inputs include an individual's location and current status metadata, while output is information synchronization between devices. Specifically, it involves forming an ad-hoc network among nearby individuals.

[0127] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0128] This invention provides a system for providing rapid and appropriate support during disasters, taking into account the user's emotional state, and is particularly implemented within mobile devices. Specifically, when a natural disaster such as an earthquake occurs, the terminal automatically activates edge AI to collect and analyze the user's voice data. At this time, it uses an emotion engine to recognize the user's emotional state and provides optimal psychological support based on that, as well as performing necessary rescue measures.

[0129] Sensor detection and the use of an emotion engine

[0130] The device uses sensors to detect earthquake vibrations, and when vibrations exceed a certain threshold are detected, the edge AI is activated. Once voice data is collected, the emotion engine analyzes it and begins to determine the user's emotional state. This allows it to infer whether the user is experiencing various emotions such as anxiety, fear, or calmness.

[0131] Improvements to voice analysis and rescue requests

[0132] The device analyzes the user's voice and uses an emotion engine to optimize the content and urgency of the emergency request. For example, if the emotion engine analyzes the user's voice as indicating high levels of fear or panic, it determines that higher priority action is needed and reflects this in the rescue signal.

[0133] Psychological support and first aid provision

[0134] The edge AI analyzes the emotion engine to suggest personalized psychological support to the user. For example, in situations where calmness is needed, it provides voice guidance encouraging relaxation, while conversely, if anxiety is high, it offers messages that include encouragement.

[0135] Integration and information sharing with other devices

[0136] Using short-range wireless communication technology, terminals connect with nearby terminals to share location information, including urgent data and current emotional states. This creates a network where other users in the same area can cooperate to ensure safety.

[0137] Emotion-based messaging

[0138] Users can send pre-configured messages to family and trusted contacts, accompanied by information about their current emotional state generated by an emotion engine. This allows recipients to understand the user's actual mental state and respond appropriately.

[0139] For example, if the emotional engine detects a user in a panic state during an earthquake and determines that the user is unable to act independently, this information will be used to strengthen rescue signals and take measures to encourage prompt assistance. In this way, this system aims to enhance multifaceted support for users during disasters and improve rescue rates.

[0140] The following describes the processing flow.

[0141] Step 1:

[0142] The device uses sensors to detect earthquake vibrations, and when it detects earthquake motion exceeding a specific threshold, it automatically activates edge AI.

[0143] Step 2:

[0144] The edge AI begins collecting voice data from the user. Using a microphone, it continuously monitors the surrounding sounds and prepares to filter and analyze meaningful voice data.

[0145] Step 3:

[0146] The device uses an emotion engine to analyze collected audio data and understand the user's emotional state. For example, it can determine emotions such as fear or relief from the pitch, tempo, and specific phrases of the voice.

[0147] Step 4:

[0148] Based on the user's emotional state, the device automatically adjusts the content of the rescue request. If the emotional engine indicates a state of panic, the urgency of the rescue request is increased to encourage a quick response.

[0149] Step 5:

[0150] Based on the results of emotion recognition, the edge AI provides users with appropriate psychological support and first-aid advice via voice. It offers specific instructions such as prompting them to take deep breaths to reduce anxiety and providing first aid for minor injuries.

[0151] Step 6:

[0152] The terminal utilizes short-range wireless communication technology to share information, including the user's location and emotional state, with nearby terminals, thereby facilitating rapid cooperation within the region.

[0153] Step 7:

[0154] The device will send messages containing emotional states to important contacts pre-configured by the user via the network, allowing them to receive accurate updates on their situation and emotional support.

[0155] (Example 2)

[0156] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0157] During natural disasters such as earthquakes, it is essential to provide immediate and appropriate support that takes into account the emotional state of users. Conventional systems are insufficient in providing optimal support based on users' psychological states, making rapid rescue operations and information sharing difficult. Furthermore, effective collaboration in environments with limited communication is a critical challenge.

[0158] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0159] In this invention, the server includes means for automatically activating a peripheral computing device when it detects environmental changes exceeding a specific threshold, which are equipped with a vibration-detecting sensor; means for collecting and analyzing voice and detecting voice information that is estimated to require assistance; and means for analyzing the user's voice and determining their emotional state. This enables the optimization of urgency based on the user's emotional state and the rapid adjustment of assistance signals, thereby providing effective support.

[0160] A "vibration sensor" is a device designed to detect physical vibrations and environmental changes, and can detect movements or vibrations that exceed a specific threshold.

[0161] "Peripheral computing devices" refer to computing resources used for data processing near user terminals, enabling rapid data analysis and processing.

[0162] "Means of collecting audio" refers to the process of acquiring the voice emitted by a user using an audio device such as a microphone and treating it as data.

[0163] "Voice information that suggests the need for support" refers to information analyzed from voice data emitted by the user, which may indicate urgency or the need for support.

[0164] "Means for analyzing voice and determining emotional state" refers to a function that analyzes voice data, evaluates its content and tone, and identifies the user's emotional state.

[0165] "Urgency optimization" is the process of assessing the urgency of the need for assistance and rescue based on collected data, and setting appropriate priorities.

[0166] "Adjusting support signals" refers to customizing signals to communicate the requested support and its priority based on the user's situation, and to relevant organizations and other users.

[0167] This invention relates to a system for providing rapid and appropriate support during disasters, taking into account the user's emotional state. Specifically, it is implemented within a mobile device, collects the user's voice data during a disaster, and analyzes it using edge AI.

[0168] The device incorporates a highly sensitive vibration sensor, enabling it to detect vibrations from natural disasters. When a specific threshold is exceeded, peripheral computing devices are activated to quickly collect audio data on-site. The audio data is acquired through a built-in microphone and then analyzed in real time by edge AI.

[0169] The edge AI incorporates an emotion engine that precisely analyzes the tone and content of voice to determine the user's emotional state. This analysis process utilizes voice analysis software and machine learning algorithms. Based on the user's situation, it optimizes the urgency level and sends a distress signal as needed.

[0170] For example, if a user urgently says "Please help me," the system will determine it to be a high-priority emergency and immediately request assistance. Additionally, as psychological support, it will play a voice message encouraging the user to relax, thus helping to stabilize their mental state.

[0171] To share information with other users and devices, terminals utilize short-range wireless communication technology. This enables data exchange with other terminals even in limited communication environments, promoting cooperation within a region.

[0172] Furthermore, messages based on emotional states can be sent to pre-registered contacts. This process allows recipients to understand the user's true emotional state and take appropriate action.

[0173] An example of a prompt message would be, "Explain the procedure for analyzing the user's voice during an earthquake and providing optimal psychological support based on their emotional state." This invention enables comprehensive and efficient user support during disasters.

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The device detects environmental vibrations. A built-in vibration sensor monitors ambient fluctuations in real time, and when vibrations exceeding a specific threshold are detected, the data is input to the edge AI. Based on this input, the edge AI generates and outputs a signal to activate peripheral computing devices.

[0177] Step 2:

[0178] The device collects voice data. When a disaster is detected, the device automatically activates the microphone and collects the user's voice. The collected voice data becomes input for voice analysis and is sent to the next processing stage. The device temporarily stores the voice data in a buffer.

[0179] Step 3:

[0180] Edge AI analyzes the voice data. Voice analysis software uses machine learning algorithms to analyze the input data, and the emotion engine identifies the user's emotional state. Here, it evaluates the tone of voice and the content of phrases, and outputs emotional states such as anxiety and fear. The analysis results become the main input for the next step.

[0181] Step 4:

[0182] The device determines the urgency and adjusts the rescue signal. Based on the analysis results of the emotion engine, the edge AI calculates the urgency. If a high urgency is indicated, the device inputs this into its external communication function and sends an enhanced rescue signal as quickly as possible. The signal adjusted here includes priority information and the user's current location.

[0183] Step 5:

[0184] The server generates psychological support content. Based on the emotional state, a generation AI model designs and outputs an audio message. The output message includes instructions to promote relaxation and words of encouragement, which are sent to the user's device and played back as audio.

[0185] Step 6:

[0186] The device shares information with other devices using short-range wireless communication. Using locally available short-range wireless technology, the device transmits emotional state and location information to other devices in the vicinity. This facilitates data exchange with other users in the same area and enables the establishment of collaborative systems.

[0187] Step 7:

[0188] The device sends its emotional state to its contacts. A message containing the user's emotional state is sent to pre-registered contacts. The message is delivered via the network, allowing recipients to understand the user's situation accurately and design their response accordingly.

[0189] (Application Example 2)

[0190] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0191] In recent years, natural disasters have become more frequent, requiring swift and accurate evacuation during emergencies. However, the confusion and panic during disasters often hinder appropriate evacuation actions, and there is a lack of support tailored to users' emotional states. Therefore, there is a need for technology that can analyze users' emotional states in real time during disasters and support appropriate evacuation actions.

[0192] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0193] In this invention, the server includes a sensor device for detecting vibrations and means for automatically activating an edge computing unit when it detects seismic motion exceeding a specific threshold; means for collecting and analyzing voice data and detecting voice information that is estimated to indicate the need for rescue; and means for analyzing the user's emotional state and optimizing evacuation movement. This enables the provision of psychological support tailored to the user's emotional state during a disaster, allowing for optimal evacuation actions.

[0194] A "vibration-detecting sensor device" is a device that detects physical vibrations and processes that information as an electrical signal.

[0195] "Means for activating the edge computing unit" refers to methods for automatically activating edge computing functions for processing data within a device.

[0196] "Means for collecting and analyzing audio data to detect audio information that suggests the need for rescue" refers to technology that identifies situations requiring rescue by recording and analyzing audio.

[0197] "Means of requesting assistance from external communication networks or people nearby" refers to means of seeking help from external networks or people in the vicinity in an emergency.

[0198] "Methods for providing psychological support and first aid instructions via audio" refer to methods for providing users with a sense of security and guiding them through simple first aid procedures via audio.

[0199] "Means of sharing a user's location information with other users or devices" refers to technologies that allow a user to share their current location with other devices or individuals through communication.

[0200] "Methods for analyzing emotional states and optimizing evacuation routes" refers to technology that instructs users to evacuate via the most optimal route and method based on the results of analyzing their emotions.

[0201] "Means for displaying relaxation content within an autonomous mobile vehicle" refers to a function for displaying content intended to alleviate user tension inside an autonomously operating mobile vehicle.

[0202] The system implementing this invention operates with multiple functions working in coordination to ensure user safety during disasters. As basic hardware, it is equipped with a vibration-detecting sensor device that can detect physical vibrations such as earthquakes. When vibrations exceeding a certain threshold are detected, the edge computing unit automatically activates.

[0203] The server uses speech recognition technology and an emotion analysis engine to collect and analyze the user's voice data. This analysis allows the server to determine the user's emotional state in real time and, if necessary, initiate a rescue request. For example, if fear or panic is detected, an emergency signal is sent via an external communication network to prompt a more immediate response.

[0204] The device utilizes voice guidance to provide psychological support to the user. This allows for the real-time delivery of relaxing voice guides and encouraging messages to help the user feel more at ease. For example, when a user is using an autonomous vehicle (e.g., a self-driving car) during an evacuation, relaxation content can be displayed inside the vehicle.

[0205] Furthermore, short-range wireless communication technology allows for the exchange of information with other devices, strengthening cooperation within a region. This makes it easier for users to form secure networks in cooperation with other users and devices.

[0206] As a concrete example, consider a scenario where a user experiences an earthquake and their smartphone analyzes their emotions. In this case, the smartphone uses a "generative AI model" to determine their emotional state and provide optimal route guidance to a safe evacuation site. Furthermore, when boarding a vehicle, a video designed to help the user regain their composure is played on the in-car screen.

[0207] Examples of prompts include the following:

[0208] "In the event of a disaster, how can we provide rapid support to users who are in a state of panic?"

[0209] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0210] Step 1:

[0211] The device detects physical vibrations using a vibration sensor. Vibration data is obtained from the sensor as input, and the edge computing unit is activated when the vibration exceeds a certain threshold. A signal is generated as output that activates edge computing.

[0212] Step 2:

[0213] The server collects audio data and converts it into text data using speech recognition technology. The user's voice is sent to the server as input, and text data is generated as output. At this stage, the collected audio data is ready for analysis.

[0214] Step 3:

[0215] The server inputs text data into an emotion analysis engine to determine the user's emotional state. Text data is passed to the emotion analysis engine as input, and the emotional state (e.g., normal, panic, fear, etc.) is determined as output. A generative AI model is used for emotion determination.

[0216] Step 4:

[0217] Based on the emotion analysis results, the server determines the priority of the rescue request and, if necessary, sends an emergency signal via the external communication network. Emotional state data is provided as input, and a rescue request signal is generated as output. High priority requests are immediately notified to the communication network.

[0218] Step 5:

[0219] The device automatically generates and delivers psychological support messages via voice based on the user's emotional state. It generates prompts based on emotional state data as input, and plays voice guidance to the user as output. For example, if the user is in a panic state, a voice guide encouraging relaxation will play.

[0220] Step 6:

[0221] The autonomous mobile vehicle follows instructions from a server, determines an evacuation route based on the user's emotional state, and guides the user safely. Evacuation route data and the user's current location information are used as input, and the optimal route guidance is reflected in the vehicle's navigation system as output.

[0222] Step 7:

[0223] Relaxation content is played inside the vehicle. The user's emotional state and related content information are provided to the in-car display system as input, and relaxing video and audio content is presented as output. The user can view the content and experience stress reduction.

[0224] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0225] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0226] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0227] [Second Embodiment]

[0228] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0229] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0230] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0231] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0232] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0233] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0234] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0235] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0236] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0237] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0238] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0239] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0240] This invention is a system implemented within a smart device to provide rapid and effective support during disasters. The system aims to ensure the safety and peace of mind of users in the event of disasters such as earthquakes. The program's functions and specific examples are described below in natural language.

[0241] Sensor detection and edge AI activation

[0242] The device is equipped with a sensor that detects earthquake vibrations, and when the sensor detects earthquake vibrations exceeding a certain threshold, the edge AI is automatically activated. Because this edge AI processes within the device, it can operate independently of network communication conditions.

[0243] Collection of audio data and rescue request

[0244] The device constantly monitors the user's voice and detects unusual sounds or voice information indicating a need for assistance. For example, if a specific keyword such as "help" is detected, an appropriate rescue request is sent. If a communication environment is available, the rescue request is automatically sent to the server. Even if communication is impossible, the device will request assistance from those around it via voice or short-range wireless communication.

[0245] Psychological support and first aid provision

[0246] Edge AI provides users with voice-based psychological support and first-aid guidance tailored to their current situation. For example, it might give specific instructions such as "Try applying pressure to stop the bleeding" to a user who is bleeding, encouraging appropriate action on the spot and ensuring safety.

[0247] Location information sharing and regional collaboration

[0248] By utilizing short-range wireless communication technology, the terminal will connect with other nearby devices to share the location and status of disaster victims. This will create an environment where users can support each other and promote mutual assistance.

[0249] Last message sent

[0250] The device also includes a feature that allows users to send pre-set messages to family and friends when a network connection is available. This helps users ensure their safety and allows them to receive emotional support from their family.

[0251] As a concrete example, consider a situation where a user is trapped and unable to move during an earthquake. In this situation, the device immediately detects the user's body movement, and the edge AI is activated. It picks up the user's cry for help, automatically sends an appropriate rescue signal, and shares location information with other nearby devices. This process allows the user to receive rescue quickly.

[0252] Thus, this system provides multifaceted support for disaster victims and contributes to increasing the rescue rate.

[0253] The following describes the processing flow.

[0254] Step 1:

[0255] The device uses sensors to detect earthquake vibrations and identifies vibrations exceeding a specific threshold. If this threshold is exceeded, the device automatically activates its built-in edge AI.

[0256] Step 2:

[0257] The edge AI is activated and begins collecting audio data on the device. It uses the microphone to capture the user's voice and ambient sounds, filtering important audio information in real time.

[0258] Step 3:

[0259] The device analyzes the user's voice in real time to determine if rescue is needed. Specifically, it uses an algorithm to identify phrases such as "help" and "I'm in pain" to determine the necessity of rescue.

[0260] Step 4:

[0261] If communication is possible, the device will automatically send a distress signal to the server. This signal will include the device's current location and voice information. If communication is not possible, the device will use short-range radio to call for assistance from nearby devices.

[0262] Step 5:

[0263] Edge AI provides voice-based psychological support and first-aid instructions tailored to the user's situation. For example, if bleeding occurs, it will offer specific advice such as, "Apply pressure with a cloth to stop the bleeding."

[0264] Step 6:

[0265] The device exchanges location information with nearby devices using Bluetooth or other short-range communication technologies. This information sharing aims to promote cooperation and mutual assistance activities within the community.

[0266] Step 7:

[0267] The system selects messages that the user has registered in advance and sends them to family and trusted contacts as network conditions permit, thereby communicating the user's situation and providing emotional support.

[0268] (Example 1)

[0269] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0270] Responding quickly and effectively during disasters is difficult, and in situations where communication methods are limited, the inability to share information appropriately and request rescue is a major challenge. Furthermore, there is a lack of systems that can immediately provide psychological support and first-aid guidance in situations where these are needed. To solve these problems, a system is needed that provides real-time information detection and sharing, rapid rescue requests, and appropriate psychological and medical support.

[0271] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0272] In this invention, the server includes sensing means for detecting vibrations, means for automatically activating a distributed processing unit when motion exceeding a specific threshold is detected, means for collecting and analyzing voice information and detecting voice data that is estimated to indicate the need for assistance, and means for automatically requesting rescue from an external network or nearby individuals based on the voice data. This enables rapid information acquisition and rescue activities without communication constraints even during disasters, and makes it possible to provide multifaceted support to disaster victims.

[0273] "Means of detecting vibrations" refers to devices and technologies for detecting earthquakes and other movements, and includes, for example, acceleration sensors and gyroscopes.

[0274] "Movement exceeding a specific threshold" refers to vibrations or movements of an intensity that exceeds a pre-set standard value by the system, and indicates the occurrence of a disaster.

[0275] "Means for automatically activating a distributed processing unit" refers to technology that starts computer processing based on data detected by sensing means, without requiring human intervention.

[0276] "Means for collecting and analyzing voice information and detecting voice data that is estimated to require assistance" refers to technology that picks up the voice emitted by the user and analyzes it to identify situations in which assistance is needed.

[0277] "Means of requesting rescue from an external network or nearby individuals" refers to a function that transmits a rescue signal using the internet or short-range communication technology based on voice data and detection results.

[0278] "Means of providing psychological support and first aid guidance via audio" refers to technologies that use audio to guide disaster victims with calming words and first aid procedures.

[0279] "Means of sharing a user's location information with other users or devices" refers to a system that communicates a user's current location to other devices or users.

[0280] An environment with restricted communication refers to a situation where network connectivity is unstable or nonexistent, and efficient communication methods are required in such situations.

[0281] "Short-range wireless communication technology" refers to technologies such as Bluetooth and Zigbee that enable data exchange over short distances.

[0282] The "function of providing psychological support" refers to the function of providing support to disaster-affected users, such as offering encouragement and promoting a stable mental state.

[0283] This invention is a system designed to provide rapid and effective assistance during disasters. It is primarily implemented in smart devices and aims to ensure user safety and peace of mind during earthquakes and other disasters.

[0284] The device is equipped with vibration detection mechanisms, constantly monitoring vibrations using accelerometers and gyroscopes. When motion exceeding a threshold is detected, the device automatically activates a distributed processing unit, and the edge AI begins operation. This edge AI uses lightweight machine learning libraries such as TensorFlow Lite to perform necessary data analysis and decision-making on the device.

[0285] The terminal utilizes speech recognition technology to collect and analyze the user's voice information. For example, by using Google's speech recognition API, the terminal can analyze the user's speech in real-time and detect keywords such as "help" that require assistance. As a result, if the user is determined to be in a critical situation, a rescue request will be automatically issued.

[0286] The server checks the availability of external communication and, if possible, sends a rescue signal via the network. If communication is restricted, the terminal uses short-range wireless communication technologies such as Bluetooth or Zigbee to notify other surrounding terminals of the rescue request. In addition, the terminal provides the user with psychological support and voice guidance on first aid procedures. This enables the user to respond quickly and appropriately on-site.

[0287] As a specific example, when the terminal detects an earthquake and senses the user's cry for "help", it can immediately share location information with nearby devices while issuing a rescue request.

[0288] Examples of prompt texts include "Please explain the method for quickly providing support for smart devices during disasters."

[0289] In this way, the terminal is a system that provides various supports in disaster situations and contributes to the rescue and safety of users.

[0290] The flow of the specific process in Example 1 will be described using FIG. 11.

[0291] Step 1:

[0292] The device continuously collects data using sensing means that detect ambient vibrations. The input is vibration data acquired from an accelerometer, and the output is a determination of vibrations exceeding a specific threshold. The real-time vibration data acquired by the device is recorded in internal memory and compared with the threshold. If the threshold is exceeded, the following processing is performed.

[0293] Step 2:

[0294] When the device detects vibrations exceeding a threshold, it activates the distributed processing unit. The input is the output of step 1, i.e., the vibration detection, and the output is the activation signal for the edge AI. The device activates the distributed processing unit, and the edge AI begins operation. At this time, a lightweight model such as TensorFlow Lite is loaded on the device, and it is ready to perform analysis.

[0295] Step 3:

[0296] The device collects the user's voice and analyzes it through speech recognition. The input is voice data from the microphone, and the output is the detection result of keywords indicating the need for assistance. The voice is analyzed in real time using a speech recognition API, and when phrases indicating the need for assistance, such as "help," are identified, the system proceeds to process the rescue request.

[0297] Step 4:

[0298] Based on the results of voice analysis, the terminal sends a rescue request to an external network via the server. The input is the keyword detection result, which is the output of step 3, and the output is the transmission of the rescue request signal. If a network connection is available, the terminal sends an emergency signal via the server. If communication is unstable, short-range wireless communication technology is used to notify other nearby devices of the request.

[0299] Step 5:

[0300] The terminal uses edge AI to generate voice guidance for the user and provide psychological and medical support. The input is the data processing result from steps 1 to 4, and the output is the voice guidance for the user. The terminal immediately provides the user with voice instructions for specific first aid measures such as "Please try the compression hemostasis method" through the speaker.

[0301] (Application Example 1)

[0302] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0303] In modern society, there are situations where it is difficult for individuals to receive prompt and effective assistance during disasters. There are also issues regarding information sharing in environments with restricted communication and how to ensure the safety and security of individuals.

[0304] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0305] In this invention, the server includes a sensor device for detecting vibrations, means for automatically starting the edge computing unit when seismic vibrations exceeding a specific threshold are sensed, means for automatically sending a rescue request to an external communication network or surrounding individuals when a specific keyword is recognized, and means for leveraging short - range wireless communication technology to link and share the user's location information with other devices. This enables reliable and rapid rescue requests and information sharing even during disasters.

[0306] The "sensor device for detecting vibrations" is a device that senses ambient vibrations and notifies them when they exceed a certain standard.

[0307] The "edge computing unit" is a computer department for processing data within the terminal to enable quick responses.

[0308] "Means for collecting and analyzing audio data" refers to technologies for recording ambient sounds and analyzing their content.

[0309] "Recognizing specific keywords" refers to the function of identifying predefined important words from audio data.

[0310] "Means of requesting rescue via an external communication network" refers to a method of requesting assistance through an external communication network in an emergency.

[0311] "Short-range wireless communication technology" refers to technology that enables data transmission between devices over short distances.

[0312] "Means of sharing user location information" refers to technologies for exchanging a user's geographical location with other devices or systems.

[0313] The system program for realizing this invention utilizes a vibration-detecting sensor device, an edge computing unit, voice data collection and analysis means, and short-range wireless communication technology, and is designed to enable users to receive rapid and effective assistance during disasters.

[0314] The devices used are smartphones and other mobile information terminals, which are equipped with hardware including accelerometers, microphones, and GPS. The software utilizes edge AI based on TensorFlow Lite, performing real-time speech recognition and anomaly detection. When the accelerometer detects shaking exceeding a specific threshold during an earthquake, the edge AI immediately initiates the necessary processing.

[0315] The edge AI continuously collects voice data and, upon recognizing "help" or other specific keywords, sends a rescue request to the surrounding area via an external communication network or short-range wireless communication. Furthermore, by sharing location information with nearby devices using short-range wireless communication technology, it facilitates communication among people in disaster areas.

[0316] The server also has a function that automatically sends messages to pre-registered contacts to provide emotional support when a stable communication environment is established.

[0317] As a concrete example, if an earthquake occurs in a park and a user is injured and cries out for help, the device will detect the shaking and the sound and immediately send out a rescue signal. This signal will be received by surrounding devices, and support will be provided quickly from there.

[0318] An example of a prompt message is, "My smartphone has an AI-powered emergency response app that automatically activates during earthquakes and accidents, recognizing cries for help and sending out rescue requests. How do you usually think about emergency preparedness?" This allows users to be more mindful of disaster preparedness on a daily basis.

[0319] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0320] Step 1:

[0321] The device constantly monitors ambient vibrations using an accelerometer. Its input is acceleration data related to seismic motion. When it detects vibrations exceeding a specific threshold, it generates a signal to activate the edge computing unit. The operation involves setting a certain threshold value, and activating the on-device AI when that value is exceeded.

[0322] Step 2:

[0323] The edge computing unit collects and analyzes audio data in real time. The input data is ambient noise, and the output is a determination of whether or not a rescue request is necessary. Specifically, it uses a generative AI model to analyze the audio and activates the rescue request process when it detects a specific keyword (e.g., "help").

[0324] Step 3:

[0325] The terminal transmits a rescue request via an external communication network or short-range wireless communication. The input is keyword detection results from the edge computing unit, and the output is the generation of an emergency signal. Short-range communication utilizes technologies such as Bluetooth and Wi-Fi Direct.

[0326] Step 4:

[0327] The server sends a message to pre-registered contacts when communication conditions are met. The input is a set of rescue request and location information, and the output is a reassuring message automatically sent to family and close friends. In practice, a regular automatic message sending schedule is set up.

[0328] Step 5:

[0329] This system allows users to share their location and status with other devices via short-range wireless communication. Inputs include an individual's location and current status metadata, while output is information synchronization between devices. Specifically, it involves forming an ad-hoc network among nearby individuals.

[0330] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0331] This invention provides a system for providing rapid and appropriate support during disasters, taking into account the user's emotional state, and is particularly implemented within mobile devices. Specifically, when a natural disaster such as an earthquake occurs, the terminal automatically activates edge AI to collect and analyze the user's voice data. At this time, it uses an emotion engine to recognize the user's emotional state and provides optimal psychological support based on that, as well as performing necessary rescue measures.

[0332] Sensor detection and the use of an emotion engine

[0333] The device uses sensors to detect earthquake vibrations, and when vibrations exceed a certain threshold are detected, the edge AI is activated. Once voice data is collected, the emotion engine analyzes it and begins to determine the user's emotional state. This allows it to infer whether the user is experiencing various emotions such as anxiety, fear, or calmness.

[0334] Improvements to voice analysis and rescue requests

[0335] The device analyzes the user's voice and uses an emotion engine to optimize the content and urgency of the emergency request. For example, if the emotion engine analyzes the user's voice as indicating high levels of fear or panic, it determines that higher priority action is needed and reflects this in the rescue signal.

[0336] Psychological support and first aid provision

[0337] The edge AI analyzes the emotion engine to suggest personalized psychological support to the user. For example, in situations where calmness is needed, it provides voice guidance encouraging relaxation, while conversely, if anxiety is high, it offers messages that include encouragement.

[0338] Integration and information sharing with other devices

[0339] Using short-range wireless communication technology, terminals connect with nearby terminals to share location information, including urgent data and current emotional states. This creates a network where other users in the same area can cooperate to ensure safety.

[0340] Emotion-based messaging

[0341] Users can send pre-configured messages to family and trusted contacts, accompanied by information about their current emotional state generated by an emotion engine. This allows recipients to understand the user's actual mental state and respond appropriately.

[0342] For example, if the emotional engine detects a user in a panic state during an earthquake and determines that the user is unable to act independently, this information will be used to strengthen rescue signals and take measures to encourage prompt assistance. In this way, this system aims to enhance multifaceted support for users during disasters and improve rescue rates.

[0343] The following describes the processing flow.

[0344] Step 1:

[0345] The device uses sensors to detect earthquake vibrations, and when it detects earthquake motion exceeding a specific threshold, it automatically activates edge AI.

[0346] Step 2:

[0347] The edge AI begins collecting voice data from the user. Using a microphone, it continuously monitors the surrounding sounds and prepares to filter and analyze meaningful voice data.

[0348] Step 3:

[0349] The device uses an emotion engine to analyze collected audio data and understand the user's emotional state. For example, it can determine emotions such as fear or relief from the pitch, tempo, and specific phrases of the voice.

[0350] Step 4:

[0351] Based on the user's emotional state, the device automatically adjusts the content of the rescue request. If the emotional engine indicates a state of panic, the urgency of the rescue request is increased to encourage a quick response.

[0352] Step 5:

[0353] Based on the results of emotion recognition, the edge AI provides users with appropriate psychological support and first-aid advice via voice. It offers specific instructions such as prompting them to take deep breaths to reduce anxiety and providing first aid for minor injuries.

[0354] Step 6:

[0355] The terminal utilizes short-range wireless communication technology to share information, including the user's location and emotional state, with nearby terminals, thereby facilitating rapid cooperation within the region.

[0356] Step 7:

[0357] The device will send messages containing emotional states to important contacts pre-configured by the user via the network, allowing them to receive accurate updates on their situation and emotional support.

[0358] (Example 2)

[0359] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0360] During natural disasters such as earthquakes, it is essential to provide immediate and appropriate support that takes into account the emotional state of users. Conventional systems are insufficient in providing optimal support based on users' psychological states, making rapid rescue operations and information sharing difficult. Furthermore, effective collaboration in environments with limited communication is a critical challenge.

[0361] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0362] In this invention, the server includes means for automatically activating a peripheral computing device when it detects environmental changes exceeding a specific threshold, which are equipped with a vibration-detecting sensor; means for collecting and analyzing voice and detecting voice information that is estimated to require assistance; and means for analyzing the user's voice and determining their emotional state. This enables the optimization of urgency based on the user's emotional state and the rapid adjustment of assistance signals, thereby providing effective support.

[0363] A "vibration sensor" is a device designed to detect physical vibrations and environmental changes, and can detect movements or vibrations that exceed a specific threshold.

[0364] "Peripheral computing devices" refer to computing resources used for data processing near user terminals, enabling rapid data analysis and processing.

[0365] "Means of collecting audio" refers to the process of acquiring the voice emitted by a user using an audio device such as a microphone and treating it as data.

[0366] "Voice information that suggests the need for support" refers to information analyzed from voice data emitted by the user, which may indicate urgency or the need for support.

[0367] "Means for analyzing voice and determining emotional state" refers to a function that analyzes voice data, evaluates its content and tone, and identifies the user's emotional state.

[0368] "Urgency optimization" is the process of assessing the urgency of the need for assistance and rescue based on collected data, and setting appropriate priorities.

[0369] "Adjusting support signals" refers to customizing signals to communicate the requested support and its priority based on the user's situation, and to relevant organizations and other users.

[0370] This invention relates to a system for providing rapid and appropriate support during disasters, taking into account the user's emotional state. Specifically, it is implemented within a mobile device, collects the user's voice data during a disaster, and analyzes it using edge AI.

[0371] The device incorporates a highly sensitive vibration sensor, enabling it to detect vibrations from natural disasters. When a specific threshold is exceeded, peripheral computing devices are activated to quickly collect audio data on-site. The audio data is acquired through a built-in microphone and then analyzed in real time by edge AI.

[0372] The edge AI incorporates an emotion engine that precisely analyzes the tone and content of voice to determine the user's emotional state. This analysis process utilizes voice analysis software and machine learning algorithms. Based on the user's situation, it optimizes the urgency level and sends a distress signal as needed.

[0373] For example, if a user urgently says "Please help me," the system will determine it to be a high-priority emergency and immediately request assistance. Additionally, as psychological support, it will play a voice message encouraging the user to relax, thus helping to stabilize their mental state.

[0374] To share information with other users and devices, terminals utilize short-range wireless communication technology. This enables data exchange with other terminals even in limited communication environments, promoting cooperation within a region.

[0375] Furthermore, messages based on emotional states can be sent to pre-registered contacts. This process allows recipients to understand the user's true emotional state and take appropriate action.

[0376] An example of a prompt message would be, "Explain the procedure for analyzing the user's voice during an earthquake and providing optimal psychological support based on their emotional state." This invention enables comprehensive and efficient user support during disasters.

[0377] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0378] Step 1:

[0379] The device detects environmental vibrations. A built-in vibration sensor monitors ambient fluctuations in real time, and when vibrations exceeding a specific threshold are detected, the data is input to the edge AI. Based on this input, the edge AI generates and outputs a signal to activate peripheral computing devices.

[0380] Step 2:

[0381] The device collects voice data. When a disaster is detected, the device automatically activates the microphone and collects the user's voice. The collected voice data becomes input for voice analysis and is sent to the next processing stage. The device temporarily stores the voice data in a buffer.

[0382] Step 3:

[0383] Edge AI analyzes the voice data. Voice analysis software uses machine learning algorithms to analyze the input data, and the emotion engine identifies the user's emotional state. Here, it evaluates the tone of voice and the content of phrases, and outputs emotional states such as anxiety and fear. The analysis results become the main input for the next step.

[0384] Step 4:

[0385] The device determines the urgency and adjusts the rescue signal. Based on the analysis results of the emotion engine, the edge AI calculates the urgency. If a high urgency is indicated, the device inputs this into its external communication function and sends an enhanced rescue signal as quickly as possible. The signal adjusted here includes priority information and the user's current location.

[0386] Step 5:

[0387] The server generates psychological support content. Based on the emotional state, a generation AI model designs and outputs an audio message. The output message includes instructions to promote relaxation and words of encouragement, which are sent to the user's device and played back as audio.

[0388] Step 6:

[0389] The device shares information with other devices using short-range wireless communication. Using locally available short-range wireless technology, the device transmits emotional state and location information to other devices in the vicinity. This facilitates data exchange with other users in the same area and enables the establishment of collaborative systems.

[0390] Step 7:

[0391] The device sends its emotional state to its contacts. A message containing the user's emotional state is sent to pre-registered contacts. The message is delivered via the network, allowing recipients to understand the user's situation accurately and design their response accordingly.

[0392] (Application Example 2)

[0393] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0394] In recent years, natural disasters have become more frequent, requiring swift and accurate evacuation during emergencies. However, the confusion and panic during disasters often hinder appropriate evacuation actions, and there is a lack of support tailored to users' emotional states. Therefore, there is a need for technology that can analyze users' emotional states in real time during disasters and support appropriate evacuation actions.

[0395] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0396] In this invention, the server includes a sensor device for detecting vibrations and means for automatically activating an edge computing unit when it detects seismic motion exceeding a specific threshold; means for collecting and analyzing voice data and detecting voice information that is estimated to indicate the need for rescue; and means for analyzing the user's emotional state and optimizing evacuation movement. This enables the provision of psychological support tailored to the user's emotional state during a disaster, allowing for optimal evacuation actions.

[0397] A "vibration-detecting sensor device" is a device that detects physical vibrations and processes that information as an electrical signal.

[0398] "Means for activating the edge computing unit" refers to methods for automatically activating edge computing functions for processing data within a device.

[0399] "Means for collecting and analyzing audio data to detect audio information that suggests the need for rescue" refers to technology that identifies situations requiring rescue by recording and analyzing audio.

[0400] "Means of requesting assistance from external communication networks or people nearby" refers to means of seeking help from external networks or people in the vicinity in an emergency.

[0401] "Methods for providing psychological support and first aid instructions via audio" refer to methods for providing users with a sense of security and guiding them through simple first aid procedures via audio.

[0402] "Means of sharing a user's location information with other users or devices" refers to technologies that allow a user to share their current location with other devices or individuals through communication.

[0403] "Methods for analyzing emotional states and optimizing evacuation routes" refers to technology that instructs users to evacuate via the most optimal route and method based on the results of analyzing their emotions.

[0404] "Means for displaying relaxation content within an autonomous mobile vehicle" refers to a function for displaying content intended to alleviate user tension inside an autonomously operating mobile vehicle.

[0405] The system implementing this invention operates with multiple functions working in coordination to ensure user safety during disasters. As basic hardware, it is equipped with a vibration-detecting sensor device that can detect physical vibrations such as earthquakes. When vibrations exceeding a certain threshold are detected, the edge computing unit automatically activates.

[0406] The server uses speech recognition technology and an emotion analysis engine to collect and analyze the user's voice data. This analysis allows the server to determine the user's emotional state in real time and, if necessary, initiate a rescue request. For example, if fear or panic is detected, an emergency signal is sent via an external communication network to prompt a more immediate response.

[0407] The device utilizes voice guidance to provide psychological support to the user. This allows for the real-time delivery of relaxing voice guides and encouraging messages to help the user feel more at ease. For example, when a user is using an autonomous vehicle (e.g., a self-driving car) during an evacuation, relaxation content can be displayed inside the vehicle.

[0408] Furthermore, short-range wireless communication technology allows for the exchange of information with other devices, strengthening cooperation within a region. This makes it easier for users to form secure networks in cooperation with other users and devices.

[0409] As a concrete example, consider a scenario where a user experiences an earthquake and their smartphone analyzes their emotions. In this case, the smartphone uses a "generative AI model" to determine their emotional state and provide optimal route guidance to a safe evacuation site. Furthermore, when boarding a vehicle, a video designed to help the user regain their composure is played on the in-car screen.

[0410] Examples of prompts include the following:

[0411] "In the event of a disaster, how can we provide rapid support to users who are in a state of panic?"

[0412] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0413] Step 1:

[0414] The device detects physical vibrations using a vibration sensor. Vibration data is obtained from the sensor as input, and the edge computing unit is activated when the vibration exceeds a certain threshold. A signal is generated as output that activates edge computing.

[0415] Step 2:

[0416] The server collects audio data and converts it into text data using speech recognition technology. The user's voice is sent to the server as input, and text data is generated as output. At this stage, the collected audio data is ready for analysis.

[0417] Step 3:

[0418] The server inputs text data into an emotion analysis engine to determine the user's emotional state. Text data is passed to the emotion analysis engine as input, and the emotional state (e.g., normal, panic, fear, etc.) is determined as output. A generative AI model is used for emotion determination.

[0419] Step 4:

[0420] Based on the emotion analysis results, the server determines the priority of the rescue request and, if necessary, sends an emergency signal via the external communication network. Emotional state data is provided as input, and a rescue request signal is generated as output. High priority requests are immediately notified to the communication network.

[0421] Step 5:

[0422] The device automatically generates and delivers psychological support messages via voice based on the user's emotional state. It generates prompts based on emotional state data as input, and plays voice guidance to the user as output. For example, if the user is in a panic state, a voice guide encouraging relaxation will play.

[0423] Step 6:

[0424] The autonomous mobile vehicle follows instructions from a server, determines an evacuation route based on the user's emotional state, and guides the user safely. Evacuation route data and the user's current location information are used as input, and the optimal route guidance is reflected in the vehicle's navigation system as output.

[0425] Step 7:

[0426] Relaxation content is played inside the vehicle. The user's emotional state and related content information are provided to the in-car display system as input, and relaxing video and audio content is presented as output. The user can view the content and experience stress reduction.

[0427] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0428] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0429] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0430] [Third Embodiment]

[0431] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0432] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0433] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0434] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0435] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0436] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0437] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0438] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0439] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0440] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0441] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0442] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0443] This invention is a system implemented within a smart device to provide rapid and effective support during disasters. The system aims to ensure the safety and peace of mind of users in the event of disasters such as earthquakes. The program's functions and specific examples are described below in natural language.

[0444] Sensor detection and edge AI activation

[0445] The device is equipped with a sensor that detects earthquake vibrations, and when the sensor detects earthquake vibrations exceeding a certain threshold, the edge AI is automatically activated. Because this edge AI processes within the device, it can operate independently of network communication conditions.

[0446] Collection of audio data and rescue request

[0447] The device constantly monitors the user's voice and detects unusual sounds or voice information indicating a need for assistance. For example, if a specific keyword such as "help" is detected, an appropriate rescue request is sent. If a communication environment is available, the rescue request is automatically sent to the server. Even if communication is impossible, the device will request assistance from those around it via voice or short-range wireless communication.

[0448] Psychological support and first aid provision

[0449] Edge AI provides users with voice-based psychological support and first-aid guidance tailored to their current situation. For example, it might give specific instructions such as "Try applying pressure to stop the bleeding" to a user who is bleeding, encouraging appropriate action on the spot and ensuring safety.

[0450] Location information sharing and regional collaboration

[0451] By utilizing short-range wireless communication technology, the terminal will connect with other nearby devices to share the location and status of disaster victims. This will create an environment where users can support each other and promote mutual assistance.

[0452] Last message sent

[0453] The device also includes a feature that allows users to send pre-set messages to family and friends when a network connection is available. This helps users ensure their safety and allows them to receive emotional support from their family.

[0454] As a concrete example, consider a situation where a user is trapped and unable to move during an earthquake. In this situation, the device immediately detects the user's body movement, and the edge AI is activated. It picks up the user's cry for help, automatically sends an appropriate rescue signal, and shares location information with other nearby devices. This process allows the user to receive rescue quickly.

[0455] Thus, this system provides multifaceted support for disaster victims and contributes to increasing the rescue rate.

[0456] The following describes the processing flow.

[0457] Step 1:

[0458] The device uses sensors to detect earthquake vibrations and identifies vibrations exceeding a specific threshold. If this threshold is exceeded, the device automatically activates its built-in edge AI.

[0459] Step 2:

[0460] The edge AI is activated and begins collecting audio data on the device. It uses the microphone to capture the user's voice and ambient sounds, filtering important audio information in real time.

[0461] Step 3:

[0462] The device analyzes the user's voice in real time to determine if rescue is needed. Specifically, it uses an algorithm to identify phrases such as "help" and "I'm in pain" to determine the necessity of rescue.

[0463] Step 4:

[0464] If communication is possible, the device will automatically send a distress signal to the server. This signal will include the device's current location and voice information. If communication is not possible, the device will use short-range radio to call for assistance from nearby devices.

[0465] Step 5:

[0466] Edge AI provides voice-based psychological support and first-aid instructions tailored to the user's situation. For example, if bleeding occurs, it will offer specific advice such as, "Apply pressure with a cloth to stop the bleeding."

[0467] Step 6:

[0468] The device exchanges location information with nearby devices using Bluetooth or other short-range communication technologies. This information sharing aims to promote cooperation and mutual assistance activities within the community.

[0469] Step 7:

[0470] The system selects messages that the user has registered in advance and sends them to family and trusted contacts as network conditions permit, thereby communicating the user's situation and providing emotional support.

[0471] (Example 1)

[0472] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0473] Responding quickly and effectively during disasters is difficult, and in situations where communication methods are limited, the inability to share information appropriately and request rescue is a major challenge. Furthermore, there is a lack of systems that can immediately provide psychological support and first-aid guidance in situations where these are needed. To solve these problems, a system is needed that provides real-time information detection and sharing, rapid rescue requests, and appropriate psychological and medical support.

[0474] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0475] In this invention, the server includes sensing means for detecting vibrations, means for automatically activating a distributed processing unit when motion exceeding a specific threshold is detected, means for collecting and analyzing voice information and detecting voice data that is estimated to indicate the need for assistance, and means for automatically requesting rescue from an external network or nearby individuals based on the voice data. This enables rapid information acquisition and rescue activities without communication constraints even during disasters, and makes it possible to provide multifaceted support to disaster victims.

[0476] "Means of detecting vibrations" refers to devices and technologies for detecting earthquakes and other movements, and includes, for example, acceleration sensors and gyroscopes.

[0477] "Movement exceeding a specific threshold" refers to vibrations or movements of an intensity that exceeds a pre-set standard value by the system, and indicates the occurrence of a disaster.

[0478] "Means for automatically activating a distributed processing unit" refers to technology that starts computer processing based on data detected by sensing means, without requiring human intervention.

[0479] "Means for collecting and analyzing voice information and detecting voice data that is estimated to require assistance" refers to technology that picks up the voice emitted by the user and analyzes it to identify situations in which assistance is needed.

[0480] "Means of requesting rescue from an external network or nearby individuals" refers to a function that transmits a rescue signal using the internet or short-range communication technology based on voice data and detection results.

[0481] "Means of providing psychological support and first aid guidance via audio" refers to technologies that use audio to guide disaster victims with calming words and first aid procedures.

[0482] "Means of sharing a user's location information with other users or devices" refers to a system that communicates a user's current location to other devices or users.

[0483] An environment with restricted communication refers to a situation where network connectivity is unstable or nonexistent, and efficient communication methods are required in such situations.

[0484] "Short-range wireless communication technology" refers to technologies such as Bluetooth and Zigbee that enable data exchange over short distances.

[0485] The "function of providing psychological support" refers to the function of providing support to disaster-affected users, such as offering encouragement and promoting a stable mental state.

[0486] This invention is a system designed to provide rapid and effective assistance during disasters. It is primarily implemented in smart devices and aims to ensure user safety and peace of mind during earthquakes and other disasters.

[0487] The device is equipped with vibration detection mechanisms, constantly monitoring vibrations using accelerometers and gyroscopes. When motion exceeding a threshold is detected, the device automatically activates a distributed processing unit, and the edge AI begins operation. This edge AI uses lightweight machine learning libraries such as TensorFlow Lite to perform necessary data analysis and decision-making on the device.

[0488] The device utilizes speech recognition technology to collect and analyze user voice information. For example, by using Google's speech recognition API, the device analyzes the user's speech in real time and detects keywords indicating a need for assistance, such as "help." If the device determines that the user is in a critical situation, it automatically sends a rescue request.

[0489] The server checks whether external communication is possible and, if so, sends a rescue signal over the network. If communication is restricted, the device uses short-range wireless communication technologies such as Bluetooth or Zigbee to notify other nearby devices of the rescue request. The device also provides the user with voice guidance on psychological support and first aid. This allows the user to respond quickly and appropriately at the scene.

[0490] A concrete example would be a scenario where, upon detecting an earthquake, the device senses the user's cry for help, immediately shares location information with nearby devices, and requests rescue.

[0491] An example of a prompt message would be, "Please explain how to provide rapid support using smart devices during a disaster."

[0492] Thus, the terminal provides diverse support in disaster situations, contributing to the rescue and safety of users.

[0493] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0494] Step 1:

[0495] The device continuously collects data using sensing means that detect ambient vibrations. The input is vibration data acquired from an accelerometer, and the output is a determination of vibrations exceeding a specific threshold. The real-time vibration data acquired by the device is recorded in internal memory and compared with the threshold. If the threshold is exceeded, the following processing is performed.

[0496] Step 2:

[0497] When the device detects vibrations exceeding a threshold, it activates the distributed processing unit. The input is the output of step 1, i.e., the vibration detection, and the output is the activation signal for the edge AI. The device activates the distributed processing unit, and the edge AI begins operation. At this time, a lightweight model such as TensorFlow Lite is loaded on the device, and it is ready to perform analysis.

[0498] Step 3:

[0499] The device collects the user's voice and analyzes it through speech recognition. The input is voice data from the microphone, and the output is the detection result of keywords indicating the need for assistance. The voice is analyzed in real time using a speech recognition API, and when phrases indicating the need for assistance, such as "help," are identified, the system proceeds to process the rescue request.

[0500] Step 4:

[0501] Based on the results of voice analysis, the terminal sends a rescue request to an external network via the server. The input is the keyword detection result, which is the output of step 3, and the output is the transmission of the rescue request signal. If a network connection is available, the terminal sends an emergency signal via the server. If communication is unstable, short-range wireless communication technology is used to notify other nearby devices of the request.

[0502] Step 5:

[0503] The device uses edge AI to generate voice guidance for the user, providing psychological and medical support. The input is the data processing results from steps 1 to 4, and the output is voice guidance for the user. The device immediately provides the user with specific first-aid instructions via voice through its speaker, such as "Try applying pressure to stop the bleeding."

[0504] (Application Example 1)

[0505] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0506] In modern society, it is difficult for individuals to receive prompt and effective rescue during disasters. Furthermore, there are challenges in sharing information in environments with limited communication, and in ensuring safety and security among individuals.

[0507] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0508] In this invention, the server includes a sensor device for detecting vibrations and means for automatically activating the edge computing unit when it detects seismic motion exceeding a certain threshold; means for automatically requesting rescue from an external communication network or nearby individuals when it recognizes a specific keyword; and means for sharing the user's location information with other devices using short-range wireless communication technology. This enables reliable and rapid rescue requests and information sharing even in the event of a disaster.

[0509] A "vibration detection sensor device" is a device that senses ambient vibrations and notifies the user if they exceed a certain threshold.

[0510] The "Edge Computing Department" is a computer division that processes data within the terminal to enable quick responses.

[0511] "Means for collecting and analyzing audio data" refers to technologies for recording ambient sounds and analyzing their content.

[0512] "Recognizing specific keywords" refers to the function of identifying predefined important words from audio data.

[0513] "Means of requesting rescue via an external communication network" refers to a method of requesting assistance through an external communication network in an emergency.

[0514] "Short-range wireless communication technology" refers to technology that enables data transmission between devices over short distances.

[0515] "Means of sharing user location information" refers to technologies for exchanging a user's geographical location with other devices or systems.

[0516] The system program for realizing this invention utilizes a vibration-detecting sensor device, an edge computing unit, voice data collection and analysis means, and short-range wireless communication technology, and is designed to enable users to receive rapid and effective assistance during disasters.

[0517] The devices used are smartphones and other mobile information terminals, which are equipped with hardware including accelerometers, microphones, and GPS. The software utilizes edge AI based on TensorFlow Lite, performing real-time speech recognition and anomaly detection. When the accelerometer detects shaking exceeding a specific threshold during an earthquake, the edge AI immediately initiates the necessary processing.

[0518] The edge AI continuously collects voice data and, upon recognizing "help" or other specific keywords, sends a rescue request to the surrounding area via an external communication network or short-range wireless communication. Furthermore, by sharing location information with nearby devices using short-range wireless communication technology, it facilitates communication among people in disaster areas.

[0519] The server also has a function that automatically sends messages to pre-registered contacts to provide emotional support when a stable communication environment is established.

[0520] As a concrete example, if an earthquake occurs in a park and a user is injured and cries out for help, the device will detect the shaking and the sound and immediately send out a rescue signal. This signal will be received by surrounding devices, and support will be provided quickly from there.

[0521] An example of a prompt message is, "My smartphone has an AI-powered emergency response app that automatically activates during earthquakes and accidents, recognizing cries for help and sending out rescue requests. How do you usually think about emergency preparedness?" This allows users to be more mindful of disaster preparedness on a daily basis.

[0522] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0523] Step 1:

[0524] The device constantly monitors ambient vibrations using an accelerometer. Its input is acceleration data related to seismic motion. When it detects vibrations exceeding a specific threshold, it generates a signal to activate the edge computing unit. The operation involves setting a certain threshold value, and activating the on-device AI when that value is exceeded.

[0525] Step 2:

[0526] The edge computing unit collects and analyzes audio data in real time. The input data is ambient noise, and the output is a determination of whether or not a rescue request is necessary. Specifically, it uses a generative AI model to analyze the audio and activates the rescue request process when it detects a specific keyword (e.g., "help").

[0527] Step 3:

[0528] The terminal transmits a rescue request via an external communication network or short-range wireless communication. The input is keyword detection results from the edge computing unit, and the output is the generation of an emergency signal. Short-range communication utilizes technologies such as Bluetooth and Wi-Fi Direct.

[0529] Step 4:

[0530] The server sends a message to pre-registered contacts when communication conditions are met. The input is a set of rescue request and location information, and the output is a reassuring message automatically sent to family and close friends. In practice, a regular automatic message sending schedule is set up.

[0531] Step 5:

[0532] This system allows users to share their location and status with other devices via short-range wireless communication. Inputs include an individual's location and current status metadata, while output is information synchronization between devices. Specifically, it involves forming an ad-hoc network among nearby individuals.

[0533] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0534] This invention provides a system for providing rapid and appropriate support during disasters, taking into account the user's emotional state, and is particularly implemented within mobile devices. Specifically, when a natural disaster such as an earthquake occurs, the terminal automatically activates edge AI to collect and analyze the user's voice data. At this time, it uses an emotion engine to recognize the user's emotional state and provides optimal psychological support based on that, as well as performing necessary rescue measures.

[0535] Sensor detection and the use of an emotion engine

[0536] The device uses sensors to detect earthquake vibrations, and when vibrations exceed a certain threshold are detected, the edge AI is activated. Once voice data is collected, the emotion engine analyzes it and begins to determine the user's emotional state. This allows it to infer whether the user is experiencing various emotions such as anxiety, fear, or calmness.

[0537] Improvements to voice analysis and rescue requests

[0538] The device analyzes the user's voice and uses an emotion engine to optimize the content and urgency of the emergency request. For example, if the emotion engine analyzes the user's voice as indicating high levels of fear or panic, it determines that higher priority action is needed and reflects this in the rescue signal.

[0539] Psychological support and first aid provision

[0540] The edge AI analyzes the emotion engine to suggest personalized psychological support to the user. For example, in situations where calmness is needed, it provides voice guidance encouraging relaxation, while conversely, if anxiety is high, it offers messages that include encouragement.

[0541] Integration and information sharing with other devices

[0542] Using short-range wireless communication technology, terminals connect with nearby terminals to share location information, including urgent data and current emotional states. This creates a network where other users in the same area can cooperate to ensure safety.

[0543] Emotion-based messaging

[0544] Users can send pre-configured messages to family and trusted contacts, accompanied by information about their current emotional state generated by an emotion engine. This allows recipients to understand the user's actual mental state and respond appropriately.

[0545] For example, if the emotional engine detects a user in a panic state during an earthquake and determines that the user is unable to act independently, this information will be used to strengthen rescue signals and take measures to encourage prompt assistance. In this way, this system aims to enhance multifaceted support for users during disasters and improve rescue rates.

[0546] The following describes the processing flow.

[0547] Step 1:

[0548] The device uses sensors to detect earthquake vibrations, and when it detects earthquake motion exceeding a specific threshold, it automatically activates edge AI.

[0549] Step 2:

[0550] The edge AI begins collecting voice data from the user. Using a microphone, it continuously monitors the surrounding sounds and prepares to filter and analyze meaningful voice data.

[0551] Step 3:

[0552] The device uses an emotion engine to analyze collected audio data and understand the user's emotional state. For example, it can determine emotions such as fear or relief from the pitch, tempo, and specific phrases of the voice.

[0553] Step 4:

[0554] Based on the user's emotional state, the device automatically adjusts the content of the rescue request. If the emotional engine indicates a state of panic, the urgency of the rescue request is increased to encourage a quick response.

[0555] Step 5:

[0556] Based on the results of emotion recognition, the edge AI provides users with appropriate psychological support and first-aid advice via voice. It offers specific instructions such as prompting them to take deep breaths to reduce anxiety and providing first aid for minor injuries.

[0557] Step 6:

[0558] The terminal utilizes short-range wireless communication technology to share information, including the user's location and emotional state, with nearby terminals, thereby facilitating rapid cooperation within the region.

[0559] Step 7:

[0560] The device will send messages containing emotional states to important contacts pre-configured by the user via the network, allowing them to receive accurate updates on their situation and emotional support.

[0561] (Example 2)

[0562] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0563] During natural disasters such as earthquakes, it is essential to provide immediate and appropriate support that takes into account the emotional state of users. Conventional systems are insufficient in providing optimal support based on users' psychological states, making rapid rescue operations and information sharing difficult. Furthermore, effective collaboration in environments with limited communication is a critical challenge.

[0564] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0565] In this invention, the server includes means for automatically activating a peripheral computing device when it detects environmental changes exceeding a specific threshold, which are equipped with a vibration-detecting sensor; means for collecting and analyzing voice and detecting voice information that is estimated to require assistance; and means for analyzing the user's voice and determining their emotional state. This enables the optimization of urgency based on the user's emotional state and the rapid adjustment of assistance signals, thereby providing effective support.

[0566] A "vibration sensor" is a device designed to detect physical vibrations and environmental changes, and can detect movements or vibrations that exceed a specific threshold.

[0567] "Peripheral computing devices" refer to computing resources used for data processing near user terminals, enabling rapid data analysis and processing.

[0568] "Means of collecting audio" refers to the process of acquiring the voice emitted by a user using an audio device such as a microphone and treating it as data.

[0569] "Voice information that suggests the need for support" refers to information analyzed from voice data emitted by the user, which may indicate urgency or the need for support.

[0570] "Means for analyzing voice and determining emotional state" refers to a function that analyzes voice data, evaluates its content and tone, and identifies the user's emotional state.

[0571] "Urgency optimization" is the process of assessing the urgency of the need for assistance and rescue based on collected data, and setting appropriate priorities.

[0572] "Adjusting support signals" refers to customizing signals to communicate the requested support and its priority based on the user's situation, and to relevant organizations and other users.

[0573] This invention relates to a system for providing rapid and appropriate support during disasters, taking into account the user's emotional state. Specifically, it is implemented within a mobile device, collects the user's voice data during a disaster, and analyzes it using edge AI.

[0574] The device incorporates a highly sensitive vibration sensor, enabling it to detect vibrations from natural disasters. When a specific threshold is exceeded, peripheral computing devices are activated to quickly collect audio data on-site. The audio data is acquired through a built-in microphone and then analyzed in real time by edge AI.

[0575] The edge AI incorporates an emotion engine that precisely analyzes the tone and content of voice to determine the user's emotional state. This analysis process utilizes voice analysis software and machine learning algorithms. Based on the user's situation, it optimizes the urgency level and sends a distress signal as needed.

[0576] For example, if a user urgently says "Please help me," the system will determine it to be a high-priority emergency and immediately request assistance. Additionally, as psychological support, it will play a voice message encouraging the user to relax, thus helping to stabilize their mental state.

[0577] To share information with other users and devices, terminals utilize short-range wireless communication technology. This enables data exchange with other terminals even in limited communication environments, promoting cooperation within a region.

[0578] Furthermore, messages based on emotional states can be sent to pre-registered contacts. This process allows recipients to understand the user's true emotional state and take appropriate action.

[0579] An example of a prompt message would be, "Explain the procedure for analyzing the user's voice during an earthquake and providing optimal psychological support based on their emotional state." This invention enables comprehensive and efficient user support during disasters.

[0580] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0581] Step 1:

[0582] The device detects environmental vibrations. A built-in vibration sensor monitors ambient fluctuations in real time, and when vibrations exceeding a specific threshold are detected, the data is input to the edge AI. Based on this input, the edge AI generates and outputs a signal to activate peripheral computing devices.

[0583] Step 2:

[0584] The device collects voice data. When a disaster is detected, the device automatically activates the microphone and collects the user's voice. The collected voice data becomes input for voice analysis and is sent to the next processing stage. The device temporarily stores the voice data in a buffer.

[0585] Step 3:

[0586] Edge AI analyzes the voice data. Voice analysis software uses machine learning algorithms to analyze the input data, and the emotion engine identifies the user's emotional state. Here, it evaluates the tone of voice and the content of phrases, and outputs emotional states such as anxiety and fear. The analysis results become the main input for the next step.

[0587] Step 4:

[0588] The device determines the urgency and adjusts the rescue signal. Based on the analysis results of the emotion engine, the edge AI calculates the urgency. If a high urgency is indicated, the device inputs this into its external communication function and sends an enhanced rescue signal as quickly as possible. The signal adjusted here includes priority information and the user's current location.

[0589] Step 5:

[0590] The server generates psychological support content. Based on the emotional state, a generation AI model designs and outputs an audio message. The output message includes instructions to promote relaxation and words of encouragement, which are sent to the user's device and played back as audio.

[0591] Step 6:

[0592] The device shares information with other devices using short-range wireless communication. Using locally available short-range wireless technology, the device transmits emotional state and location information to other devices in the vicinity. This facilitates data exchange with other users in the same area and enables the establishment of collaborative systems.

[0593] Step 7:

[0594] The device sends its emotional state to its contacts. A message containing the user's emotional state is sent to pre-registered contacts. The message is delivered via the network, allowing recipients to understand the user's situation accurately and design their response accordingly.

[0595] (Application Example 2)

[0596] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0597] In recent years, natural disasters have become more frequent, requiring swift and accurate evacuation during emergencies. However, the confusion and panic during disasters often hinder appropriate evacuation actions, and there is a lack of support tailored to users' emotional states. Therefore, there is a need for technology that can analyze users' emotional states in real time during disasters and support appropriate evacuation actions.

[0598] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0599] In this invention, the server includes a sensor device for detecting vibrations and means for automatically activating an edge computing unit when it detects seismic motion exceeding a specific threshold; means for collecting and analyzing voice data and detecting voice information that is estimated to indicate the need for rescue; and means for analyzing the user's emotional state and optimizing evacuation movement. This enables the provision of psychological support tailored to the user's emotional state during a disaster, allowing for optimal evacuation actions.

[0600] A "vibration-detecting sensor device" is a device that detects physical vibrations and processes that information as an electrical signal.

[0601] "Means for activating the edge computing unit" refers to methods for automatically activating edge computing functions for processing data within a device.

[0602] "Means for collecting and analyzing audio data to detect audio information that suggests the need for rescue" refers to technology that identifies situations requiring rescue by recording and analyzing audio.

[0603] "Means of requesting assistance from external communication networks or people nearby" refers to means of seeking help from external networks or people in the vicinity in an emergency.

[0604] "Methods for providing psychological support and first aid instructions via audio" refer to methods for providing users with a sense of security and guiding them through simple first aid procedures via audio.

[0605] "Means of sharing a user's location information with other users or devices" refers to technologies that allow a user to share their current location with other devices or individuals through communication.

[0606] "Methods for analyzing emotional states and optimizing evacuation routes" refers to technology that instructs users to evacuate via the most optimal route and method based on the results of analyzing their emotions.

[0607] "Means for displaying relaxation content within an autonomous mobile vehicle" refers to a function for displaying content intended to alleviate user tension inside an autonomously operating mobile vehicle.

[0608] The system implementing this invention operates with multiple functions working in coordination to ensure user safety during disasters. As basic hardware, it is equipped with a vibration-detecting sensor device that can detect physical vibrations such as earthquakes. When vibrations exceeding a certain threshold are detected, the edge computing unit automatically activates.

[0609] The server uses speech recognition technology and an emotion analysis engine to collect and analyze the user's voice data. This analysis allows the server to determine the user's emotional state in real time and, if necessary, initiate a rescue request. For example, if fear or panic is detected, an emergency signal is sent via an external communication network to prompt a more immediate response.

[0610] The device utilizes voice guidance to provide psychological support to the user. This allows for the real-time delivery of relaxing voice guides and encouraging messages to help the user feel more at ease. For example, when a user is using an autonomous vehicle (e.g., a self-driving car) during an evacuation, relaxation content can be displayed inside the vehicle.

[0611] Furthermore, short-range wireless communication technology allows for the exchange of information with other devices, strengthening cooperation within a region. This makes it easier for users to form secure networks in cooperation with other users and devices.

[0612] As a concrete example, consider a scenario where a user experiences an earthquake and their smartphone analyzes their emotions. In this case, the smartphone uses a "generative AI model" to determine their emotional state and provide optimal route guidance to a safe evacuation site. Furthermore, when boarding a vehicle, a video designed to help the user regain their composure is played on the in-car screen.

[0613] Examples of prompts include the following:

[0614] "In the event of a disaster, how can we provide rapid support to users who are in a state of panic?"

[0615] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0616] Step 1:

[0617] The device detects physical vibrations using a vibration sensor. Vibration data is obtained from the sensor as input, and the edge computing unit is activated when the vibration exceeds a certain threshold. A signal is generated as output that activates edge computing.

[0618] Step 2:

[0619] The server collects audio data and converts it into text data using speech recognition technology. The user's voice is sent to the server as input, and text data is generated as output. At this stage, the collected audio data is ready for analysis.

[0620] Step 3:

[0621] The server inputs text data into an emotion analysis engine to determine the user's emotional state. Text data is passed to the emotion analysis engine as input, and the emotional state (e.g., normal, panic, fear, etc.) is determined as output. A generative AI model is used for emotion determination.

[0622] Step 4:

[0623] Based on the emotion analysis results, the server determines the priority of the rescue request and, if necessary, sends an emergency signal via the external communication network. Emotional state data is provided as input, and a rescue request signal is generated as output. High priority requests are immediately notified to the communication network.

[0624] Step 5:

[0625] The device automatically generates and delivers psychological support messages via voice based on the user's emotional state. It generates prompts based on emotional state data as input, and plays voice guidance to the user as output. For example, if the user is in a panic state, a voice guide encouraging relaxation will play.

[0626] Step 6:

[0627] The autonomous mobile vehicle follows instructions from a server, determines an evacuation route based on the user's emotional state, and guides the user safely. Evacuation route data and the user's current location information are used as input, and the optimal route guidance is reflected in the vehicle's navigation system as output.

[0628] Step 7:

[0629] Relaxation content is played inside the vehicle. The user's emotional state and related content information are provided to the in-car display system as input, and relaxing video and audio content is presented as output. The user can view the content and experience stress reduction.

[0630] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0631] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0632] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0633] [Fourth Embodiment]

[0634] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0635] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0636] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0637] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0638] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0639] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0640] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0641] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0642] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0643] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0644] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0645] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0646] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0647] This invention is a system implemented within a smart device to provide rapid and effective support during disasters. The system aims to ensure the safety and peace of mind of users in the event of disasters such as earthquakes. The program's functions and specific examples are described below in natural language.

[0648] Sensor detection and edge AI activation

[0649] The device is equipped with a sensor that detects earthquake vibrations, and when the sensor detects earthquake vibrations exceeding a certain threshold, the edge AI is automatically activated. Because this edge AI processes within the device, it can operate independently of network communication conditions.

[0650] Collection of audio data and rescue request

[0651] The device constantly monitors the user's voice and detects unusual sounds or voice information indicating a need for assistance. For example, if a specific keyword such as "help" is detected, an appropriate rescue request is sent. If a communication environment is available, the rescue request is automatically sent to the server. Even if communication is impossible, the device will request assistance from those around it via voice or short-range wireless communication.

[0652] Psychological support and first aid provision

[0653] Edge AI provides users with voice-based psychological support and first-aid guidance tailored to their current situation. For example, it might give specific instructions such as "Try applying pressure to stop the bleeding" to a user who is bleeding, encouraging appropriate action on the spot and ensuring safety.

[0654] Location information sharing and regional collaboration

[0655] By utilizing short-range wireless communication technology, the terminal will connect with other nearby devices to share the location and status of disaster victims. This will create an environment where users can support each other and promote mutual assistance.

[0656] Last message sent

[0657] The device also includes a feature that allows users to send pre-set messages to family and friends when a network connection is available. This helps users ensure their safety and allows them to receive emotional support from their family.

[0658] As a concrete example, consider a situation where a user is trapped and unable to move during an earthquake. In this situation, the device immediately detects the user's body movement, and the edge AI is activated. It picks up the user's cry for help, automatically sends an appropriate rescue signal, and shares location information with other nearby devices. This process allows the user to receive rescue quickly.

[0659] Thus, this system provides multifaceted support for disaster victims and contributes to increasing the rescue rate.

[0660] The following describes the processing flow.

[0661] Step 1:

[0662] The device uses sensors to detect earthquake vibrations and identifies vibrations exceeding a specific threshold. If this threshold is exceeded, the device automatically activates its built-in edge AI.

[0663] Step 2:

[0664] The edge AI is activated and begins collecting audio data on the device. It uses the microphone to capture the user's voice and ambient sounds, filtering important audio information in real time.

[0665] Step 3:

[0666] The device analyzes the user's voice in real time to determine if rescue is needed. Specifically, it uses an algorithm to identify phrases such as "help" and "I'm in pain" to determine the necessity of rescue.

[0667] Step 4:

[0668] If communication is possible, the device will automatically send a distress signal to the server. This signal will include the device's current location and voice information. If communication is not possible, the device will use short-range radio to call for assistance from nearby devices.

[0669] Step 5:

[0670] Edge AI provides voice-based psychological support and first-aid instructions tailored to the user's situation. For example, if bleeding occurs, it will offer specific advice such as, "Apply pressure with a cloth to stop the bleeding."

[0671] Step 6:

[0672] The device exchanges location information with nearby devices using Bluetooth or other short-range communication technologies. This information sharing aims to promote cooperation and mutual assistance activities within the community.

[0673] Step 7:

[0674] The system selects messages that the user has registered in advance and sends them to family and trusted contacts as network conditions permit, thereby communicating the user's situation and providing emotional support.

[0675] (Example 1)

[0676] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0677] Responding quickly and effectively during disasters is difficult, and in situations where communication methods are limited, the inability to share information appropriately and request rescue is a major challenge. Furthermore, there is a lack of systems that can immediately provide psychological support and first-aid guidance in situations where these are needed. To solve these problems, a system is needed that provides real-time information detection and sharing, rapid rescue requests, and appropriate psychological and medical support.

[0678] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0679] In this invention, the server includes sensing means for detecting vibrations, means for automatically activating a distributed processing unit when motion exceeding a specific threshold is detected, means for collecting and analyzing voice information and detecting voice data that is estimated to indicate the need for assistance, and means for automatically requesting rescue from an external network or nearby individuals based on the voice data. This enables rapid information acquisition and rescue activities without communication constraints even during disasters, and makes it possible to provide multifaceted support to disaster victims.

[0680] "Means of detecting vibrations" refers to devices and technologies for detecting earthquakes and other movements, and includes, for example, acceleration sensors and gyroscopes.

[0681] "Movement exceeding a specific threshold" refers to vibrations or movements of an intensity that exceeds a pre-set standard value by the system, and indicates the occurrence of a disaster.

[0682] "Means for automatically activating a distributed processing unit" refers to technology that starts computer processing based on data detected by sensing means, without requiring human intervention.

[0683] "Means for collecting and analyzing voice information and detecting voice data that is estimated to require assistance" refers to technology that picks up the voice emitted by the user and analyzes it to identify situations in which assistance is needed.

[0684] "Means of requesting rescue from an external network or nearby individuals" refers to a function that transmits a rescue signal using the internet or short-range communication technology based on voice data and detection results.

[0685] "Means of providing psychological support and first aid guidance via audio" refers to technologies that use audio to guide disaster victims with calming words and first aid procedures.

[0686] "Means of sharing a user's location information with other users or devices" refers to a system that communicates a user's current location to other devices or users.

[0687] An environment with restricted communication refers to a situation where network connectivity is unstable or nonexistent, and efficient communication methods are required in such situations.

[0688] "Short-range wireless communication technology" refers to technologies such as Bluetooth and Zigbee that enable data exchange over short distances.

[0689] The "function of providing psychological support" refers to the function of providing support to disaster-affected users, such as offering encouragement and promoting a stable mental state.

[0690] This invention is a system designed to provide rapid and effective assistance during disasters. It is primarily implemented in smart devices and aims to ensure user safety and peace of mind during earthquakes and other disasters.

[0691] The device is equipped with vibration detection mechanisms, constantly monitoring vibrations using accelerometers and gyroscopes. When motion exceeding a threshold is detected, the device automatically activates a distributed processing unit, and the edge AI begins operation. This edge AI uses lightweight machine learning libraries such as TensorFlow Lite to perform necessary data analysis and decision-making on the device.

[0692] The device utilizes speech recognition technology to collect and analyze user voice information. For example, by using Google's speech recognition API, the device analyzes the user's speech in real time and detects keywords indicating a need for assistance, such as "help." If the device determines that the user is in a critical situation, it automatically sends a rescue request.

[0693] The server checks whether external communication is possible and, if so, sends a rescue signal over the network. If communication is restricted, the device uses short-range wireless communication technologies such as Bluetooth or Zigbee to notify other nearby devices of the rescue request. The device also provides the user with voice guidance on psychological support and first aid. This allows the user to respond quickly and appropriately at the scene.

[0694] A concrete example would be a scenario where, upon detecting an earthquake, the device senses the user's cry for help, immediately shares location information with nearby devices, and requests rescue.

[0695] An example of a prompt message would be, "Please explain how to provide rapid support using smart devices during a disaster."

[0696] Thus, the terminal provides diverse support in disaster situations, contributing to the rescue and safety of users.

[0697] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0698] Step 1:

[0699] The device continuously collects data using sensing means that detect ambient vibrations. The input is vibration data acquired from an accelerometer, and the output is a determination of vibrations exceeding a specific threshold. The real-time vibration data acquired by the device is recorded in internal memory and compared with the threshold. If the threshold is exceeded, the following processing is performed.

[0700] Step 2:

[0701] When the device detects vibrations exceeding a threshold, it activates the distributed processing unit. The input is the output of step 1, i.e., the vibration detection, and the output is the activation signal for the edge AI. The device activates the distributed processing unit, and the edge AI begins operation. At this time, a lightweight model such as TensorFlow Lite is loaded on the device, and it is ready to perform analysis.

[0702] Step 3:

[0703] The device collects the user's voice and analyzes it through speech recognition. The input is voice data from the microphone, and the output is the detection result of keywords indicating the need for assistance. The voice is analyzed in real time using a speech recognition API, and when phrases indicating the need for assistance, such as "help," are identified, the system proceeds to process the rescue request.

[0704] Step 4:

[0705] Based on the results of voice analysis, the terminal sends a rescue request to an external network via the server. The input is the keyword detection result, which is the output of step 3, and the output is the transmission of the rescue request signal. If a network connection is available, the terminal sends an emergency signal via the server. If communication is unstable, short-range wireless communication technology is used to notify other nearby devices of the request.

[0706] Step 5:

[0707] The device uses edge AI to generate voice guidance for the user, providing psychological and medical support. The input is the data processing results from steps 1 to 4, and the output is voice guidance for the user. The device immediately provides the user with specific first-aid instructions via voice through its speaker, such as "Try applying pressure to stop the bleeding."

[0708] (Application Example 1)

[0709] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0710] In modern society, it is difficult for individuals to receive prompt and effective rescue during disasters. Furthermore, there are challenges in sharing information in environments with limited communication, and in ensuring safety and security among individuals.

[0711] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0712] In this invention, the server includes a sensor device for detecting vibrations and means for automatically activating the edge computing unit when it detects seismic motion exceeding a certain threshold; means for automatically requesting rescue from an external communication network or nearby individuals when it recognizes a specific keyword; and means for sharing the user's location information with other devices using short-range wireless communication technology. This enables reliable and rapid rescue requests and information sharing even in the event of a disaster.

[0713] A "vibration detection sensor device" is a device that senses ambient vibrations and notifies the user if they exceed a certain threshold.

[0714] The "Edge Computing Department" is a computer division that processes data within the terminal to enable quick responses.

[0715] "Means for collecting and analyzing audio data" refers to technologies for recording ambient sounds and analyzing their content.

[0716] "Recognizing specific keywords" refers to the function of identifying predefined important words from audio data.

[0717] "Means of requesting rescue via an external communication network" refers to a method of requesting assistance through an external communication network in an emergency.

[0718] "Short-range wireless communication technology" refers to technology that enables data transmission between devices over short distances.

[0719] "Means of sharing user location information" refers to technologies for exchanging a user's geographical location with other devices or systems.

[0720] The system program for realizing this invention utilizes a vibration-detecting sensor device, an edge computing unit, voice data collection and analysis means, and short-range wireless communication technology, and is designed to enable users to receive rapid and effective assistance during disasters.

[0721] The devices used are smartphones and other mobile information terminals, which are equipped with hardware including accelerometers, microphones, and GPS. The software utilizes edge AI based on TensorFlow Lite, performing real-time speech recognition and anomaly detection. When the accelerometer detects shaking exceeding a specific threshold during an earthquake, the edge AI immediately initiates the necessary processing.

[0722] The edge AI continuously collects voice data and, upon recognizing "help" or other specific keywords, sends a rescue request to the surrounding area via an external communication network or short-range wireless communication. Furthermore, by sharing location information with nearby devices using short-range wireless communication technology, it facilitates communication among people in disaster areas.

[0723] The server also has a function that automatically sends messages to pre-registered contacts to provide emotional support when a stable communication environment is established.

[0724] As a concrete example, if an earthquake occurs in a park and a user is injured and cries out for help, the device will detect the shaking and the sound and immediately send out a rescue signal. This signal will be received by surrounding devices, and support will be provided quickly from there.

[0725] An example of a prompt message is, "My smartphone has an AI-powered emergency response app that automatically activates during earthquakes and accidents, recognizing cries for help and sending out rescue requests. How do you usually think about emergency preparedness?" This allows users to be more mindful of disaster preparedness on a daily basis.

[0726] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0727] Step 1:

[0728] The device constantly monitors ambient vibrations using an accelerometer. Its input is acceleration data related to seismic motion. When it detects vibrations exceeding a specific threshold, it generates a signal to activate the edge computing unit. The operation involves setting a certain threshold value, and activating the on-device AI when that value is exceeded.

[0729] Step 2:

[0730] The edge computing unit collects and analyzes audio data in real time. The input data is ambient noise, and the output is a determination of whether or not a rescue request is necessary. Specifically, it uses a generative AI model to analyze the audio and activates the rescue request process when it detects a specific keyword (e.g., "help").

[0731] Step 3:

[0732] The terminal transmits a rescue request via an external communication network or short-range wireless communication. The input is keyword detection results from the edge computing unit, and the output is the generation of an emergency signal. Short-range communication utilizes technologies such as Bluetooth and Wi-Fi Direct.

[0733] Step 4:

[0734] The server sends a message to pre-registered contacts when communication conditions are met. The input is a set of rescue request and location information, and the output is a reassuring message automatically sent to family and close friends. In practice, a regular automatic message sending schedule is set up.

[0735] Step 5:

[0736] This system allows users to share their location and status with other devices via short-range wireless communication. Inputs include an individual's location and current status metadata, while output is information synchronization between devices. Specifically, it involves forming an ad-hoc network among nearby individuals.

[0737] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0738] This invention provides a system for providing rapid and appropriate support during disasters, taking into account the user's emotional state, and is particularly implemented within mobile devices. Specifically, when a natural disaster such as an earthquake occurs, the terminal automatically activates edge AI to collect and analyze the user's voice data. At this time, it uses an emotion engine to recognize the user's emotional state and provides optimal psychological support based on that, as well as performing necessary rescue measures.

[0739] Sensor detection and the use of an emotion engine

[0740] The device uses sensors to detect earthquake vibrations, and when vibrations exceed a certain threshold are detected, the edge AI is activated. Once voice data is collected, the emotion engine analyzes it and begins to determine the user's emotional state. This allows it to infer whether the user is experiencing various emotions such as anxiety, fear, or calmness.

[0741] Improvements to voice analysis and rescue requests

[0742] The device analyzes the user's voice and uses an emotion engine to optimize the content and urgency of the emergency request. For example, if the emotion engine analyzes the user's voice as indicating high levels of fear or panic, it determines that higher priority action is needed and reflects this in the rescue signal.

[0743] Psychological support and first aid provision

[0744] The edge AI analyzes the emotion engine to suggest personalized psychological support to the user. For example, in situations where calmness is needed, it provides voice guidance encouraging relaxation, while conversely, if anxiety is high, it offers messages that include encouragement.

[0745] Integration and information sharing with other devices

[0746] Using short-range wireless communication technology, terminals connect with nearby terminals to share location information, including urgent data and current emotional states. This creates a network where other users in the same area can cooperate to ensure safety.

[0747] Emotion-based messaging

[0748] Users can send pre-configured messages to family and trusted contacts, accompanied by information about their current emotional state generated by an emotion engine. This allows recipients to understand the user's actual mental state and respond appropriately.

[0749] For example, if the emotional engine detects a user in a panic state during an earthquake and determines that the user is unable to act independently, this information will be used to strengthen rescue signals and take measures to encourage prompt assistance. In this way, this system aims to enhance multifaceted support for users during disasters and improve rescue rates.

[0750] The following describes the processing flow.

[0751] Step 1:

[0752] The device uses sensors to detect earthquake vibrations, and when it detects earthquake motion exceeding a specific threshold, it automatically activates edge AI.

[0753] Step 2:

[0754] The edge AI begins collecting voice data from the user. Using a microphone, it continuously monitors the surrounding sounds and prepares to filter and analyze meaningful voice data.

[0755] Step 3:

[0756] The device uses an emotion engine to analyze collected audio data and understand the user's emotional state. For example, it can determine emotions such as fear or relief from the pitch, tempo, and specific phrases of the voice.

[0757] Step 4:

[0758] Based on the user's emotional state, the device automatically adjusts the content of the rescue request. If the emotional engine indicates a state of panic, the urgency of the rescue request is increased to encourage a quick response.

[0759] Step 5:

[0760] Based on the results of emotion recognition, the edge AI provides users with appropriate psychological support and first-aid advice via voice. It offers specific instructions such as prompting them to take deep breaths to reduce anxiety and providing first aid for minor injuries.

[0761] Step 6:

[0762] The terminal utilizes short-range wireless communication technology to share information, including the user's location and emotional state, with nearby terminals, thereby facilitating rapid cooperation within the region.

[0763] Step 7:

[0764] The device will send messages containing emotional states to important contacts pre-configured by the user via the network, allowing them to receive accurate updates on their situation and emotional support.

[0765] (Example 2)

[0766] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0767] During natural disasters such as earthquakes, it is essential to provide immediate and appropriate support that takes into account the emotional state of users. Conventional systems are insufficient in providing optimal support based on users' psychological states, making rapid rescue operations and information sharing difficult. Furthermore, effective collaboration in environments with limited communication is a critical challenge.

[0768] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0769] In this invention, the server includes means for automatically activating a peripheral computing device when it detects environmental changes exceeding a specific threshold, which are equipped with a vibration-detecting sensor; means for collecting and analyzing voice and detecting voice information that is estimated to require assistance; and means for analyzing the user's voice and determining their emotional state. This enables the optimization of urgency based on the user's emotional state and the rapid adjustment of assistance signals, thereby providing effective support.

[0770] A "vibration sensor" is a device designed to detect physical vibrations and environmental changes, and can detect movements or vibrations that exceed a specific threshold.

[0771] "Peripheral computing devices" refer to computing resources used for data processing near user terminals, enabling rapid data analysis and processing.

[0772] "Means of collecting audio" refers to the process of acquiring the voice emitted by a user using an audio device such as a microphone and treating it as data.

[0773] "Voice information that suggests the need for support" refers to information analyzed from voice data emitted by the user, which may indicate urgency or the need for support.

[0774] "Means for analyzing voice and determining emotional state" refers to a function that analyzes voice data, evaluates its content and tone, and identifies the user's emotional state.

[0775] "Urgency optimization" is the process of assessing the urgency of the need for assistance and rescue based on collected data, and setting appropriate priorities.

[0776] "Adjusting support signals" refers to customizing signals to communicate the requested support and its priority based on the user's situation, and to relevant organizations and other users.

[0777] This invention relates to a system for providing rapid and appropriate support during disasters, taking into account the user's emotional state. Specifically, it is implemented within a mobile device, collects the user's voice data during a disaster, and analyzes it using edge AI.

[0778] The device incorporates a highly sensitive vibration sensor, enabling it to detect vibrations from natural disasters. When a specific threshold is exceeded, peripheral computing devices are activated to quickly collect audio data on-site. The audio data is acquired through a built-in microphone and then analyzed in real time by edge AI.

[0779] The edge AI incorporates an emotion engine that precisely analyzes the tone and content of voice to determine the user's emotional state. This analysis process utilizes voice analysis software and machine learning algorithms. Based on the user's situation, it optimizes the urgency level and sends a distress signal as needed.

[0780] For example, if a user urgently says "Please help me," the system will determine it to be a high-priority emergency and immediately request assistance. Additionally, as psychological support, it will play a voice message encouraging the user to relax, thus helping to stabilize their mental state.

[0781] To share information with other users and devices, terminals utilize short-range wireless communication technology. This enables data exchange with other terminals even in limited communication environments, promoting cooperation within a region.

[0782] Furthermore, messages based on emotional states can be sent to pre-registered contacts. This process allows recipients to understand the user's true emotional state and take appropriate action.

[0783] An example of a prompt message would be, "Explain the procedure for analyzing the user's voice during an earthquake and providing optimal psychological support based on their emotional state." This invention enables comprehensive and efficient user support during disasters.

[0784] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0785] Step 1:

[0786] The device detects environmental vibrations. A built-in vibration sensor monitors ambient fluctuations in real time, and when vibrations exceeding a specific threshold are detected, the data is input to the edge AI. Based on this input, the edge AI generates and outputs a signal to activate peripheral computing devices.

[0787] Step 2:

[0788] The device collects voice data. When a disaster is detected, the device automatically activates the microphone and collects the user's voice. The collected voice data becomes input for voice analysis and is sent to the next processing stage. The device temporarily stores the voice data in a buffer.

[0789] Step 3:

[0790] Edge AI analyzes the voice data. Voice analysis software uses machine learning algorithms to analyze the input data, and the emotion engine identifies the user's emotional state. Here, it evaluates the tone of voice and the content of phrases, and outputs emotional states such as anxiety and fear. The analysis results become the main input for the next step.

[0791] Step 4:

[0792] The device determines the urgency and adjusts the rescue signal. Based on the analysis results of the emotion engine, the edge AI calculates the urgency. If a high urgency is indicated, the device inputs this into its external communication function and sends an enhanced rescue signal as quickly as possible. The signal adjusted here includes priority information and the user's current location.

[0793] Step 5:

[0794] The server generates psychological support content. Based on the emotional state, a generation AI model designs and outputs an audio message. The output message includes instructions to promote relaxation and words of encouragement, which are sent to the user's device and played back as audio.

[0795] Step 6:

[0796] The device shares information with other devices using short-range wireless communication. Using locally available short-range wireless technology, the device transmits emotional state and location information to other devices in the vicinity. This facilitates data exchange with other users in the same area and enables the establishment of collaborative systems.

[0797] Step 7:

[0798] The device sends its emotional state to its contacts. A message containing the user's emotional state is sent to pre-registered contacts. The message is delivered via the network, allowing recipients to understand the user's situation accurately and design their response accordingly.

[0799] (Application Example 2)

[0800] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0801] In recent years, natural disasters have become more frequent, requiring swift and accurate evacuation during emergencies. However, the confusion and panic during disasters often hinder appropriate evacuation actions, and there is a lack of support tailored to users' emotional states. Therefore, there is a need for technology that can analyze users' emotional states in real time during disasters and support appropriate evacuation actions.

[0802] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0803] In this invention, the server includes a sensor device for detecting vibrations and means for automatically activating an edge computing unit when it detects seismic motion exceeding a specific threshold; means for collecting and analyzing voice data and detecting voice information that is estimated to indicate the need for rescue; and means for analyzing the user's emotional state and optimizing evacuation movement. This enables the provision of psychological support tailored to the user's emotional state during a disaster, allowing for optimal evacuation actions.

[0804] A "vibration-detecting sensor device" is a device that detects physical vibrations and processes that information as an electrical signal.

[0805] "Means for activating the edge computing unit" refers to methods for automatically activating edge computing functions for processing data within a device.

[0806] "Means for collecting and analyzing audio data to detect audio information that suggests the need for rescue" refers to technology that identifies situations requiring rescue by recording and analyzing audio.

[0807] "Means of requesting assistance from external communication networks or people nearby" refers to means of seeking help from external networks or people in the vicinity in an emergency.

[0808] "Methods for providing psychological support and first aid instructions via audio" refer to methods for providing users with a sense of security and guiding them through simple first aid procedures via audio.

[0809] "Means of sharing a user's location information with other users or devices" refers to technologies that allow a user to share their current location with other devices or individuals through communication.

[0810] "Methods for analyzing emotional states and optimizing evacuation routes" refers to technology that instructs users to evacuate via the most optimal route and method based on the results of analyzing their emotions.

[0811] "Means for displaying relaxation content within an autonomous mobile vehicle" refers to a function for displaying content intended to alleviate user tension inside an autonomously operating mobile vehicle.

[0812] The system implementing this invention operates with multiple functions working in coordination to ensure user safety during disasters. As basic hardware, it is equipped with a vibration-detecting sensor device that can detect physical vibrations such as earthquakes. When vibrations exceeding a certain threshold are detected, the edge computing unit automatically activates.

[0813] The server uses speech recognition technology and an emotion analysis engine to collect and analyze the user's voice data. This analysis allows the server to determine the user's emotional state in real time and, if necessary, initiate a rescue request. For example, if fear or panic is detected, an emergency signal is sent via an external communication network to prompt a more immediate response.

[0814] The device utilizes voice guidance to provide psychological support to the user. This allows for the real-time delivery of relaxing voice guides and encouraging messages to help the user feel more at ease. For example, when a user is using an autonomous vehicle (e.g., a self-driving car) during an evacuation, relaxation content can be displayed inside the vehicle.

[0815] Furthermore, short-range wireless communication technology allows for the exchange of information with other devices, strengthening cooperation within a region. This makes it easier for users to form secure networks in cooperation with other users and devices.

[0816] As a concrete example, consider a scenario where a user experiences an earthquake and their smartphone analyzes their emotions. In this case, the smartphone uses a "generative AI model" to determine their emotional state and provide optimal route guidance to a safe evacuation site. Furthermore, when boarding a vehicle, a video designed to help the user regain their composure is played on the in-car screen.

[0817] Examples of prompts include the following:

[0818] "In the event of a disaster, how can we provide rapid support to users who are in a state of panic?"

[0819] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0820] Step 1:

[0821] The device detects physical vibrations using a vibration sensor. Vibration data is obtained from the sensor as input, and the edge computing unit is activated when the vibration exceeds a certain threshold. A signal is generated as output that activates edge computing.

[0822] Step 2:

[0823] The server collects audio data and converts it into text data using speech recognition technology. The user's voice is sent to the server as input, and text data is generated as output. At this stage, the collected audio data is ready for analysis.

[0824] Step 3:

[0825] The server inputs text data into an emotion analysis engine to determine the user's emotional state. Text data is passed to the emotion analysis engine as input, and the emotional state (e.g., normal, panic, fear, etc.) is determined as output. A generative AI model is used for emotion determination.

[0826] Step 4:

[0827] Based on the emotion analysis results, the server determines the priority of the rescue request and, if necessary, sends an emergency signal via the external communication network. Emotional state data is provided as input, and a rescue request signal is generated as output. High priority requests are immediately notified to the communication network.

[0828] Step 5:

[0829] The device automatically generates and delivers psychological support messages via voice based on the user's emotional state. It generates prompts based on emotional state data as input, and plays voice guidance to the user as output. For example, if the user is in a panic state, a voice guide encouraging relaxation will play.

[0830] Step 6:

[0831] The autonomous mobile vehicle follows instructions from a server, determines an evacuation route based on the user's emotional state, and guides the user safely. Evacuation route data and the user's current location information are used as input, and the optimal route guidance is reflected in the vehicle's navigation system as output.

[0832] Step 7:

[0833] Relaxation content is played inside the vehicle. The user's emotional state and related content information are provided to the in-car display system as input, and relaxing video and audio content is presented as output. The user can view the content and experience stress reduction.

[0834] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0835] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0836] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0837] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0838] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0839] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0840] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0841] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0842] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0843] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0844] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0845] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0846] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0847] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0848] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0849] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0850] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0851] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0852] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0853] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0854] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0855] The following is further disclosed regarding the embodiments described above.

[0856] (Claim 1)

[0857] Equipped with a sensor device to detect vibrations,

[0858] A means for automatically activating the edge computing unit when seismic motion exceeding a specific threshold is detected,

[0859] A means for collecting and analyzing audio data to detect audio information that suggests the need for rescue,

[0860] A means of automatically requesting rescue from an external communication network or to people in the vicinity based on voice information,

[0861] A means of providing users with psychological support and first aid guidance via audio,

[0862] A means of sharing a user's location information with other users or devices.

[0863] A system that includes this.

[0864] (Claim 2)

[0865] The system according to claim 1, which has a function to exchange information with other terminals using short-range wireless communication technology and promote cooperation within a region, even in environments where communication is restricted.

[0866] (Claim 3)

[0867] The system according to claim 1, comprising a function to send messages to pre-registered contacts via a network and provide emotional support.

[0868] "Example 1"

[0869] (Claim 1)

[0870] Equipped with a sensing means for detecting vibrations,

[0871] A means for automatically activating a distributed processing unit when motion exceeding a specific threshold is detected,

[0872] A means for collecting and analyzing voice information and detecting voice data that is estimated to require assistance,

[0873] A means of automatically requesting rescue from an external network or nearby individuals based on voice data,

[0874] A means of providing users with psychological support and first aid guidance via audio,

[0875] A means of sharing a user's location information with other users or devices.

[0876] A system that includes this.

[0877] (Claim 2)

[0878] The system according to claim 1, which has a function to exchange information with other devices using short-range wireless communication technology and promote cooperation within a region, even in environments where communication is restricted.

[0879] (Claim 3)

[0880] The system according to claim 1, comprising a function to send messages to pre-registered contacts via a communication network and provide psychological support.

[0881] "Application Example 1"

[0882] (Claim 1)

[0883] Equipped with a sensor device to detect vibrations,

[0884] A means for automatically activating the edge computing unit when seismic motion exceeding a specific threshold is detected,

[0885] A means for collecting and analyzing audio data to detect audio information that suggests the need for rescue,

[0886] A means of automatically requesting rescue from an external communication network or nearby individuals when a specific keyword is recognized,

[0887] A means of providing users with audio-based psychological support and first aid guidance,

[0888] A means of sharing a user's location information with other devices using short-range wireless communication technology,

[0889] A system that includes this.

[0890] (Claim 2)

[0891] The system according to claim 1, which has a function to exchange information with other devices using short-range wireless communication technology and promote cooperation within a region, even in environments where communication is restricted.

[0892] (Claim 3)

[0893] The system according to claim 1, comprising a function to periodically send messages to pre-registered contacts via a network to provide emotional support.

[0894] "Example 2 of combining an emotion engine"

[0895] (Claim 1)

[0896] Equipped with a vibration detection sensor,

[0897] A means for automatically activating peripheral computing devices when environmental changes exceeding a specific threshold are detected,

[0898] A means for collecting and analyzing audio and detecting audio information that is estimated to require assistance,

[0899] A means of automatically requesting assistance from an external communication network or surrounding individuals based on voice information,

[0900] A means of providing users with psychological support and first aid instructions via audio,

[0901] A means of sharing the user's location with other users or devices,

[0902] A means of analyzing the user's voice and determining their emotional state,

[0903] A means of optimizing urgency and adjusting support signals based on emotional state.

[0904] A system that includes this.

[0905] (Claim 2)

[0906] The system according to claim 1, which has a function to exchange information with other devices using short-range wireless communication technology and to promote cooperation within a region.

[0907] (Claim 3)

[0908] The system according to claim 1, which has a function to send messages to pre-registered contacts via an information network and provide emotional support by attaching emotional information.

[0909] "Application example 2 when combining with an emotional engine"

[0910] (Claim 1)

[0911] Equipped with a sensor device that detects vibrations,

[0912] A means for automatically activating the edge computing unit when seismic motion exceeding a specific threshold is detected,

[0913] A means for collecting and analyzing audio data to detect audio information that suggests the need for rescue,

[0914] A means of automatically requesting rescue from an external communication network or to people in the vicinity based on voice information,

[0915] A means of providing users with psychological support and first aid guidance via audio,

[0916] A means of sharing a user's location information with other users or devices,

[0917] A means of analyzing the emotional state of users and optimizing evacuation and transportation methods,

[0918] A means of displaying relaxation content inside an autonomous mobile vehicle,

[0919] A system that includes this.

[0920] (Claim 2)

[0921] The system according to claim 1, which has a function to exchange information with other terminals using short-range wireless communication technology and promote cooperation within a region, even in environments where communication is restricted.

[0922] (Claim 3)

[0923] The system according to claim 1, comprising a function to send messages to pre-registered contacts via a network and provide emotional support. [Explanation of Symbols]

[0924] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Equipped with a sensor device to detect vibrations, A means for automatically activating the edge computing unit when seismic motion exceeding a specific threshold is detected, A means for collecting and analyzing audio data to detect audio information that suggests the need for rescue, A means of automatically requesting rescue from an external communication network or to people in the vicinity based on voice information, A means of providing users with psychological support and first aid guidance via audio, A means of sharing a user's location information with other users or devices, A system that includes this.

2. The system according to claim 1, which has a function to exchange information with other terminals using short-range wireless communication technology and promote cooperation within a region, even in environments where communication is restricted.

3. The system according to claim 1, comprising a function to send messages to pre-registered contacts via a network and provide emotional support.

Images