system

Abstract

We provide the system. [Solution] A portable computing device equipped with sensors that collect voice data, location information, and health data in real time, A generative artificial intelligence means that analyzes the aforementioned data locally and provides notification by voice or vibration based on the generated results, A federated learning method utilizing a central computing device that collaborates with external data to provide disaster information and the optimal evacuation route for users, Means for distributing updated model data from the federated learning means to the portable computing device, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, the importance of providing prompt and appropriate information during disasters has been increasing, but there is a problem that means for supporting individuals' emergency evacuation are insufficient. In conventional systems, it is difficult to analyze individuals' health conditions and location information in real time and provide an optimal evacuation route. In particular, for people with limited information collection capabilities such as the elderly and disabled, flexible and personalized support for safe evacuation is required. Against this background, there is a need for a technology that provides real-time personalized evacuation support while protecting privacy.

Means for Solving the Problems

[0005] It should be noted that there seems to be a redundant "This" in the translation of line . You may want to check the original text for accuracy. If this is a misprint in the original, the translation should be adjusted accordingly.This invention provides a generative artificial intelligence means that utilizes a portable computing device equipped with sensors to collect voice data, location information, and health data in real time, and analyzes this information locally. By providing users with optimal notifications via voice or vibration, it encourages quick and accurate action during disasters. Furthermore, by linking with external data and using federated learning technology, it analyzes overall disaster information and provides evacuation routes suitable for users through a central computing device. In addition, by distributing updated model data to each portable computing device, it provides users with the latest information at all times. This enables personalized evacuation support and health management for residents in high-risk areas and users who are unfamiliar with information gathering.

[0006] "Voice data" refers to data used to digitize and analyze a user's voice information.

[0007] "Location information" refers to latitude and longitude data obtained using GPS or other positioning technologies to indicate the user's current location.

[0008] "Health data" refers to vital signs and biometric information collected to measure the user's physical condition, specifically data such as heart rate and body temperature.

[0009] "Real-time collection" means that data is continuously and instantly acquired and immediately available when needed.

[0010] A "portable computing device" is a small, portable electronic device that has data collection and processing capabilities.

[0011] "Generative artificial intelligence means" refers to a method that utilizes artificial intelligence technology to automatically analyze collected data and generate user feedback based on the results.

[0012] "External data" refers to additional information obtained from outside the system, including disaster information and traffic information.

[0013] "Federated learning" refers to machine learning techniques that are performed in a distributed manner across individual devices, enabling overall model improvement while protecting user privacy.

[0014] A "central computing device" is a central electronic computing device that aggregates and analyzes data from the entire system.

[0015] "Distributing model data" refers to the process of updating the collected learning and analysis results by sending them to individual devices. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be described.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] The system according to the present invention consists of a server, terminals, and users, and supports the safe evacuation of users in the event of a disaster. Each terminal is equipped with sensors for acquiring voice data, location information, and health data, and this data is collected in real time. The data is analyzed by local generative artificial intelligence means, and appropriate feedback is provided according to the user's situation.

[0038] Server operation

[0039] The server uses federated learning technology to aggregate anonymized information from each terminal. Based on the collected data, it optimizes risk information and evacuation routes for the entire region. It also acquires external disaster information and traffic data in real time and combines and analyzes them to generate optimal evacuation route information for each individual. The server periodically distributes these calculation results to each terminal and updates the terminal's model data.

[0040] Terminal operation

[0041] The device performs analysis using generated AI based on model data received from the server. For example, if a user lives in an area affected by an earthquake, the device monitors the user's health based on acquired heart rate, body temperature, and location information. If the user's health deteriorates, the device immediately issues voice instructions such as "Take a deep breath" or "Move to a safe place." These instructions are linked to the optimal evacuation route based on the user's location information.

[0042] User experience

[0043] Under normal circumstances, users simply wear the device without requiring any special operation. In the event of a disaster, users receive intuitive feedback from the device and take safe actions based on the instructions. For example, when a flood warning is issued, the device will give a voice instruction to "immediately evacuate to higher ground," allowing users to quickly begin evacuation without confusion.

[0044] This system effectively supports user safety by providing personalized support that can respond immediately during disasters while protecting personal information.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] The device collects voice data, location information, and health data in real time. It continuously acquires data through sensors and stores it locally.

[0048] Step 2:

[0049] The data collected by the terminal is preprocessed in real time. This includes noise reduction and data cleaning, and conversion to a format suitable for analysis.

[0050] Step 3:

[0051] The device uses AI to analyze pre-processed data. Based on the analysis results, it evaluates the user's health status and risk level based on their current location.

[0052] Step 4:

[0053] The device provides feedback to the user based on the analysis results. This feedback is delivered via voice notifications and vibrations and includes recommended actions and evacuation route information.

[0054] Step 5:

[0055] The server performs federated learning based on anonymized data collected from each terminal. The overall model is updated to calculate the optimal evacuation route, taking into account disaster information and user-specific data.

[0056] Step 6:

[0057] The server delivers updated model data to the terminal. The terminal uses this data to prepare for the next analysis and to provide more accurate feedback.

[0058] Step 7:

[0059] Users will take necessary evacuation actions according to notifications on their devices. Based on voice notifications, they will choose appropriate routes and actions to ensure their safety.

[0060] Step 8:

[0061] Users can use voice commands as needed to provide additional information to the terminal. This allows the system to improve its feedback in response to the user's specific requests.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] During disasters, it is difficult to quickly and accurately provide each individual user with the most suitable evacuation route. Furthermore, mechanisms for monitoring individuals' health status in real time and providing appropriate instructions are not yet fully established. As a result, users may not be able to evacuate properly during emergencies, potentially increasing their health risks. Additionally, obtaining useful information while maintaining data anonymity is another challenge.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes a data collection device that collects voice information, location information, and biometric information in real time; a knowledge processing means that analyzes the information on a local device and issues instructions based on the results of the analysis; an aggregate learning means that cooperates with external information to provide disaster information and user-optimized evacuation routes; and a means that distributes an updated information model from the aggregate learning means to the data collection device. This makes it possible to provide personalized evacuation instructions and health management to each user in real time, even during a disaster.

[0067] "Audio information" refers to data that represents the characteristics of sound in digital format, and is used to understand instructions and situations based on the user's voice.

[0068] "Location information" refers to geographical coordinate data, which is used to identify a user's current location and travel route.

[0069] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate and body temperature, and is collected for health management and risk assessment.

[0070] A "data collection device" is a device that includes sensors and equipment for acquiring voice information, location information, and biometric information, and is responsible for collecting information from users.

[0071] A "local device" is a computing device located near the user that analyzes collected data and generates instructions in real time.

[0072] "Knowledge processing means" refers to artificial intelligence technology that analyzes collected data to generate optimal instructions for the user.

[0073] "External information" refers to data obtained from outside the system, such as disaster information and traffic conditions, and is used to optimize evacuation routes.

[0074] "Aggregated learning methods" refer to machine learning techniques that integrate data obtained from multiple sources to perform overall optimization and analysis.

[0075] An "information model" refers to data that includes the calculation results of optimized evacuation routes and disaster information, generated by aggregate learning methods and provided to users.

[0076] This invention relates to an advanced information system for enhancing user safety during disasters, in which the server, terminal, and user components work in coordination. The embodiments thereof are described below.

[0077] Server configuration and operation

[0078] The server includes a central computing device for collecting and processing external information such as disaster information and traffic conditions. Using federated learning technology, the server aggregates anonymized voice information, location information, and biometric information from each terminal. This optimizes risk information and evacuation routes for the entire region and generates individually optimized information models for each user. These calculation results are periodically distributed to each terminal.

[0079] Terminal configuration and operation

[0080] The device is a portable computing device equipped with sensors to collect voice data, location data, and health data in real time. Each device has a locally running generative artificial intelligence model for analyzing the collected data. The device uses this generative AI model to perform analysis based on the model database received from the server. For example, if a user lives in an area prone to earthquakes, the device will monitor heart rate and body temperature data, and if the user's health condition is not good, it will immediately provide voice instructions such as "Take a deep breath and calm down."

[0081] User experience

[0082] Users simply wear the mobile device and require no special operation. In the event of a disaster, they can take safe actions based on instructions from the device, providing valuable information for decision-making during chaotic situations. For example, if a flood warning is issued, the device will provide specific instructions via voice, such as "Immediately evacuate to the nearest high ground." This allows users to evacuate quickly and ensure their safety.

[0083] Examples of specific cases and prompt statements

[0084] For example, if a user is at increased risk of heatstroke due to the summer heat, the device will provide a voice alert such as, "Please stay hydrated." An example of a prompt in this scenario might be, "If the user's body temperature exceeds 38 degrees Celsius, please generate appropriate feedback immediately."

[0085] This system allows users to receive real-time, optimized evacuation information and health instructions even during disasters, enhancing security while protecting personal information.

[0086] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0087] Step 1:

[0088] The server receives voice information, location information, and biometric information from each terminal in real time. It receives anonymized data as input and combines it with external disaster information and traffic data. By integrating the data, it analyzes risk information for the entire region and obtains output that calculates the optimal evacuation route. Specifically, federated learning is used to aggregate information and update the model.

[0089] Step 2:

[0090] The server distributes optimized evacuation route information generated from the analysis results to each terminal. It uses an optimal evacuation route and hazard information model as input and applies it to the terminal's generated AI model. User-specific model data is generated as output. Specifically, the server schedules data transmission at regular intervals.

[0091] Step 3:

[0092] The device performs analysis based on model data delivered from the server and real-time data it collects itself. It uses biometric information such as heart rate and body temperature, as well as location information, as input, and performs analysis via a generated AI model. The output is the generation of optimal instructions and warnings to provide to the user. Specifically, the AI ​​model makes predictions and judgments, and prepares to output voice instructions from the device as needed.

[0093] Step 4:

[0094] The device provides voice feedback to the user based on the analysis results. As input, it references the analyzed user status data and generates optimal voice instructions such as "Please move to a safe location." As output, it provides the user with intuitive and clear voice guidance. Specifically, the device uses speech synthesis technology to send immediate feedback to the user.

[0095] Step 5:

[0096] The user takes safe actions according to the voice guidance. The input is receiving voice instructions from the device and performing physical actions accordingly. The output is taking safe evacuation actions. Specifically, the user follows the device's instructions and begins moving to an appropriate evacuation location.

[0097] This entire process enables the system to provide individualized and rapid support to users even during disasters.

[0098] (Application Example 1)

[0099] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0100] During disasters, it is difficult for people to obtain accurate information necessary for quick and safe evacuation. Furthermore, there are very few systems that provide appropriate evacuation instructions in real time, tailored to the individual physiological state of each user. In addition, there is a need for means to facilitate evacuation actions by providing not only audio but also visual information.

[0101] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0102] In this invention, the server includes means equipped with sensors that collect voice data, location information, and physiological data in real time; generative machine learning means that analyze the data locally and provide notifications based on the generated results; distributed learning means that cooperate with external information to provide disaster information and optimal evacuation routes; and means equipped with a video display device for providing the user's visual information. As a result, the user can obtain information in real time that will allow them to evacuate quickly and safely in the event of a disaster, and receive appropriate instructions according to their physiological state.

[0103] "Audio data" refers to digital or analog data acquired as audio information.

[0104] "Location information" refers to information that indicates the geographical location of a moving object or user.

[0105] "Physiological data" refers to data that indicates a person's physiological state, such as heart rate and body temperature.

[0106] A "portable information processing device" is a device that is portable and has the function of processing data.

[0107] "Generative machine learning methods" are means of analyzing data and generating results using machine learning models.

[0108] A "central information processing system" is a device used to collect and centrally process large amounts of data.

[0109] A "distributed learning method" is a learning method that distributes information processing among multiple computers or devices.

[0110] "Model parameters" are variable elements in machine learning that are adjusted by the learning algorithm.

[0111] A "video display device" is a device that can display visual information.

[0112] An "alarm" is a signal emitted with sound or light to indicate an abnormality or danger.

[0113] "Voice instructions" are instructions or commands given using voice.

[0114] This invention is a system designed to assist users in safely evacuating during disasters. The server utilizes distributed learning methods to anonymize and receive voice data, location information, and physiological data collected from each terminal, and performs real-time analysis. Based on the calculation results, the server generates the optimal evacuation route for the user. This generation process utilizes disaster information integrated with external information.

[0115] The terminal is carried by the user as a portable information processing device and acquires necessary data using its built-in sensors. This data is locally analyzed by machine learning generation tools, and notifications are sent as needed. In addition, visual information is provided through a video display device to support smooth evacuation actions.

[0116] As a concrete example, in the event of an earthquake, the device monitors the user's physiological data and, if their heart rate exceeds normal, it will emit a voice message saying, "Please remain calm." Additionally, a real-time map of evacuation routes is displayed on the video display device, visually guiding the user to the optimal route.

[0117] The analysis is performed using a generative AI model, and accurate feedback tailored to the user's situation is provided instantly. The generative AI model is provided with the following prompts as input:

[0118] "The user's heart rate has exceeded normal levels. Please suggest ways to reduce stress. Also, please display the best evacuation route from the current location."

[0119] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0120] Step 1:

[0121] The device uses built-in sensors to collect voice data, location information, and physiological data in real time. This input data is prepared for analysis by a generative AI model within the device. Specifically, the heart rate sensor reads the user's pulse, and the location sensor acquires GPS information.

[0122] Step 2:

[0123] The device inputs the collected data into a generative AI model for analysis. This analysis estimates the user's emotional state from voice data and detects changes in physical condition from physiological data. The generative AI model combines this information to evaluate the user's health and emotional state. The output is a judgment on whether the user is experiencing stress.

[0124] Step 3:

[0125] The server utilizes distributed learning methods to receive anonymized health data and location information from each terminal and collect risk information for the entire region. The input consists of data sets from multiple terminals. The server integrates this data with external disaster information to assess the risk level for the entire region and optimize evacuation routes. The output generates risk assessments and optimal route information for each region.

[0126] Step 4:

[0127] The server distributes the generated evacuation route information to each terminal. This information is also used to update the generated AI model on each terminal. The terminal receives this information and continuously performs analysis based on the updated model. Specifically, the server transfers data asynchronously over the network.

[0128] Step 5:

[0129] The terminal provides users with visual and audible feedback based on evacuation route information received from the server and local data. The AI ​​model generates contacts based on the data, specifically by issuing voice alerts such as "Please stay calm" and displaying evacuation routes on the screen. This allows users to receive information through both sight and sound, enabling them to proceed with evacuation smoothly.

[0130] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0131] This invention is a system that primarily uses a portable computing device that collects voice data, location information, and health data, and incorporates an emotion engine to analyze the user's emotional state in real time and provide appropriate feedback. The emotion engine determines the user's emotions by analyzing the voice data and provides specific feedback when an abnormal emotional state is detected. This feedback functions as advice to promote safe evacuation actions and mental and physical stabilization.

[0132] Server operation

[0133] The server delivers emotion engine models to the terminals based on federated learning technology. The server periodically updates these models with new data to improve the accuracy of emotion analysis. At the same time, it calculates the optimal evacuation route for the user, taking disaster information and location information into consideration.

[0134] Terminal operation

[0135] The device analyzes the user's voice data using an emotion engine, identifying their emotions based on the tone and content of their speech. This data is processed in real time, providing immediate feedback based on their emotional state. For example, if the device determines that the user is in a state of anxiety, it will issue a voice command such as "Please calm down" and simultaneously encourage deep breathing to help them feel at ease. At the same time, it also analyzes the user's health data and location information to provide evacuation route instructions as needed.

[0136] User experience

[0137] Users receive emotionally responsive feedback from their devices, enabling them to act safely and calmly. This feedback system is designed to help manage stress during disasters and support rational decision-making. If a user experiences a major earthquake and their anxiety increases, the device can select and deliver reassuring words via voice, and also provide calming music or visual relaxation tools.

[0138] Thus, the present invention achieves a high level of personalization tailored to individual circumstances by continuously detecting the user's emotional state and providing appropriate support according to that state.

[0139] The following describes the processing flow.

[0140] Step 1:

[0141] The device collects voice data, location information, and health data in real time. Built-in sensors enable this, and the data is stored in a format that allows for immediate processing.

[0142] Step 2:

[0143] The device analyzes the collected audio data using an emotion engine. It analyzes parameters such as voice tone, speaking speed, and volume to identify the user's emotional state.

[0144] Step 3:

[0145] Based on the emotional state analyzed by the device, if negative emotions are detected, personalized feedback is provided to the user. For example, if anxiety or restlessness is detected, it will give a voice command such as "Please calm down" and suggest deep breathing as a relaxation method.

[0146] Step 4:

[0147] The device continues to analyze health data and location information, integrating them based on federated learning results. If an increase in heart rate or a change in body temperature is detected, it provides appropriate health advice and simultaneously suggests evacuation routes to avoid dangerous areas.

[0148] Step 5:

[0149] The server updates the overall learning model based on anonymous data collected from terminals. If new patterns or trends are discovered, the system will use them to improve evacuation routes and enhance the accuracy of emotion recognition.

[0150] Step 6:

[0151] The server sends updated model data to the device. This data is used to improve the accuracy of the device's generation AI and emotion engine, and to prepare for the next disaster.

[0152] Step 7:

[0153] By following voice instructions and vibration feedback from the device, users can receive emotional and health support and take safe actions. At appropriate times, they can also use voice commands to request additional information or adjust their emotions.

[0154] (Example 2)

[0155] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0156] In modern society, there is a growing need to respond quickly to natural disasters and changes in individual health conditions. Conventional information devices have struggled to accurately analyze users' emotional states and provide appropriate instructions and evacuation routes in real time. In particular, there is a need for methods that support mental stability and rational decision-making by providing personalized feedback.

[0157] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0158] In this invention, the server includes a portable information processing device equipped with a detection device that collects voice information, location information, and health information in real time; a generative intelligence means that analyzes the information locally and provides notifications by voice or vibration based on the generated results; and a distributed learning means that utilizes a central information processing device that cooperates with external information to provide disaster information and the user with the optimal evacuation route. This makes it possible to perform sentiment analysis on the user in real time and provide feedback and optimal evacuation instructions tailored to the individual situation.

[0159] "Voice information" refers to data collected digitally from the user's voice, which is used to analyze the content and tone of their speech.

[0160] "Location information" refers to geographical data used to determine a user's current location, and is primarily obtained through GPS technology.

[0161] "Health information" refers to physiological data such as the user's heart rate and physical activity level, and is used to evaluate their health status.

[0162] "Detection device" is a general term that includes sensor devices and technologies for acquiring voice information, location information, and health information in real time.

[0163] A "portable information processing device" refers to a portable electronic device equipped with the aforementioned detection device that can collect and perform initial analysis of data.

[0164] A "generative intelligence system" is a system equipped with artificial intelligence algorithms for performing speech analysis and feedback generation.

[0165] "Disaster information" refers to information about natural disasters such as earthquakes and floods, and is obtained from a database that is updated in real time.

[0166] A "central information processing system" refers to a centralized computer system that aggregates multiple data sources and performs advanced analysis.

[0167] "Distributed learning methods" are technologies used to efficiently update and optimize learning models across multiple information processing devices.

[0168] This invention is a system that collects voice information, location information, and health information in real time, analyzes the user's emotional state, and provides appropriate feedback. This system mainly consists of a portable information processing device, a central information processing device, and a generative intelligence means.

[0169] Terminal operation

[0170] The device is a portable information processing unit equipped with a microphone for acquiring voice information, GPS functionality for acquiring location information, and a sensor device (e.g., a smartwatch) for acquiring health information. This data is stored in the device's internal database in real time. The device is equipped with generative intelligence and uses natural language processing technology and speech analysis algorithms to analyze the user's emotional state from the collected voice information. Based on this analysis, the device uses a generative AI model to generate prompt sentences such as "How are you feeling today?" and provides feedback to the user in voice or text. Furthermore, it can instruct the user to take actions to reassure them as needed (e.g., prompting them to take a deep breath).

[0171] Server operation

[0172] The server integrates the collected data, and a central information processing unit calculates the optimal evacuation route based on disaster information. Disaster information is obtained from a real-time updated database and combined with location information to provide users with quick and appropriate evacuation instructions. The server utilizes distributed learning to constantly update the generative intelligence model, striving to improve accuracy. The updated model information is delivered to terminals and used to enhance the accuracy of sentiment analysis.

[0173] User experience

[0174] Users receive real-time feedback through their devices in the form of voice, visuals, and vibrations. For example, during a large-scale disaster, when a user's anxiety increases, the device provides reassuring voice instructions and displays a map showing evacuation routes. In this way, users can use the system to gain a sense of security and evacuate efficiently.

[0175] Thus, the present invention supports optimal actions tailored to the user's emotional state by providing highly personalized assistance.

[0176] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0177] Step 1:

[0178] The device collects voice information, location information, and health information from the user in real time. Voice information is acquired via a microphone, location information is measured by a GPS sensor, and health information is captured from sensor devices such as smartwatches. Voice information, location information, and health information are acquired as input data and stored in an internal database.

[0179] Step 2:

[0180] The terminal inputs the acquired voice information into a generative intelligence system for emotion analysis. A voice analysis algorithm is applied, and data calculations are performed to identify the emotional state from the user's speech content and tone. The result of the emotional state evaluation is obtained as output.

[0181] Step 3:

[0182] The device uses a generative AI model to generate a prompt for feedback based on the evaluation of the user's emotional state. For example, if the user is analyzed as being in a stressed state, the generative AI model will generate a prompt such as, "Try taking a deep breath to relax." The prompt is then output.

[0183] Step 4:

[0184] The server performs data calculations to determine safe evacuation routes based on collected location and disaster information. It retrieves the latest disaster information from an external disaster database and combines it with location information to identify the optimal route. The calculated evacuation route is then output.

[0185] Step 5:

[0186] The device notifies the user of the generated prompt and calculated evacuation route. It provides feedback to the user using voice instructions, text display, and vibration. It conveys reassuring action instructions and specific evacuation suggestions to the user.

[0187] Step 6:

[0188] The user takes concrete action based on feedback and evacuation instructions from the device. For example, they might take a deep breath as instructed by the device and move to a safe location along an evacuation route. As a result, the user can achieve a safer and more secure state of mind.

[0189] (Application Example 2)

[0190] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0191] In modern work environments, especially during long hours or complex tasks, the emotional state of workers can have a serious impact on productivity and safety. Conventional methods make it difficult to provide immediate feedback on workers' emotional and physical states, resulting in decreased work efficiency and an increased risk of accidents. This invention aims to solve these problems and improve safety and productivity in the work environment.

[0192] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0193] In this invention, the server includes an information processing device equipped with sensors that collect voice data, location information, and health data in real time; a generative artificial intelligence means that analyzes the data locally, provides notifications by voice or vibration based on the generated results, identifies the user's emotional state in the work environment, and provides feedback; a federated learning means that utilizes a central computing device that has the function of detecting abnormal emotional states for the safety of the work environment and presenting appropriate behavioral guidelines to the user; and a means for distributing updated model data from the federated learning means to the information processing device. This makes it possible to grasp the emotions and health status of workers in real time and maintain an efficient and safe work environment.

[0194] "Audio data" refers to digital information that records sounds and speech, and serves as fundamental data for analyzing the tone and content of a user's speech.

[0195] "Location information" refers to data that indicates the geographical location of a user or device, and is used to analyze the work environment and behavior.

[0196] "Health data" refers to data that indicates the user's physical condition, such as heart rate, stress level, and overall health.

[0197] An "information processing device" is a portable computer device used for data collection, analysis, and feedback.

[0198] "Generative artificial intelligence means" refers to a technology that analyzes data collected in real time, determines the user's emotional state, and provides appropriate feedback.

[0199] "Federated learning" is a learning method in which each device independently learns and updates its model without going through a central server, and then shares the results with other devices to improve the overall model.

[0200] A "central computing device" is a server or large computer device used to aggregate data, perform complex calculations, and manage overall information.

[0201] This invention is a system that uses a portable information processing device to collect voice data, location information, and health data in real time and analyze the user's emotional state. In particular, it aims to improve efficiency and safety in the work environment. The elements of this system are described below.

[0202] 1. Hardware

[0203] The terminal, which is an information processing device, is equipped with a voice sensor, a location information acquisition device, and a health monitoring device, and collects various data through them. This allows for real-time monitoring of the user's situation and enables the rapid provision of necessary feedback.

[0204] 2. Software

[0205] The device's generated artificial intelligence analyzes collected data to identify the user's emotional state. Based on this data, it then provides audio or visual feedback. The system is designed to continuously improve analysis accuracy by delivering the latest model to the device through federated learning.

[0206] 3. Data processing and calculations

[0207] The server acts as a central computing device, aggregating data sent from multiple terminals and updating the models deployed on each device using federated learning technology. In this process, each terminal learns the model independently, thus protecting user privacy while improving the overall accuracy of the system.

[0208] Specific example

[0209] Imagine a factory worker using an information processing device while working. This device detects that the worker is experiencing stress based on voice data and health data. The device immediately notifies the user via voice, "You should take a short break," prompting them to take a short rest. This type of feedback contributes to improved work efficiency and enhanced safety.

[0210] Example of a prompt

[0211] "Develop a system that utilizes a generative AI model to analyze user voice data, determine their emotional state in real time, and provide appropriate feedback. Clearly demonstrate how the device analyzes the data and presents optimal action guidelines."

[0212] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0213] Step 1:

[0214] The device uses a voice sensor, location information acquisition device, and health monitoring device to collect user voice data, location information, and health data in real time. In this process, the sensors capture each type of data, and the data is temporarily stored within the device. The input is various types of user data, and the output is the stored raw data.

[0215] Step 2:

[0216] The device analyzes collected audio data using generative artificial intelligence. It analyzes the tone and content of the audio data to perform data processing that identifies the user's emotional state. The input is stored raw audio data, and the output is analyzed emotional state information. Specifically, it extracts feature quantities from the audio waveform and inputs them into an emotion model to estimate emotions.

[0217] Step 3:

[0218] The device analyzes location information and health data to evaluate the user's work environment and health status. This includes a process that uses data calculations to detect abnormalities in the user's movements and physical condition. The inputs are location information and health data, and the outputs are indicators of the user's activity level and health status. Specifically, it analyzes changes in location and fluctuations in heart rate and compares them to standard values.

[0219] Step 4:

[0220] The server uses federated learning technology to integrate the data learned by each device and update the overall model. The updated model is then distributed back to the devices, improving the accuracy of the analysis. The input is the learning results from each device, and the output is the integrated, up-to-date generative AI model.

[0221] Step 5:

[0222] The device provides the user with audio or visual feedback based on the analysis results. This feedback is automatically generated based on the user's current emotional and health status. The input is the analysis results and the updated model, and the output is the content of the feedback. In a specific example, if stress is detected, a notification such as "Take a break" is issued.

[0223] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0224] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0225] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0226] [Second Embodiment]

[0227] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0228] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0229] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0230] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0231] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0232] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0233] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0234] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0235] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0236] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0237] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0238] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0239] The system according to the present invention consists of a server, terminals, and users, and supports the safe evacuation of users in the event of a disaster. Each terminal is equipped with sensors for acquiring voice data, location information, and health data, and this data is collected in real time. The data is analyzed by local generative artificial intelligence means, and appropriate feedback is provided according to the user's situation.

[0240] Server operation

[0241] The server uses federated learning technology to aggregate anonymized information from each terminal. Based on the collected data, it optimizes risk information and evacuation routes for the entire region. It also acquires external disaster information and traffic data in real time and combines and analyzes them to generate optimal evacuation route information for each individual. The server periodically distributes these calculation results to each terminal and updates the terminal's model data.

[0242] Terminal operation

[0243] The device performs analysis using generated AI based on model data received from the server. For example, if a user lives in an area affected by an earthquake, the device monitors the user's health based on acquired heart rate, body temperature, and location information. If the user's health deteriorates, the device immediately issues voice instructions such as "Take a deep breath" or "Move to a safe place." These instructions are linked to the optimal evacuation route based on the user's location information.

[0244] User experience

[0245] Under normal circumstances, users simply wear the device without requiring any special operation. In the event of a disaster, users receive intuitive feedback from the device and take safe actions based on the instructions. For example, when a flood warning is issued, the device will give a voice instruction to "immediately evacuate to higher ground," allowing users to quickly begin evacuation without confusion.

[0246] This system effectively supports user safety by providing personalized support that can respond immediately during disasters while protecting personal information.

[0247] The following describes the processing flow.

[0248] Step 1:

[0249] The device collects voice data, location information, and health data in real time. It continuously acquires data through sensors and stores it locally.

[0250] Step 2:

[0251] The data collected by the terminal is preprocessed in real time. This includes noise reduction and data cleaning, and conversion to a format suitable for analysis.

[0252] Step 3:

[0253] The device uses AI to analyze pre-processed data. Based on the analysis results, it evaluates the user's health status and risk level based on their current location.

[0254] Step 4:

[0255] The device provides feedback to the user based on the analysis results. This feedback is delivered via voice notifications and vibrations and includes recommended actions and evacuation route information.

[0256] Step 5:

[0257] The server performs federated learning based on anonymized data collected from each terminal. The overall model is updated to calculate the optimal evacuation route, taking into account disaster information and user-specific data.

[0258] Step 6:

[0259] The server delivers updated model data to the terminal. The terminal uses this data to prepare for the next analysis and to provide more accurate feedback.

[0260] Step 7:

[0261] Users will take necessary evacuation actions according to notifications on their devices. Based on voice notifications, they will choose appropriate routes and actions to ensure their safety.

[0262] Step 8:

[0263] Users can use voice commands as needed to provide additional information to the terminal. This allows the system to improve its feedback in response to the user's specific requests.

[0264] (Example 1)

[0265] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0266] During disasters, it is difficult to quickly and accurately provide each individual user with the most suitable evacuation route. Furthermore, mechanisms for monitoring individuals' health status in real time and providing appropriate instructions are not yet fully established. As a result, users may not be able to evacuate properly during emergencies, potentially increasing their health risks. Additionally, obtaining useful information while maintaining data anonymity is another challenge.

[0267] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0268] In this invention, the server includes a data collection device that collects voice information, location information, and biometric information in real time; a knowledge processing means that analyzes the information on a local device and issues instructions based on the results of the analysis; an aggregate learning means that cooperates with external information to provide disaster information and user-optimized evacuation routes; and a means that distributes an updated information model from the aggregate learning means to the data collection device. This makes it possible to provide personalized evacuation instructions and health management to each user in real time, even during a disaster.

[0269] "Audio information" refers to data that represents the characteristics of sound in digital format, and is used to understand instructions and situations based on the user's voice.

[0270] "Location information" refers to geographical coordinate data, which is used to identify a user's current location and travel route.

[0271] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate and body temperature, and is collected for health management and risk assessment.

[0272] A "data collection device" is a device that includes sensors and equipment for acquiring voice information, location information, and biometric information, and is responsible for collecting information from users.

[0273] A "local device" is a computing device located near the user that analyzes collected data and generates instructions in real time.

[0274] "Knowledge processing means" refers to artificial intelligence technology that analyzes collected data to generate optimal instructions for the user.

[0275] "External information" refers to data obtained from outside the system, such as disaster information and traffic conditions, and is used to optimize evacuation routes.

[0276] "Aggregated learning methods" refer to machine learning techniques that integrate data obtained from multiple sources to perform overall optimization and analysis.

[0277] An "information model" refers to data that includes the calculation results of optimized evacuation routes and disaster information, generated by aggregate learning methods and provided to users.

[0278] This invention relates to an advanced information system for enhancing user safety during disasters, in which the server, terminal, and user components work in coordination. The embodiments thereof are described below.

[0279] Server configuration and operation

[0280] The server includes central computing devices for collecting and processing external information such as disaster information and traffic conditions. Using federated learning technology, the server aggregates anonymized voice information, location information, and biometric information from each terminal. Thereby, it optimizes the danger information and evacuation routes for the entire region and generates an individually optimized information model for each user. This calculation result is periodically distributed to each terminal.

[0281] Configuration and Operation of the Terminal

[0282] The terminal is a portable computing device equipped with sensors for collecting voice data, location data, and health data in real time. Each terminal is equipped with a locally operating generative artificial intelligence model for analyzing the collected data. The terminal performs analysis based on the received model database of the server using this generative AI model. As a specific example, when the user lives in an area prone to being affected by an earthquake, the terminal monitors data such as heart rate and body temperature, and if the user's health condition is not good, it immediately provides a voice instruction such as "Please take a deep breath and calm down."

[0283] User Experience

[0284] The user only needs to wear the portable terminal without requiring special operations. When a disaster occurs, the user can take safe actions based on the instructions from the terminal, which serves as a basis for judgment during chaos. For example, when a flood warning is issued, the terminal gives a specific action instruction such as "Immediately evacuate to the nearest high ground" in voice. Thereby, the user can evacuate quickly and ensure safety.

[0285] Examples of Specific Cases and Prompt Sentences

[0286] For example, when the risk of a user suffering from heatstroke due to the heat in summer is increasing, the terminal provides an audible alert of "Please replenish moisture". As an example of a prompt sentence in this scenario, content such as "When the user's body temperature exceeds 38 degrees, please quickly generate appropriate feedback" can be considered.

[0287] With this system, users can obtain real-time optimized evacuation information and health instructions even during disasters, and it is possible to enhance safety while protecting personal information.

[0288] The flow of the specific process in Example 1 will be described using FIG. 11.

[0289] Step 1:

[0290] The server receives voice information, location information, and biometric information from each terminal in real time. As input, it receives anonymized data and combines it with external disaster information and traffic data. It integrates the data, analyzes the risk information for the entire region, and obtains an output of calculating the optimal evacuation route. As a specific operation, it uses federated learning to aggregate information and update the model.

[0291] Step 2:

[0292] The server distributes the optimized evacuation route information generated from the analysis results to each terminal. As input, it uses the optimal evacuation route and the risk information model and applies it to the terminal's generation AI model. As output, model data dedicated to the user is generated. As a specific operation, the server schedules to send data at regular intervals.

[0293] Step 3:

[0294] The device performs analysis based on model data delivered from the server and real-time data it collects itself. It uses biometric information such as heart rate and body temperature, as well as location information, as input, and performs analysis via a generated AI model. The output is the generation of optimal instructions and warnings to provide to the user. Specifically, the AI ​​model makes predictions and judgments, and prepares to output voice instructions from the device as needed.

[0295] Step 4:

[0296] The device provides voice feedback to the user based on the analysis results. As input, it references the analyzed user status data and generates optimal voice instructions such as "Please move to a safe location." As output, it provides the user with intuitive and clear voice guidance. Specifically, the device uses speech synthesis technology to send immediate feedback to the user.

[0297] Step 5:

[0298] The user takes safe actions according to the voice guidance. The input is receiving voice instructions from the device and performing physical actions accordingly. The output is taking safe evacuation actions. Specifically, the user follows the device's instructions and begins moving to an appropriate evacuation location.

[0299] This entire process enables the system to provide individualized and rapid support to users even during disasters.

[0300] (Application Example 1)

[0301] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0302] In the event of a disaster, it is difficult for users to accurately obtain information for a quick and safe evacuation. Moreover, there are almost no systems that can provide appropriate evacuation instructions in real time according to the physiological conditions of individual users. Furthermore, there is a need for means to make the evacuation process smoother by providing not only voice but also visual information.

[0303] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0304] In this invention, the server includes means equipped with sensors for collecting voice data, location information, and physiological data in real time, generative machine learning means for locally analyzing the data and making notifications based on the generated results, distributed learning means for collaborating with external information to provide disaster information and optimal evacuation routes, and means equipped with a video display device for providing visual information of the user. As a result, users can obtain in real time information that enables them to evacuate quickly and safely during a disaster and can receive appropriate instructions according to their physiological conditions.

[0305] "Voice data" refers to digital or analog data obtained as voice information.

[0306] "Location information" refers to information indicating the geographical location of a moving object or user.

[0307] "Physiological data" refers to data indicating the physiological state of a human, such as heart rate and body temperature.

[0308] "Portable information processing device" refers to a device that is portable and has the function of performing data processing.

[0309] "Generative machine learning means" refers to means for analyzing data using a machine learning model and generating results.

[0310] <0000%77>"Central information processing device" refers to a device for aggregating a large amount of data and performing centralized processing.

[0311] A "distributed learning method" is a learning method that distributes information processing among multiple computers or devices.

[0312] "Model parameters" are variable elements in machine learning that are adjusted by the learning algorithm.

[0313] A "video display device" is a device that can display visual information.

[0314] An "alarm" is a signal emitted with sound or light to indicate an abnormality or danger.

[0315] "Voice instructions" are instructions or commands given using voice.

[0316] This invention is a system designed to assist users in safely evacuating during disasters. The server utilizes distributed learning methods to anonymize and receive voice data, location information, and physiological data collected from each terminal, and performs real-time analysis. Based on the calculation results, the server generates the optimal evacuation route for the user. This generation process utilizes disaster information integrated with external information.

[0317] The terminal is carried by the user as a portable information processing device and acquires necessary data using its built-in sensors. This data is locally analyzed by machine learning generation tools, and notifications are sent as needed. In addition, visual information is provided through a video display device to support smooth evacuation actions.

[0318] As a concrete example, in the event of an earthquake, the device monitors the user's physiological data and, if their heart rate exceeds normal, it will emit a voice message saying, "Please remain calm." Additionally, a real-time map of evacuation routes is displayed on the video display device, visually guiding the user to the optimal route.

[0319] The analysis is performed using a generative AI model, and accurate feedback tailored to the user's situation is provided instantly. The generative AI model is provided with the following prompts as input:

[0320] "The user's heart rate has exceeded normal levels. Please suggest ways to reduce stress. Also, please display the best evacuation route from the current location."

[0321] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0322] Step 1:

[0323] The device uses built-in sensors to collect voice data, location information, and physiological data in real time. This input data is prepared for analysis by a generative AI model within the device. Specifically, the heart rate sensor reads the user's pulse, and the location sensor acquires GPS information.

[0324] Step 2:

[0325] The device inputs the collected data into a generative AI model for analysis. This analysis estimates the user's emotional state from voice data and detects changes in physical condition from physiological data. The generative AI model combines this information to evaluate the user's health and emotional state. The output is a judgment on whether the user is experiencing stress.

[0326] Step 3:

[0327] The server utilizes distributed learning methods to receive anonymized health data and location information from each terminal and collect risk information for the entire region. The input consists of data sets from multiple terminals. The server integrates this data with external disaster information to assess the risk level for the entire region and optimize evacuation routes. The output generates risk assessments and optimal route information for each region.

[0328] Step 4:

[0329] The server distributes the generated evacuation route information to each terminal. This information is also used to update the generated AI model on each terminal. The terminal receives this information and continuously performs analysis based on the updated model. Specifically, the server transfers data asynchronously over the network.

[0330] Step 5:

[0331] The terminal provides users with visual and audible feedback based on evacuation route information received from the server and local data. The AI ​​model generates contacts based on the data, specifically by issuing voice alerts such as "Please stay calm" and displaying evacuation routes on the screen. This allows users to receive information through both sight and sound, enabling them to proceed with evacuation smoothly.

[0332] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0333] This invention is a system that primarily uses a portable computing device that collects voice data, location information, and health data, and incorporates an emotion engine to analyze the user's emotional state in real time and provide appropriate feedback. The emotion engine determines the user's emotions by analyzing the voice data and provides specific feedback when an abnormal emotional state is detected. This feedback functions as advice to promote safe evacuation actions and mental and physical stabilization.

[0334] Server operation

[0335] The server delivers emotion engine models to the terminals based on federated learning technology. The server periodically updates these models with new data to improve the accuracy of emotion analysis. At the same time, it calculates the optimal evacuation route for the user, taking disaster information and location information into consideration.

[0336] Terminal operation

[0337] The device analyzes the user's voice data using an emotion engine, identifying their emotions based on the tone and content of their speech. This data is processed in real time, providing immediate feedback based on their emotional state. For example, if the device determines that the user is in a state of anxiety, it will issue a voice command such as "Please calm down" and simultaneously encourage deep breathing to help them feel at ease. At the same time, it also analyzes the user's health data and location information to provide evacuation route instructions as needed.

[0338] User experience

[0339] Users receive emotionally responsive feedback from their devices, enabling them to act safely and calmly. This feedback system is designed to help manage stress during disasters and support rational decision-making. If a user experiences a major earthquake and their anxiety increases, the device can select and deliver reassuring words via voice, and also provide calming music or visual relaxation tools.

[0340] Thus, the present invention achieves a high level of personalization tailored to individual circumstances by continuously detecting the user's emotional state and providing appropriate support according to that state.

[0341] The following describes the processing flow.

[0342] Step 1:

[0343] The device collects voice data, location information, and health data in real time. Built-in sensors enable this, and the data is stored in a format that allows for immediate processing.

[0344] Step 2:

[0345] The device analyzes the collected audio data using an emotion engine. It analyzes parameters such as voice tone, speaking speed, and volume to identify the user's emotional state.

[0346] Step 3:

[0347] Based on the emotional state analyzed by the device, if negative emotions are detected, personalized feedback is provided to the user. For example, if anxiety or restlessness is detected, it will give a voice command such as "Please calm down" and suggest deep breathing as a relaxation method.

[0348] Step 4:

[0349] The device continues to analyze health data and location information, integrating them based on federated learning results. If an increase in heart rate or a change in body temperature is detected, it provides appropriate health advice and simultaneously suggests evacuation routes to avoid dangerous areas.

[0350] Step 5:

[0351] The server updates the overall learning model based on anonymous data collected from terminals. If new patterns or trends are discovered, the system will use them to improve evacuation routes and enhance the accuracy of emotion recognition.

[0352] Step 6:

[0353] The server sends updated model data to the device. This data is used to improve the accuracy of the device's generation AI and emotion engine, and to prepare for the next disaster.

[0354] Step 7:

[0355] By following voice instructions and vibration feedback from the device, users can receive emotional and health support and take safe actions. At appropriate times, they can also use voice commands to request additional information or adjust their emotions.

[0356] (Example 2)

[0357] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0358] In modern society, there is a growing need to respond quickly to natural disasters and changes in individual health conditions. Conventional information devices have struggled to accurately analyze users' emotional states and provide appropriate instructions and evacuation routes in real time. In particular, there is a need for methods that support mental stability and rational decision-making by providing personalized feedback.

[0359] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0360] In this invention, the server includes a portable information processing device equipped with a detection device that collects voice information, location information, and health information in real time; a generative intelligence means that analyzes the information locally and provides notifications by voice or vibration based on the generated results; and a distributed learning means that utilizes a central information processing device that cooperates with external information to provide disaster information and the user with the optimal evacuation route. This makes it possible to perform sentiment analysis on the user in real time and provide feedback and optimal evacuation instructions tailored to the individual situation.

[0361] "Voice information" refers to data collected digitally from the user's voice, which is used to analyze the content and tone of their speech.

[0362] "Location information" refers to geographical data used to determine a user's current location, and is primarily obtained through GPS technology.

[0363] "Health information" refers to physiological data such as the user's heart rate and physical activity level, and is used to evaluate their health status.

[0364] "Detection device" is a general term that includes sensor devices and technologies for acquiring voice information, location information, and health information in real time.

[0365] A "portable information processing device" refers to a portable electronic device equipped with the aforementioned detection device that can collect and perform initial analysis of data.

[0366] A "generative intelligence system" is a system equipped with artificial intelligence algorithms for performing speech analysis and feedback generation.

[0367] "Disaster information" refers to information about natural disasters such as earthquakes and floods, and is obtained from a database that is updated in real time.

[0368] A "central information processing system" refers to a centralized computer system that aggregates multiple data sources and performs advanced analysis.

[0369] "Distributed learning methods" are technologies used to efficiently update and optimize learning models across multiple information processing devices.

[0370] This invention is a system that collects voice information, location information, and health information in real time, analyzes the user's emotional state, and provides appropriate feedback. This system mainly consists of a portable information processing device, a central information processing device, and a generative intelligence means.

[0371] Terminal operation

[0372] The device is a portable information processing unit equipped with a microphone for acquiring voice information, GPS functionality for acquiring location information, and a sensor device (e.g., a smartwatch) for acquiring health information. This data is stored in the device's internal database in real time. The device is equipped with generative intelligence and uses natural language processing technology and speech analysis algorithms to analyze the user's emotional state from the collected voice information. Based on this analysis, the device uses a generative AI model to generate prompt sentences such as "How are you feeling today?" and provides feedback to the user in voice or text. Furthermore, it can instruct the user to take actions to reassure them as needed (e.g., prompting them to take a deep breath).

[0373] Server operation

[0374] The server integrates the collected data, and a central information processing unit calculates the optimal evacuation route based on disaster information. Disaster information is obtained from a real-time updated database and combined with location information to provide users with quick and appropriate evacuation instructions. The server utilizes distributed learning to constantly update the generative intelligence model, striving to improve accuracy. The updated model information is delivered to terminals and used to enhance the accuracy of sentiment analysis.

[0375] User experience

[0376] Users receive real-time feedback through their devices in the form of voice, visuals, and vibrations. For example, during a large-scale disaster, when a user's anxiety increases, the device provides reassuring voice instructions and displays a map showing evacuation routes. In this way, users can use the system to gain a sense of security and evacuate efficiently.

[0377] Thus, the present invention supports optimal actions tailored to the user's emotional state by providing highly personalized assistance.

[0378] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0379] Step 1:

[0380] The device collects voice information, location information, and health information from the user in real time. Voice information is acquired via a microphone, location information is measured by a GPS sensor, and health information is captured from sensor devices such as smartwatches. Voice information, location information, and health information are acquired as input data and stored in an internal database.

[0381] Step 2:

[0382] The terminal inputs the acquired voice information into a generative intelligence system for emotion analysis. A voice analysis algorithm is applied, and data calculations are performed to identify the emotional state from the user's speech content and tone. The result of the emotional state evaluation is obtained as output.

[0383] Step 3:

[0384] The device uses a generative AI model to generate a prompt for feedback based on the evaluation of the user's emotional state. For example, if the user is analyzed as being in a stressed state, the generative AI model will generate a prompt such as, "Try taking a deep breath to relax." The prompt is then output.

[0385] Step 4:

[0386] The server performs data calculations to determine safe evacuation routes based on collected location and disaster information. It retrieves the latest disaster information from an external disaster database and combines it with location information to identify the optimal route. The calculated evacuation route is then output.

[0387] Step 5:

[0388] The device notifies the user of the generated prompt and calculated evacuation route. It provides feedback to the user using voice instructions, text display, and vibration. It conveys reassuring action instructions and specific evacuation suggestions to the user.

[0389] Step 6:

[0390] The user takes concrete action based on feedback and evacuation instructions from the device. For example, they might take a deep breath as instructed by the device and move to a safe location along an evacuation route. As a result, the user can achieve a safer and more secure state of mind.

[0391] (Application Example 2)

[0392] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0393] In modern work environments, especially during long hours or complex tasks, the emotional state of workers can have a serious impact on productivity and safety. Conventional methods make it difficult to provide immediate feedback on workers' emotional and physical states, resulting in decreased work efficiency and an increased risk of accidents. This invention aims to solve these problems and improve safety and productivity in the work environment.

[0394] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0395] In this invention, the server includes an information processing device equipped with sensors that collect voice data, location information, and health data in real time; a generative artificial intelligence means that analyzes the data locally, provides notifications by voice or vibration based on the generated results, identifies the user's emotional state in the work environment, and provides feedback; a federated learning means that utilizes a central computing device that has the function of detecting abnormal emotional states for the safety of the work environment and presenting appropriate behavioral guidelines to the user; and a means for distributing updated model data from the federated learning means to the information processing device. This makes it possible to grasp the emotions and health status of workers in real time and maintain an efficient and safe work environment.

[0396] "Audio data" refers to digital information that records sounds and speech, and serves as fundamental data for analyzing the tone and content of a user's speech.

[0397] "Location information" refers to data that indicates the geographical location of a user or device, and is used to analyze the work environment and behavior.

[0398] "Health data" refers to data that indicates the user's physical condition, such as heart rate, stress level, and overall health.

[0399] An "information processing device" is a portable computer device used for data collection, analysis, and feedback.

[0400] "Generative artificial intelligence means" refers to a technology that analyzes data collected in real time, determines the user's emotional state, and provides appropriate feedback.

[0401] "Federated learning" is a learning method in which each device independently learns and updates its model without going through a central server, and then shares the results with other devices to improve the overall model.

[0402] A "central computing device" is a server or large computer device used to aggregate data, perform complex calculations, and manage overall information.

[0403] This invention is a system that uses a portable information processing device to collect voice data, location information, and health data in real time and analyze the user's emotional state. In particular, it aims to improve efficiency and safety in the work environment. The elements of this system are described below.

[0404] 1. Hardware

[0405] The terminal, which is an information processing device, is equipped with a voice sensor, a location information acquisition device, and a health monitoring device, and collects various data through them. This allows for real-time monitoring of the user's situation and enables the rapid provision of necessary feedback.

[0406] 2. Software

[0407] The device's generated artificial intelligence analyzes collected data to identify the user's emotional state. Based on this data, it then provides audio or visual feedback. The system is designed to continuously improve analysis accuracy by delivering the latest model to the device through federated learning.

[0408] 3. Data processing and calculations

[0409] The server acts as a central computing device, aggregating data sent from multiple terminals and updating the models deployed on each device using federated learning technology. In this process, each terminal learns the model independently, thus protecting user privacy while improving the overall accuracy of the system.

[0410] Specific example

[0411] Imagine a factory worker using an information processing device while working. This device detects that the worker is experiencing stress based on voice data and health data. The device immediately notifies the user via voice, "You should take a short break," prompting them to take a short rest. This type of feedback contributes to improved work efficiency and enhanced safety.

[0412] Example of a prompt

[0413] "Develop a system that utilizes a generative AI model to analyze user voice data, determine their emotional state in real time, and provide appropriate feedback. Clearly demonstrate how the device analyzes the data and presents optimal action guidelines."

[0414] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0415] Step 1:

[0416] The device uses a voice sensor, location information acquisition device, and health monitoring device to collect user voice data, location information, and health data in real time. In this process, the sensors capture each type of data, and the data is temporarily stored within the device. The input is various types of user data, and the output is the stored raw data.

[0417] Step 2:

[0418] The device analyzes collected audio data using generative artificial intelligence. It analyzes the tone and content of the audio data to perform data processing that identifies the user's emotional state. The input is stored raw audio data, and the output is analyzed emotional state information. Specifically, it extracts feature quantities from the audio waveform and inputs them into an emotion model to estimate emotions.

[0419] Step 3:

[0420] The device analyzes location information and health data to evaluate the user's work environment and health status. This includes a process that uses data calculations to detect abnormalities in the user's movements and physical condition. The inputs are location information and health data, and the outputs are indicators of the user's activity level and health status. Specifically, it analyzes changes in location and fluctuations in heart rate and compares them to standard values.

[0421] Step 4:

[0422] The server uses federated learning technology to integrate the data learned by each device and update the overall model. The updated model is then distributed back to the devices, improving the accuracy of the analysis. The input is the learning results from each device, and the output is the integrated, up-to-date generative AI model.

[0423] Step 5:

[0424] The device provides the user with audio or visual feedback based on the analysis results. This feedback is automatically generated based on the user's current emotional and health status. The input is the analysis results and the updated model, and the output is the content of the feedback. In a specific example, if stress is detected, a notification such as "Take a break" is issued.

[0425] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0426] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0427] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0428] [Third Embodiment]

[0429] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0430] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0431] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0432] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0433] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0434] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0435] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0436] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0437] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0438] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0439] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0440] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0441] The system according to the present invention consists of a server, terminals, and users, and supports the safe evacuation of users in the event of a disaster. Each terminal is equipped with sensors for acquiring voice data, location information, and health data, and this data is collected in real time. The data is analyzed by local generative artificial intelligence means, and appropriate feedback is provided according to the user's situation.

[0442] Server operation

[0443] The server uses federated learning technology to aggregate anonymized information from each terminal. Based on the collected data, it optimizes risk information and evacuation routes for the entire region. It also acquires external disaster information and traffic data in real time and combines and analyzes them to generate optimal evacuation route information for each individual. The server periodically distributes these calculation results to each terminal and updates the terminal's model data.

[0444] Terminal operation

[0445] The device performs analysis using generated AI based on model data received from the server. For example, if a user lives in an area affected by an earthquake, the device monitors the user's health based on acquired heart rate, body temperature, and location information. If the user's health deteriorates, the device immediately issues voice instructions such as "Take a deep breath" or "Move to a safe place." These instructions are linked to the optimal evacuation route based on the user's location information.

[0446] User experience

[0447] Under normal circumstances, users simply wear the device without requiring any special operation. In the event of a disaster, users receive intuitive feedback from the device and take safe actions based on the instructions. For example, when a flood warning is issued, the device will give a voice instruction to "immediately evacuate to higher ground," allowing users to quickly begin evacuation without confusion.

[0448] This system effectively supports user safety by providing personalized support that can respond immediately during disasters while protecting personal information.

[0449] The following describes the processing flow.

[0450] Step 1:

[0451] The device collects voice data, location information, and health data in real time. It continuously acquires data through sensors and stores it locally.

[0452] Step 2:

[0453] The data collected by the terminal is preprocessed in real time. This includes noise reduction and data cleaning, and conversion to a format suitable for analysis.

[0454] Step 3:

[0455] The device uses AI to analyze pre-processed data. Based on the analysis results, it evaluates the user's health status and risk level based on their current location.

[0456] Step 4:

[0457] The device provides feedback to the user based on the analysis results. This feedback is delivered via voice notifications and vibrations and includes recommended actions and evacuation route information.

[0458] Step 5:

[0459] The server performs federated learning based on anonymized data collected from each terminal. The overall model is updated to calculate the optimal evacuation route, taking into account disaster information and user-specific data.

[0460] Step 6:

[0461] The server delivers updated model data to the terminal. The terminal uses this data to prepare for the next analysis and to provide more accurate feedback.

[0462] Step 7:

[0463] Users will take necessary evacuation actions according to notifications on their devices. Based on voice notifications, they will choose appropriate routes and actions to ensure their safety.

[0464] Step 8:

[0465] Users can use voice commands as needed to provide additional information to the terminal. This allows the system to improve its feedback in response to the user's specific requests.

[0466] (Example 1)

[0467] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0468] During disasters, it is difficult to quickly and accurately provide each individual user with the most suitable evacuation route. Furthermore, mechanisms for monitoring individuals' health status in real time and providing appropriate instructions are not yet fully established. As a result, users may not be able to evacuate properly during emergencies, potentially increasing their health risks. Additionally, obtaining useful information while maintaining data anonymity is another challenge.

[0469] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0470] In this invention, the server includes a data collection device that collects voice information, location information, and biometric information in real time; a knowledge processing means that analyzes the information on a local device and issues instructions based on the results of the analysis; an aggregate learning means that cooperates with external information to provide disaster information and user-optimized evacuation routes; and a means that distributes an updated information model from the aggregate learning means to the data collection device. This makes it possible to provide personalized evacuation instructions and health management to each user in real time, even during a disaster.

[0471] "Audio information" refers to data that represents the characteristics of sound in digital format, and is used to understand instructions and situations based on the user's voice.

[0472] "Location information" refers to geographical coordinate data, which is used to identify a user's current location and travel route.

[0473] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate and body temperature, and is collected for health management and risk assessment.

[0474] A "data collection device" is a device that includes sensors and equipment for acquiring voice information, location information, and biometric information, and is responsible for collecting information from users.

[0475] A "local device" is a computing device located near the user that analyzes collected data and generates instructions in real time.

[0476] "Knowledge processing means" refers to artificial intelligence technology that analyzes collected data to generate optimal instructions for the user.

[0477] "External information" refers to data obtained from outside the system, such as disaster information and traffic conditions, and is used to optimize evacuation routes.

[0478] "Aggregated learning methods" refer to machine learning techniques that integrate data obtained from multiple sources to perform overall optimization and analysis.

[0479] An "information model" refers to data that includes the calculation results of optimized evacuation routes and disaster information, generated by aggregate learning methods and provided to users.

[0480] This invention relates to an advanced information system for enhancing user safety during disasters, in which the server, terminal, and user components work in coordination. The embodiments thereof are described below.

[0481] Server configuration and operation

[0482] The server includes a central computing device for collecting and processing external information such as disaster information and traffic conditions. Using federated learning technology, the server aggregates anonymized voice information, location information, and biometric information from each terminal. This optimizes risk information and evacuation routes for the entire region and generates individually optimized information models for each user. These calculation results are periodically distributed to each terminal.

[0483] Terminal configuration and operation

[0484] The device is a portable computing device equipped with sensors to collect voice data, location data, and health data in real time. Each device has a locally running generative artificial intelligence model for analyzing the collected data. The device uses this generative AI model to perform analysis based on the model database received from the server. For example, if a user lives in an area prone to earthquakes, the device will monitor heart rate and body temperature data, and if the user's health condition is not good, it will immediately provide voice instructions such as "Take a deep breath and calm down."

[0485] User experience

[0486] Users simply wear the mobile device and require no special operation. In the event of a disaster, they can take safe actions based on instructions from the device, providing valuable information for decision-making during chaotic situations. For example, if a flood warning is issued, the device will provide specific instructions via voice, such as "Immediately evacuate to the nearest high ground." This allows users to evacuate quickly and ensure their safety.

[0487] Examples of specific cases and prompt statements

[0488] For example, if a user is at increased risk of heatstroke due to the summer heat, the device will provide a voice alert such as, "Please stay hydrated." An example of a prompt in this scenario might be, "If the user's body temperature exceeds 38 degrees Celsius, please generate appropriate feedback immediately."

[0489] This system allows users to receive real-time, optimized evacuation information and health instructions even during disasters, enhancing security while protecting personal information.

[0490] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0491] Step 1:

[0492] The server receives voice information, location information, and biometric information from each terminal in real time. It receives anonymized data as input and combines it with external disaster information and traffic data. By integrating the data, it analyzes risk information for the entire region and obtains output that calculates the optimal evacuation route. Specifically, federated learning is used to aggregate information and update the model.

[0493] Step 2:

[0494] The server distributes optimized evacuation route information generated from the analysis results to each terminal. It uses an optimal evacuation route and hazard information model as input and applies it to the terminal's generated AI model. User-specific model data is generated as output. Specifically, the server schedules data transmission at regular intervals.

[0495] Step 3:

[0496] The device performs analysis based on model data delivered from the server and real-time data it collects itself. It uses biometric information such as heart rate and body temperature, as well as location information, as input, and performs analysis via a generated AI model. The output is the generation of optimal instructions and warnings to provide to the user. Specifically, the AI ​​model makes predictions and judgments, and prepares to output voice instructions from the device as needed.

[0497] Step 4:

[0498] The device provides voice feedback to the user based on the analysis results. As input, it references the analyzed user status data and generates optimal voice instructions such as "Please move to a safe location." As output, it provides the user with intuitive and clear voice guidance. Specifically, the device uses speech synthesis technology to send immediate feedback to the user.

[0499] Step 5:

[0500] The user takes safe actions according to the voice guidance. The input is receiving voice instructions from the device and performing physical actions accordingly. The output is taking safe evacuation actions. Specifically, the user follows the device's instructions and begins moving to an appropriate evacuation location.

[0501] This entire process enables the system to provide individualized and rapid support to users even during disasters.

[0502] (Application Example 1)

[0503] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0504] During disasters, it is difficult for people to obtain accurate information necessary for quick and safe evacuation. Furthermore, there are very few systems that provide appropriate evacuation instructions in real time, tailored to the individual physiological state of each user. In addition, there is a need for means to facilitate evacuation actions by providing not only audio but also visual information.

[0505] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0506] In this invention, the server includes means equipped with sensors that collect voice data, location information, and physiological data in real time; generative machine learning means that analyze the data locally and provide notifications based on the generated results; distributed learning means that cooperate with external information to provide disaster information and optimal evacuation routes; and means equipped with a video display device for providing the user's visual information. As a result, the user can obtain information in real time that will allow them to evacuate quickly and safely in the event of a disaster, and receive appropriate instructions according to their physiological state.

[0507] "Audio data" refers to digital or analog data acquired as audio information.

[0508] "Location information" refers to information that indicates the geographical location of a moving object or user.

[0509] "Physiological data" refers to data that indicates a person's physiological state, such as heart rate and body temperature.

[0510] A "portable information processing device" is a device that is portable and has the function of processing data.

[0511] "Generative machine learning methods" are means of analyzing data and generating results using machine learning models.

[0512] A "central information processing system" is a device used to collect and centrally process large amounts of data.

[0513] A "distributed learning method" is a learning method that distributes information processing among multiple computers or devices.

[0514] "Model parameters" are variable elements in machine learning that are adjusted by the learning algorithm.

[0515] A "video display device" is a device that can display visual information.

[0516] An "alarm" is a signal emitted with sound or light to indicate an abnormality or danger.

[0517] "Voice instructions" are instructions or commands given using voice.

[0518] This invention is a system designed to assist users in safely evacuating during disasters. The server utilizes distributed learning methods to anonymize and receive voice data, location information, and physiological data collected from each terminal, and performs real-time analysis. Based on the calculation results, the server generates the optimal evacuation route for the user. This generation process utilizes disaster information integrated with external information.

[0519] The terminal is carried by the user as a portable information processing device and acquires necessary data using its built-in sensors. This data is locally analyzed by machine learning generation tools, and notifications are sent as needed. In addition, visual information is provided through a video display device to support smooth evacuation actions.

[0520] As a concrete example, in the event of an earthquake, the device monitors the user's physiological data and, if their heart rate exceeds normal, it will emit a voice message saying, "Please remain calm." Additionally, a real-time map of evacuation routes is displayed on the video display device, visually guiding the user to the optimal route.

[0521] The analysis is performed using a generative AI model, and accurate feedback tailored to the user's situation is provided instantly. The generative AI model is provided with the following prompts as input:

[0522] "The user's heart rate has exceeded normal levels. Please suggest ways to reduce stress. Also, please display the best evacuation route from the current location."

[0523] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0524] Step 1:

[0525] The device uses built-in sensors to collect voice data, location information, and physiological data in real time. This input data is prepared for analysis by a generative AI model within the device. Specifically, the heart rate sensor reads the user's pulse, and the location sensor acquires GPS information.

[0526] Step 2:

[0527] The device inputs the collected data into a generative AI model for analysis. This analysis estimates the user's emotional state from voice data and detects changes in physical condition from physiological data. The generative AI model combines this information to evaluate the user's health and emotional state. The output is a judgment on whether the user is experiencing stress.

[0528] Step 3:

[0529] The server utilizes distributed learning methods to receive anonymized health data and location information from each terminal and collect risk information for the entire region. The input consists of data sets from multiple terminals. The server integrates this data with external disaster information to assess the risk level for the entire region and optimize evacuation routes. The output generates risk assessments and optimal route information for each region.

[0530] Step 4:

[0531] The server distributes the generated evacuation route information to each terminal. This information is also used to update the generated AI model on each terminal. The terminal receives this information and continuously performs analysis based on the updated model. Specifically, the server transfers data asynchronously over the network.

[0532] Step 5:

[0533] The terminal provides users with visual and audible feedback based on evacuation route information received from the server and local data. The AI ​​model generates contacts based on the data, specifically by issuing voice alerts such as "Please stay calm" and displaying evacuation routes on the screen. This allows users to receive information through both sight and sound, enabling them to proceed with evacuation smoothly.

[0534] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0535] This invention is a system that primarily uses a portable computing device that collects voice data, location information, and health data, and incorporates an emotion engine to analyze the user's emotional state in real time and provide appropriate feedback. The emotion engine determines the user's emotions by analyzing the voice data and provides specific feedback when an abnormal emotional state is detected. This feedback functions as advice to promote safe evacuation actions and mental and physical stabilization.

[0536] Server operation

[0537] The server delivers emotion engine models to the terminals based on federated learning technology. The server periodically updates these models with new data to improve the accuracy of emotion analysis. At the same time, it calculates the optimal evacuation route for the user, taking disaster information and location information into consideration.

[0538] Terminal operation

[0539] The device analyzes the user's voice data using an emotion engine, identifying their emotions based on the tone and content of their speech. This data is processed in real time, providing immediate feedback based on their emotional state. For example, if the device determines that the user is in a state of anxiety, it will issue a voice command such as "Please calm down" and simultaneously encourage deep breathing to help them feel at ease. At the same time, it also analyzes the user's health data and location information to provide evacuation route instructions as needed.

[0540] User experience

[0541] Users receive emotionally responsive feedback from their devices, enabling them to act safely and calmly. This feedback system is designed to help manage stress during disasters and support rational decision-making. If a user experiences a major earthquake and their anxiety increases, the device can select and deliver reassuring words via voice, and also provide calming music or visual relaxation tools.

[0542] Thus, the present invention achieves a high level of personalization tailored to individual circumstances by continuously detecting the user's emotional state and providing appropriate support according to that state.

[0543] The following describes the processing flow.

[0544] Step 1:

[0545] The device collects voice data, location information, and health data in real time. Built-in sensors enable this, and the data is stored in a format that allows for immediate processing.

[0546] Step 2:

[0547] The device analyzes the collected audio data using an emotion engine. It analyzes parameters such as voice tone, speaking speed, and volume to identify the user's emotional state.

[0548] Step 3:

[0549] Based on the emotional state analyzed by the device, if negative emotions are detected, personalized feedback is provided to the user. For example, if anxiety or restlessness is detected, it will give a voice command such as "Please calm down" and suggest deep breathing as a relaxation method.

[0550] Step 4:

[0551] The device continues to analyze health data and location information, integrating them based on federated learning results. If an increase in heart rate or a change in body temperature is detected, it provides appropriate health advice and simultaneously suggests evacuation routes to avoid dangerous areas.

[0552] Step 5:

[0553] The server updates the overall learning model based on anonymous data collected from terminals. If new patterns or trends are discovered, the system will use them to improve evacuation routes and enhance the accuracy of emotion recognition.

[0554] Step 6:

[0555] The server sends updated model data to the device. This data is used to improve the accuracy of the device's generation AI and emotion engine, and to prepare for the next disaster.

[0556] Step 7:

[0557] By following voice instructions and vibration feedback from the device, users can receive emotional and health support and take safe actions. At appropriate times, they can also use voice commands to request additional information or adjust their emotions.

[0558] (Example 2)

[0559] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0560] In modern society, there is a growing need to respond quickly to natural disasters and changes in individual health conditions. Conventional information devices have struggled to accurately analyze users' emotional states and provide appropriate instructions and evacuation routes in real time. In particular, there is a need for methods that support mental stability and rational decision-making by providing personalized feedback.

[0561] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0562] In this invention, the server includes a portable information processing device equipped with a detection device that collects voice information, location information, and health information in real time; a generative intelligence means that analyzes the information locally and provides notifications by voice or vibration based on the generated results; and a distributed learning means that utilizes a central information processing device that cooperates with external information to provide disaster information and the user with the optimal evacuation route. This makes it possible to perform sentiment analysis on the user in real time and provide feedback and optimal evacuation instructions tailored to the individual situation.

[0563] "Voice information" refers to data collected digitally from the user's voice, which is used to analyze the content and tone of their speech.

[0564] "Location information" refers to geographical data used to determine a user's current location, and is primarily obtained through GPS technology.

[0565] "Health information" refers to physiological data such as the user's heart rate and physical activity level, and is used to evaluate their health status.

[0566] "Detection device" is a general term that includes sensor devices and technologies for acquiring voice information, location information, and health information in real time.

[0567] A "portable information processing device" refers to a portable electronic device equipped with the aforementioned detection device that can collect and perform initial analysis of data.

[0568] A "generative intelligence system" is a system equipped with artificial intelligence algorithms for performing speech analysis and feedback generation.

[0569] "Disaster information" refers to information about natural disasters such as earthquakes and floods, and is obtained from a database that is updated in real time.

[0570] A "central information processing system" refers to a centralized computer system that aggregates multiple data sources and performs advanced analysis.

[0571] "Distributed learning methods" are technologies used to efficiently update and optimize learning models across multiple information processing devices.

[0572] This invention is a system that collects voice information, location information, and health information in real time, analyzes the user's emotional state, and provides appropriate feedback. This system mainly consists of a portable information processing device, a central information processing device, and a generative intelligence means.

[0573] Terminal operation

[0574] The device is a portable information processing unit equipped with a microphone for acquiring voice information, GPS functionality for acquiring location information, and a sensor device (e.g., a smartwatch) for acquiring health information. This data is stored in the device's internal database in real time. The device is equipped with generative intelligence and uses natural language processing technology and speech analysis algorithms to analyze the user's emotional state from the collected voice information. Based on this analysis, the device uses a generative AI model to generate prompt sentences such as "How are you feeling today?" and provides feedback to the user in voice or text. Furthermore, it can instruct the user to take actions to reassure them as needed (e.g., prompting them to take a deep breath).

[0575] Server operation

[0576] The server integrates the collected data, and a central information processing unit calculates the optimal evacuation route based on disaster information. Disaster information is obtained from a real-time updated database and combined with location information to provide users with quick and appropriate evacuation instructions. The server utilizes distributed learning to constantly update the generative intelligence model, striving to improve accuracy. The updated model information is delivered to terminals and used to enhance the accuracy of sentiment analysis.

[0577] User experience

[0578] Users receive real-time feedback through their devices in the form of voice, visuals, and vibrations. For example, during a large-scale disaster, when a user's anxiety increases, the device provides reassuring voice instructions and displays a map showing evacuation routes. In this way, users can use the system to gain a sense of security and evacuate efficiently.

[0579] Thus, the present invention supports optimal actions tailored to the user's emotional state by providing highly personalized assistance.

[0580] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0581] Step 1:

[0582] The device collects voice information, location information, and health information from the user in real time. Voice information is acquired via a microphone, location information is measured by a GPS sensor, and health information is captured from sensor devices such as smartwatches. Voice information, location information, and health information are acquired as input data and stored in an internal database.

[0583] Step 2:

[0584] The terminal inputs the acquired voice information into a generative intelligence system for emotion analysis. A voice analysis algorithm is applied, and data calculations are performed to identify the emotional state from the user's speech content and tone. The result of the emotional state evaluation is obtained as output.

[0585] Step 3:

[0586] The device uses a generative AI model to generate a prompt for feedback based on the evaluation of the user's emotional state. For example, if the user is analyzed as being in a stressed state, the generative AI model will generate a prompt such as, "Try taking a deep breath to relax." The prompt is then output.

[0587] Step 4:

[0588] The server performs data calculations to determine safe evacuation routes based on collected location and disaster information. It retrieves the latest disaster information from an external disaster database and combines it with location information to identify the optimal route. The calculated evacuation route is then output.

[0589] Step 5:

[0590] The device notifies the user of the generated prompt and calculated evacuation route. It provides feedback to the user using voice instructions, text display, and vibration. It conveys reassuring action instructions and specific evacuation suggestions to the user.

[0591] Step 6:

[0592] The user takes concrete action based on feedback and evacuation instructions from the device. For example, they might take a deep breath as instructed by the device and move to a safe location along an evacuation route. As a result, the user can achieve a safer and more secure state of mind.

[0593] (Application Example 2)

[0594] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0595] In modern work environments, especially during long hours or complex tasks, the emotional state of workers can have a serious impact on productivity and safety. Conventional methods make it difficult to provide immediate feedback on workers' emotional and physical states, resulting in decreased work efficiency and an increased risk of accidents. This invention aims to solve these problems and improve safety and productivity in the work environment.

[0596] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0597] In this invention, the server includes an information processing device equipped with sensors that collect voice data, location information, and health data in real time; a generative artificial intelligence means that analyzes the data locally, provides notifications by voice or vibration based on the generated results, identifies the user's emotional state in the work environment, and provides feedback; a federated learning means that utilizes a central computing device that has the function of detecting abnormal emotional states for the safety of the work environment and presenting appropriate behavioral guidelines to the user; and a means for distributing updated model data from the federated learning means to the information processing device. This makes it possible to grasp the emotions and health status of workers in real time and maintain an efficient and safe work environment.

[0598] "Audio data" refers to digital information that records sounds and speech, and serves as fundamental data for analyzing the tone and content of a user's speech.

[0599] "Location information" refers to data that indicates the geographical location of a user or device, and is used to analyze the work environment and behavior.

[0600] "Health data" refers to data that indicates the user's physical condition, such as heart rate, stress level, and overall health.

[0601] An "information processing device" is a portable computer device used for data collection, analysis, and feedback.

[0602] "Generative artificial intelligence means" refers to a technology that analyzes data collected in real time, determines the user's emotional state, and provides appropriate feedback.

[0603] "Federated learning" is a learning method in which each device independently learns and updates its model without going through a central server, and then shares the results with other devices to improve the overall model.

[0604] A "central computing device" is a server or large computer device used to aggregate data, perform complex calculations, and manage overall information.

[0605] This invention is a system that uses a portable information processing device to collect voice data, location information, and health data in real time and analyze the user's emotional state. In particular, it aims to improve efficiency and safety in the work environment. The elements of this system are described below.

[0606] 1. Hardware

[0607] The terminal, which is an information processing device, is equipped with a voice sensor, a location information acquisition device, and a health monitoring device, and collects various data through them. This allows for real-time monitoring of the user's situation and enables the rapid provision of necessary feedback.

[0608] 2. Software

[0609] The device's generated artificial intelligence analyzes collected data to identify the user's emotional state. Based on this data, it then provides audio or visual feedback. The system is designed to continuously improve analysis accuracy by delivering the latest model to the device through federated learning.

[0610] 3. Data processing and calculations

[0611] The server acts as a central computing device, aggregating data sent from multiple terminals and updating the models deployed on each device using federated learning technology. In this process, each terminal learns the model independently, thus protecting user privacy while improving the overall accuracy of the system.

[0612] Specific example

[0613] Imagine a factory worker using an information processing device while working. This device detects that the worker is experiencing stress based on voice data and health data. The device immediately notifies the user via voice, "You should take a short break," prompting them to take a short rest. This type of feedback contributes to improved work efficiency and enhanced safety.

[0614] Example of a prompt

[0615] "Develop a system that utilizes a generative AI model to analyze user voice data, determine their emotional state in real time, and provide appropriate feedback. Clearly demonstrate how the device analyzes the data and presents optimal action guidelines."

[0616] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0617] Step 1:

[0618] The device uses a voice sensor, location information acquisition device, and health monitoring device to collect user voice data, location information, and health data in real time. In this process, the sensors capture each type of data, and the data is temporarily stored within the device. The input is various types of user data, and the output is the stored raw data.

[0619] Step 2:

[0620] The device analyzes collected audio data using generative artificial intelligence. It analyzes the tone and content of the audio data to perform data processing that identifies the user's emotional state. The input is stored raw audio data, and the output is analyzed emotional state information. Specifically, it extracts feature quantities from the audio waveform and inputs them into an emotion model to estimate emotions.

[0621] Step 3:

[0622] The device analyzes location information and health data to evaluate the user's work environment and health status. This includes a process that uses data calculations to detect abnormalities in the user's movements and physical condition. The inputs are location information and health data, and the outputs are indicators of the user's activity level and health status. Specifically, it analyzes changes in location and fluctuations in heart rate and compares them to standard values.

[0623] Step 4:

[0624] The server uses federated learning technology to integrate the data learned by each device and update the overall model. The updated model is then distributed back to the devices, improving the accuracy of the analysis. The input is the learning results from each device, and the output is the integrated, up-to-date generative AI model.

[0625] Step 5:

[0626] The device provides the user with audio or visual feedback based on the analysis results. This feedback is automatically generated based on the user's current emotional and health status. The input is the analysis results and the updated model, and the output is the content of the feedback. In a specific example, if stress is detected, a notification such as "Take a break" is issued.

[0627] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0628] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0629] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0630] [Fourth Embodiment]

[0631] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0632] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0633] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0634] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0635] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0636] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0637] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0638] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0639] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0640] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0641] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0642] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0643] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0644] The system according to the present invention consists of a server, terminals, and users, and supports the safe evacuation of users in the event of a disaster. Each terminal is equipped with sensors for acquiring voice data, location information, and health data, and this data is collected in real time. The data is analyzed by local generative artificial intelligence means, and appropriate feedback is provided according to the user's situation.

[0645] Server operation

[0646] The server uses federated learning technology to aggregate anonymized information from each terminal. Based on the collected data, it optimizes risk information and evacuation routes for the entire region. It also acquires external disaster information and traffic data in real time and combines and analyzes them to generate optimal evacuation route information for each individual. The server periodically distributes these calculation results to each terminal and updates the terminal's model data.

[0647] Terminal operation

[0648] The device performs analysis using generated AI based on model data received from the server. For example, if a user lives in an area affected by an earthquake, the device monitors the user's health based on acquired heart rate, body temperature, and location information. If the user's health deteriorates, the device immediately issues voice instructions such as "Take a deep breath" or "Move to a safe place." These instructions are linked to the optimal evacuation route based on the user's location information.

[0649] User experience

[0650] Under normal circumstances, users simply wear the device without requiring any special operation. In the event of a disaster, users receive intuitive feedback from the device and take safe actions based on the instructions. For example, when a flood warning is issued, the device will give a voice instruction to "immediately evacuate to higher ground," allowing users to quickly begin evacuation without confusion.

[0651] This system effectively supports user safety by providing personalized support that can respond immediately during disasters while protecting personal information.

[0652] The following describes the processing flow.

[0653] Step 1:

[0654] The device collects voice data, location information, and health data in real time. It continuously acquires data through sensors and stores it locally.

[0655] Step 2:

[0656] The data collected by the terminal is preprocessed in real time. This includes noise reduction and data cleaning, and conversion to a format suitable for analysis.

[0657] Step 3:

[0658] The device uses AI to analyze pre-processed data. Based on the analysis results, it evaluates the user's health status and risk level based on their current location.

[0659] Step 4:

[0660] The device provides feedback to the user based on the analysis results. This feedback is delivered via voice notifications and vibrations and includes recommended actions and evacuation route information.

[0661] Step 5:

[0662] The server performs federated learning based on anonymized data collected from each terminal. The overall model is updated to calculate the optimal evacuation route, taking into account disaster information and user-specific data.

[0663] Step 6:

[0664] The server delivers updated model data to the terminal. The terminal uses this data to prepare for the next analysis and to provide more accurate feedback.

[0665] Step 7:

[0666] Users will take necessary evacuation actions according to notifications on their devices. Based on voice notifications, they will choose appropriate routes and actions to ensure their safety.

[0667] Step 8:

[0668] Users can use voice commands as needed to provide additional information to the terminal. This allows the system to improve its feedback in response to the user's specific requests.

[0669] (Example 1)

[0670] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0671] During disasters, it is difficult to quickly and accurately provide each individual user with the most suitable evacuation route. Furthermore, mechanisms for monitoring individuals' health status in real time and providing appropriate instructions are not yet fully established. As a result, users may not be able to evacuate properly during emergencies, potentially increasing their health risks. Additionally, obtaining useful information while maintaining data anonymity is another challenge.

[0672] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0673] In this invention, the server includes a data collection device that collects voice information, location information, and biometric information in real time; a knowledge processing means that analyzes the information on a local device and issues instructions based on the results of the analysis; an aggregate learning means that cooperates with external information to provide disaster information and user-optimized evacuation routes; and a means that distributes an updated information model from the aggregate learning means to the data collection device. This makes it possible to provide personalized evacuation instructions and health management to each user in real time, even during a disaster.

[0674] "Audio information" refers to data that represents the characteristics of sound in digital format, and is used to understand instructions and situations based on the user's voice.

[0675] "Location information" refers to geographical coordinate data, which is used to identify a user's current location and travel route.

[0676] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate and body temperature, and is collected for health management and risk assessment.

[0677] A "data collection device" is a device that includes sensors and equipment for acquiring voice information, location information, and biometric information, and is responsible for collecting information from users.

[0678] A "local device" is a computing device located near the user that analyzes collected data and generates instructions in real time.

[0679] "Knowledge processing means" refers to artificial intelligence technology that analyzes collected data to generate optimal instructions for the user.

[0680] "External information" refers to data obtained from outside the system, such as disaster information and traffic conditions, and is used to optimize evacuation routes.

[0681] "Aggregated learning methods" refer to machine learning techniques that integrate data obtained from multiple sources to perform overall optimization and analysis.

[0682] An "information model" refers to data that includes the calculation results of optimized evacuation routes and disaster information, generated by aggregate learning methods and provided to users.

[0683] This invention relates to an advanced information system for enhancing user safety during disasters, in which the server, terminal, and user components work in coordination. The embodiments thereof are described below.

[0684] Server configuration and operation

[0685] The server includes a central computing device for collecting and processing external information such as disaster information and traffic conditions. Using federated learning technology, the server aggregates anonymized voice information, location information, and biometric information from each terminal. This optimizes risk information and evacuation routes for the entire region and generates individually optimized information models for each user. These calculation results are periodically distributed to each terminal.

[0686] Terminal configuration and operation

[0687] The device is a portable computing device equipped with sensors to collect voice data, location data, and health data in real time. Each device has a locally running generative artificial intelligence model for analyzing the collected data. The device uses this generative AI model to perform analysis based on the model database received from the server. For example, if a user lives in an area prone to earthquakes, the device will monitor heart rate and body temperature data, and if the user's health condition is not good, it will immediately provide voice instructions such as "Take a deep breath and calm down."

[0688] User experience

[0689] Users simply wear the mobile device and require no special operation. In the event of a disaster, they can take safe actions based on instructions from the device, providing valuable information for decision-making during chaotic situations. For example, if a flood warning is issued, the device will provide specific instructions via voice, such as "Immediately evacuate to the nearest high ground." This allows users to evacuate quickly and ensure their safety.

[0690] Examples of specific cases and prompt statements

[0691] For example, if a user is at increased risk of heatstroke due to the summer heat, the device will provide a voice alert such as, "Please stay hydrated." An example of a prompt in this scenario might be, "If the user's body temperature exceeds 38 degrees Celsius, please generate appropriate feedback immediately."

[0692] This system allows users to receive real-time, optimized evacuation information and health instructions even during disasters, enhancing security while protecting personal information.

[0693] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0694] Step 1:

[0695] The server receives voice information, location information, and biometric information from each terminal in real time. It receives anonymized data as input and combines it with external disaster information and traffic data. By integrating the data, it analyzes risk information for the entire region and obtains output that calculates the optimal evacuation route. Specifically, federated learning is used to aggregate information and update the model.

[0696] Step 2:

[0697] The server distributes optimized evacuation route information generated from the analysis results to each terminal. It uses an optimal evacuation route and hazard information model as input and applies it to the terminal's generated AI model. User-specific model data is generated as output. Specifically, the server schedules data transmission at regular intervals.

[0698] Step 3:

[0699] The device performs analysis based on model data delivered from the server and real-time data it collects itself. It uses biometric information such as heart rate and body temperature, as well as location information, as input, and performs analysis via a generated AI model. The output is the generation of optimal instructions and warnings to provide to the user. Specifically, the AI ​​model makes predictions and judgments, and prepares to output voice instructions from the device as needed.

[0700] Step 4:

[0701] The device provides voice feedback to the user based on the analysis results. As input, it references the analyzed user status data and generates optimal voice instructions such as "Please move to a safe location." As output, it provides the user with intuitive and clear voice guidance. Specifically, the device uses speech synthesis technology to send immediate feedback to the user.

[0702] Step 5:

[0703] The user takes safe actions according to the voice guidance. The input is receiving voice instructions from the device and performing physical actions accordingly. The output is taking safe evacuation actions. Specifically, the user follows the device's instructions and begins moving to an appropriate evacuation location.

[0704] This entire process enables the system to provide individualized and rapid support to users even during disasters.

[0705] (Application Example 1)

[0706] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0707] During disasters, it is difficult for people to obtain accurate information necessary for quick and safe evacuation. Furthermore, there are very few systems that provide appropriate evacuation instructions in real time, tailored to the individual physiological state of each user. In addition, there is a need for means to facilitate evacuation actions by providing not only audio but also visual information.

[0708] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0709] In this invention, the server includes means equipped with sensors that collect voice data, location information, and physiological data in real time; generative machine learning means that analyze the data locally and provide notifications based on the generated results; distributed learning means that cooperate with external information to provide disaster information and optimal evacuation routes; and means equipped with a video display device for providing the user's visual information. As a result, the user can obtain information in real time that will allow them to evacuate quickly and safely in the event of a disaster, and receive appropriate instructions according to their physiological state.

[0710] "Audio data" refers to digital or analog data acquired as audio information.

[0711] "Location information" refers to information that indicates the geographical location of a moving object or user.

[0712] "Physiological data" refers to data that indicates a person's physiological state, such as heart rate and body temperature.

[0713] A "portable information processing device" is a device that is portable and has the function of processing data.

[0714] "Generative machine learning methods" are means of analyzing data and generating results using machine learning models.

[0715] A "central information processing system" is a device used to collect and centrally process large amounts of data.

[0716] A "distributed learning method" is a learning method that distributes information processing among multiple computers or devices.

[0717] "Model parameters" are variable elements in machine learning that are adjusted by the learning algorithm.

[0718] A "video display device" is a device that can display visual information.

[0719] An "alarm" is a signal emitted with sound or light to indicate an abnormality or danger.

[0720] "Voice instructions" are instructions or commands given using voice.

[0721] This invention is a system designed to assist users in safely evacuating during disasters. The server utilizes distributed learning methods to anonymize and receive voice data, location information, and physiological data collected from each terminal, and performs real-time analysis. Based on the calculation results, the server generates the optimal evacuation route for the user. This generation process utilizes disaster information integrated with external information.

[0722] The terminal is carried by the user as a portable information processing device and acquires necessary data using its built-in sensors. This data is locally analyzed by machine learning generation tools, and notifications are sent as needed. In addition, visual information is provided through a video display device to support smooth evacuation actions.

[0723] As a concrete example, in the event of an earthquake, the device monitors the user's physiological data and, if their heart rate exceeds normal, it will emit a voice message saying, "Please remain calm." Additionally, a real-time map of evacuation routes is displayed on the video display device, visually guiding the user to the optimal route.

[0724] The analysis is performed using a generative AI model, and accurate feedback tailored to the user's situation is provided instantly. The generative AI model is provided with the following prompts as input:

[0725] "The user's heart rate has exceeded normal levels. Please suggest ways to reduce stress. Also, please display the best evacuation route from the current location."

[0726] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0727] Step 1:

[0728] The device uses built-in sensors to collect voice data, location information, and physiological data in real time. This input data is prepared for analysis by a generative AI model within the device. Specifically, the heart rate sensor reads the user's pulse, and the location sensor acquires GPS information.

[0729] Step 2:

[0730] The device inputs the collected data into a generative AI model for analysis. This analysis estimates the user's emotional state from voice data and detects changes in physical condition from physiological data. The generative AI model combines this information to evaluate the user's health and emotional state. The output is a judgment on whether the user is experiencing stress.

[0731] Step 3:

[0732] The server utilizes distributed learning methods to receive anonymized health data and location information from each terminal and collect risk information for the entire region. The input consists of data sets from multiple terminals. The server integrates this data with external disaster information to assess the risk level for the entire region and optimize evacuation routes. The output generates risk assessments and optimal route information for each region.

[0733] Step 4:

[0734] The server distributes the generated evacuation route information to each terminal. This information is also used to update the generated AI model on each terminal. The terminal receives this information and continuously performs analysis based on the updated model. Specifically, the server transfers data asynchronously over the network.

[0735] Step 5:

[0736] The terminal provides users with visual and audible feedback based on evacuation route information received from the server and local data. The AI ​​model generates contacts based on the data, specifically by issuing voice alerts such as "Please stay calm" and displaying evacuation routes on the screen. This allows users to receive information through both sight and sound, enabling them to proceed with evacuation smoothly.

[0737] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0738] This invention is a system that primarily uses a portable computing device that collects voice data, location information, and health data, and incorporates an emotion engine to analyze the user's emotional state in real time and provide appropriate feedback. The emotion engine determines the user's emotions by analyzing the voice data and provides specific feedback when an abnormal emotional state is detected. This feedback functions as advice to promote safe evacuation actions and mental and physical stabilization.

[0739] Server operation

[0740] The server delivers emotion engine models to the terminals based on federated learning technology. The server periodically updates these models with new data to improve the accuracy of emotion analysis. At the same time, it calculates the optimal evacuation route for the user, taking disaster information and location information into consideration.

[0741] Terminal operation

[0742] The device analyzes the user's voice data using an emotion engine, identifying their emotions based on the tone and content of their speech. This data is processed in real time, providing immediate feedback based on their emotional state. For example, if the device determines that the user is in a state of anxiety, it will issue a voice command such as "Please calm down" and simultaneously encourage deep breathing to help them feel at ease. At the same time, it also analyzes the user's health data and location information to provide evacuation route instructions as needed.

[0743] User experience

[0744] Users receive emotionally responsive feedback from their devices, enabling them to act safely and calmly. This feedback system is designed to help manage stress during disasters and support rational decision-making. If a user experiences a major earthquake and their anxiety increases, the device can select and deliver reassuring words via voice, and also provide calming music or visual relaxation tools.

[0745] Thus, the present invention achieves a high level of personalization tailored to individual circumstances by continuously detecting the user's emotional state and providing appropriate support according to that state.

[0746] The following describes the processing flow.

[0747] Step 1:

[0748] The device collects voice data, location information, and health data in real time. Built-in sensors enable this, and the data is stored in a format that allows for immediate processing.

[0749] Step 2:

[0750] The device analyzes the collected audio data using an emotion engine. It analyzes parameters such as voice tone, speaking speed, and volume to identify the user's emotional state.

[0751] Step 3:

[0752] Based on the emotional state analyzed by the device, if negative emotions are detected, personalized feedback is provided to the user. For example, if anxiety or restlessness is detected, it will give a voice command such as "Please calm down" and suggest deep breathing as a relaxation method.

[0753] Step 4:

[0754] The device continues to analyze health data and location information, integrating them based on federated learning results. If an increase in heart rate or a change in body temperature is detected, it provides appropriate health advice and simultaneously suggests evacuation routes to avoid dangerous areas.

[0755] Step 5:

[0756] The server updates the overall learning model based on anonymous data collected from terminals. If new patterns or trends are discovered, the system will use them to improve evacuation routes and enhance the accuracy of emotion recognition.

[0757] Step 6:

[0758] The server sends updated model data to the device. This data is used to improve the accuracy of the device's generation AI and emotion engine, and to prepare for the next disaster.

[0759] Step 7:

[0760] By following voice instructions and vibration feedback from the device, users can receive emotional and health support and take safe actions. At appropriate times, they can also use voice commands to request additional information or adjust their emotions.

[0761] (Example 2)

[0762] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0763] In modern society, there is a growing need to respond quickly to natural disasters and changes in individual health conditions. Conventional information devices have struggled to accurately analyze users' emotional states and provide appropriate instructions and evacuation routes in real time. In particular, there is a need for methods that support mental stability and rational decision-making by providing personalized feedback.

[0764] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0765] In this invention, the server includes a portable information processing device equipped with a detection device that collects voice information, location information, and health information in real time; a generative intelligence means that analyzes the information locally and provides notifications by voice or vibration based on the generated results; and a distributed learning means that utilizes a central information processing device that cooperates with external information to provide disaster information and the user with the optimal evacuation route. This makes it possible to perform sentiment analysis on the user in real time and provide feedback and optimal evacuation instructions tailored to the individual situation.

[0766] "Voice information" refers to data collected digitally from the user's voice, which is used to analyze the content and tone of their speech.

[0767] "Location information" refers to geographical data used to determine a user's current location, and is primarily obtained through GPS technology.

[0768] "Health information" refers to physiological data such as the user's heart rate and physical activity level, and is used to evaluate their health status.

[0769] "Detection device" is a general term that includes sensor devices and technologies for acquiring voice information, location information, and health information in real time.

[0770] A "portable information processing device" refers to a portable electronic device equipped with the aforementioned detection device that can collect and perform initial analysis of data.

[0771] A "generative intelligence system" is a system equipped with artificial intelligence algorithms for performing speech analysis and feedback generation.

[0772] "Disaster information" refers to information about natural disasters such as earthquakes and floods, and is obtained from a database that is updated in real time.

[0773] A "central information processing system" refers to a centralized computer system that aggregates multiple data sources and performs advanced analysis.

[0774] "Distributed learning methods" are technologies used to efficiently update and optimize learning models across multiple information processing devices.

[0775] This invention is a system that collects voice information, location information, and health information in real time, analyzes the user's emotional state, and provides appropriate feedback. This system mainly consists of a portable information processing device, a central information processing device, and a generative intelligence means.

[0776] Terminal operation

[0777] The device is a portable information processing unit equipped with a microphone for acquiring voice information, GPS functionality for acquiring location information, and a sensor device (e.g., a smartwatch) for acquiring health information. This data is stored in the device's internal database in real time. The device is equipped with generative intelligence and uses natural language processing technology and speech analysis algorithms to analyze the user's emotional state from the collected voice information. Based on this analysis, the device uses a generative AI model to generate prompt sentences such as "How are you feeling today?" and provides feedback to the user in voice or text. Furthermore, it can instruct the user to take actions to reassure them as needed (e.g., prompting them to take a deep breath).

[0778] Server operation

[0779] The server integrates the collected data, and a central information processing unit calculates the optimal evacuation route based on disaster information. Disaster information is obtained from a real-time updated database and combined with location information to provide users with quick and appropriate evacuation instructions. The server utilizes distributed learning to constantly update the generative intelligence model, striving to improve accuracy. The updated model information is delivered to terminals and used to enhance the accuracy of sentiment analysis.

[0780] User experience

[0781] Users receive real-time feedback through their devices in the form of voice, visuals, and vibrations. For example, during a large-scale disaster, when a user's anxiety increases, the device provides reassuring voice instructions and displays a map showing evacuation routes. In this way, users can use the system to gain a sense of security and evacuate efficiently.

[0782] Thus, the present invention supports optimal actions tailored to the user's emotional state by providing highly personalized assistance.

[0783] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0784] Step 1:

[0785] The device collects voice information, location information, and health information from the user in real time. Voice information is acquired via a microphone, location information is measured by a GPS sensor, and health information is captured from sensor devices such as smartwatches. Voice information, location information, and health information are acquired as input data and stored in an internal database.

[0786] Step 2:

[0787] The terminal inputs the acquired voice information into a generative intelligence system for emotion analysis. A voice analysis algorithm is applied, and data calculations are performed to identify the emotional state from the user's speech content and tone. The result of the emotional state evaluation is obtained as output.

[0788] Step 3:

[0789] The device uses a generative AI model to generate a prompt for feedback based on the evaluation of the user's emotional state. For example, if the user is analyzed as being in a stressed state, the generative AI model will generate a prompt such as, "Try taking a deep breath to relax." The prompt is then output.

[0790] Step 4:

[0791] The server performs data calculations to determine safe evacuation routes based on collected location and disaster information. It retrieves the latest disaster information from an external disaster database and combines it with location information to identify the optimal route. The calculated evacuation route is then output.

[0792] Step 5:

[0793] The device notifies the user of the generated prompt and calculated evacuation route. It provides feedback to the user using voice instructions, text display, and vibration. It conveys reassuring action instructions and specific evacuation suggestions to the user.

[0794] Step 6:

[0795] The user takes concrete action based on feedback and evacuation instructions from the device. For example, they might take a deep breath as instructed by the device and move to a safe location along an evacuation route. As a result, the user can achieve a safer and more secure state of mind.

[0796] (Application Example 2)

[0797] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0798] In modern work environments, especially during long hours or complex tasks, the emotional state of workers can have a serious impact on productivity and safety. Conventional methods make it difficult to provide immediate feedback on workers' emotional and physical states, resulting in decreased work efficiency and an increased risk of accidents. This invention aims to solve these problems and improve safety and productivity in the work environment.

[0799] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0800] In this invention, the server includes an information processing device equipped with sensors that collect voice data, location information, and health data in real time; a generative artificial intelligence means that analyzes the data locally, provides notifications by voice or vibration based on the generated results, identifies the user's emotional state in the work environment, and provides feedback; a federated learning means that utilizes a central computing device that has the function of detecting abnormal emotional states for the safety of the work environment and presenting appropriate behavioral guidelines to the user; and a means for distributing updated model data from the federated learning means to the information processing device. This makes it possible to grasp the emotions and health status of workers in real time and maintain an efficient and safe work environment.

[0801] "Audio data" refers to digital information that records sounds and speech, and serves as fundamental data for analyzing the tone and content of a user's speech.

[0802] "Location information" refers to data that indicates the geographical location of a user or device, and is used to analyze the work environment and behavior.

[0803] "Health data" refers to data that indicates the user's physical condition, such as heart rate, stress level, and overall health.

[0804] An "information processing device" is a portable computer device used for data collection, analysis, and feedback.

[0805] "Generative artificial intelligence means" refers to a technology that analyzes data collected in real time, determines the user's emotional state, and provides appropriate feedback.

[0806] "Federated learning" is a learning method in which each device independently learns and updates its model without going through a central server, and then shares the results with other devices to improve the overall model.

[0807] A "central computing device" is a server or large computer device used to aggregate data, perform complex calculations, and manage overall information.

[0808] This invention is a system that uses a portable information processing device to collect voice data, location information, and health data in real time and analyze the user's emotional state. In particular, it aims to improve efficiency and safety in the work environment. The elements of this system are described below.

[0809] 1. Hardware

[0810] The terminal, which is an information processing device, is equipped with a voice sensor, a location information acquisition device, and a health monitoring device, and collects various data through them. This allows for real-time monitoring of the user's situation and enables the rapid provision of necessary feedback.

[0811] 2. Software

[0812] The device's generated artificial intelligence analyzes collected data to identify the user's emotional state. Based on this data, it then provides audio or visual feedback. The system is designed to continuously improve analysis accuracy by delivering the latest model to the device through federated learning.

[0813] 3. Data processing and calculations

[0814] The server acts as a central computing device, aggregating data sent from multiple terminals and updating the models deployed on each device using federated learning technology. In this process, each terminal learns the model independently, thus protecting user privacy while improving the overall accuracy of the system.

[0815] Specific example

[0816] Imagine a factory worker using an information processing device while working. This device detects that the worker is experiencing stress based on voice data and health data. The device immediately notifies the user via voice, "You should take a short break," prompting them to take a short rest. This type of feedback contributes to improved work efficiency and enhanced safety.

[0817] Example of a prompt

[0818] "Develop a system that utilizes a generative AI model to analyze user voice data, determine their emotional state in real time, and provide appropriate feedback. Clearly demonstrate how the device analyzes the data and presents optimal action guidelines."

[0819] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0820] Step 1:

[0821] The device uses a voice sensor, location information acquisition device, and health monitoring device to collect user voice data, location information, and health data in real time. In this process, the sensors capture each type of data, and the data is temporarily stored within the device. The input is various types of user data, and the output is the stored raw data.

[0822] Step 2:

[0823] The device analyzes collected audio data using generative artificial intelligence. It analyzes the tone and content of the audio data to perform data processing that identifies the user's emotional state. The input is stored raw audio data, and the output is analyzed emotional state information. Specifically, it extracts feature quantities from the audio waveform and inputs them into an emotion model to estimate emotions.

[0824] Step 3:

[0825] The device analyzes location information and health data to evaluate the user's work environment and health status. This includes a process that uses data calculations to detect abnormalities in the user's movements and physical condition. The inputs are location information and health data, and the outputs are indicators of the user's activity level and health status. Specifically, it analyzes changes in location and fluctuations in heart rate and compares them to standard values.

[0826] Step 4:

[0827] The server uses federated learning technology to integrate the data learned by each device and update the overall model. The updated model is then distributed back to the devices, improving the accuracy of the analysis. The input is the learning results from each device, and the output is the integrated, up-to-date generative AI model.

[0828] Step 5:

[0829] The device provides the user with audio or visual feedback based on the analysis results. This feedback is automatically generated based on the user's current emotional and health status. The input is the analysis results and the updated model, and the output is the content of the feedback. In a specific example, if stress is detected, a notification such as "Take a break" is issued.

[0830] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0831] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0832] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0833] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0834] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0835] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0836] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0837] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0838] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0839] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0840] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0841] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0842] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0843] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0844] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0845] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0846] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0847] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0848] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0849] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0850] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0851] The following is further disclosed regarding the embodiments described above.

[0852] (Claim 1)

[0853] A portable computing device equipped with sensors that collect voice data, location information, and health data in real time,

[0854] A generative artificial intelligence means that analyzes the aforementioned data locally and provides notification by voice or vibration based on the generated results,

[0855] A federated learning method utilizing a central computing device that collaborates with external data to provide disaster information and the optimal evacuation route for users,

[0856] Means for distributing updated model data from the federated learning means to the portable computing device,

[0857] A system that includes this.

[0858] (Claim 2)

[0859] The system according to claim 1, which issues an emergency alert when health data exceeds a certain threshold.

[0860] (Claim 3)

[0861] The system according to claim 1, wherein the generating artificial intelligence means has a function to provide feedback in response to a user's voice command.

[0862] "Example 1"

[0863] (Claim 1)

[0864] A data collection device that collects voice information, location information, and biometric information in real time,

[0865] A knowledge processing means that analyzes the aforementioned information using a local device and issues instructions based on the results of the analysis,

[0866] An aggregate learning method for providing disaster information and user-optimized evacuation routes in cooperation with external information,

[0867] A means for distributing the updated information model from the aggregate learning means to the data collection device,

[0868] A system that includes this.

[0869] (Claim 2)

[0870] The system according to claim 1, which issues a warning when biological information exceeds a set threshold value.

[0871] (Claim 3)

[0872] The system according to claim 1, wherein the knowledge processing means has a function of providing information in response to a user's voice instructions.

[0873] "Application Example 1"

[0874] (Claim 1)

[0875] A portable information processing device equipped with sensors that collect voice data, location information, and physiological data in real time,

[0876] A generative machine learning means that analyzes the aforementioned data locally and provides notification by voice or vibration based on the generated results,

[0877] A distributed learning method utilizing a central information processing system that links with external information to provide disaster information and the most suitable evacuation routes for users,

[0878] A means for distributing the updated model parameters from the distributed learning means to the portable information processing device,

[0879] A means equipped with a video display device for providing visual information to the user,

[0880] A system that includes this.

[0881] (Claim 2)

[0882] The system according to claim 1, which issues an alarm when physiological data exceeds a certain standard.

[0883] (Claim 3)

[0884] The system according to claim 1, wherein the generation machine learning means has the function of providing a response in accordance with the user's voice instructions.

[0885] "Example 2 of combining an emotion engine"

[0886] (Claim 1)

[0887] A portable information processing device equipped with a detection device that collects voice information, location information, and health information in real time,

[0888] A generative intelligence means that locally analyzes the aforementioned information and provides notification by voice or vibration based on the generated results,

[0889] A distributed learning method utilizing a central information processing unit that collaborates with external information to provide disaster information and the optimal evacuation route for users,

[0890] A means for distributing the updated model information from the distributed learning means to the portable information processing device,

[0891] The aforementioned generative intelligence means includes means for analyzing the user's emotional state and providing appropriate feedback,

[0892] A system that includes this.

[0893] (Claim 2)

[0894] The system according to claim 1, which issues an emergency warning when health information exceeds a certain standard.

[0895] (Claim 3)

[0896] The system according to claim 1, wherein the generating intelligence means has a function to provide feedback in response to a user's voice command.

[0897] "Application example 2 when combining with an emotional engine"

[0898] (Claim 1)

[0899] An information processing device equipped with sensors that collect voice data, location information, and health data in real time,

[0900] A generative artificial intelligence means that analyzes the aforementioned data locally, provides notifications via voice or vibration based on the generated results, identifies the user's emotional state in the work environment, and provides feedback;

[0901] A federated learning method utilizing a central computing device that has the function of detecting abnormal emotional states for the safety of the work environment and providing appropriate behavioral guidelines to the user,

[0902] means for distributing the updated model data from the federated learning means to the information processing device,

[0903] A system that includes this.

[0904] (Claim 2)

[0905] The system according to claim 1, which issues an emergency alert considering work efficiency and safety when a user's health data exceeds a certain threshold.

[0906] (Claim 3)

[0907] The system according to claim 1, wherein the generating artificial intelligence means has a function to provide feedback for improving work efficiency based on the user's emotional state. [Explanation of symbols]

[0908] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A portable computing device equipped with sensors that collect voice data, location information, and health data in real time, A generative artificial intelligence means that analyzes the aforementioned data locally and provides notification by voice or vibration based on the generated results, A federated learning method utilizing a central computing device that collaborates with external data to provide disaster information and the optimal evacuation route for users, Means for distributing updated model data from the federated learning means to the portable computing device, A system that includes this.

2. The system according to claim 1, which issues an emergency alert when health data exceeds a certain threshold.

3. The system according to claim 1, wherein the generating artificial intelligence means has a function to provide feedback in response to a user's voice command.

Images