System

Abstract

A system is provided.SOLUTION: The system includes a voice input means, a means for analyzing voice data and converting them into text data, a means for analyzing a situation and generating an instruction for first aid, a means for encoding the generated instruction as voice data, a means for transmitting the voice instruction data to a user terminal, and a means for reproducing the encoded voice instruction data on the user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In an emergency, many lives can be saved if appropriate first aid is given within one minute, but ordinary people often lack knowledge of first aid and often fail to provide appropriate treatment. Also, since it takes time for an ambulance to arrive, there is a risk that many lives will be lost in the meantime. Therefore, there is a need for a support system that enables first aid to be given quickly and accurately. [Means for solving the problem]

[0005] The present invention provides a system including a voice input means, a means for analyzing voice data and converting it into text data, a means for analyzing a situation and generating first aid instructions, a means for encoding the generated instructions as voice data, a means for transmitting the voice instruction data to a user terminal, and a means for playing the encoded voice instruction data on the user terminal. This system enables a user to quickly and appropriately receive first aid instructions in an emergency, enabling necessary treatment to be performed before the arrival of an ambulance. Furthermore, by including a means for monitoring the user's actions in accordance with the first aid instructions and generating additional instructions based on the actions, the system can respond optimally according to the situation. Furthermore, when the received voice data relates to multiple situations, the system can analyze the situation and identify and instruct the first aid with the highest priority.

[0006] The "voice input means" is a device or function that acquires the user's speech as digital voice data.

[0007] The "means for analyzing voice data and converting it into text data" refers to the processing power or software that analyzes the acquired voice data and converts it into corresponding text format data.

[0008] The "means for analyzing the situation and generating emergency instructions" is a function or program that analyzes the details of the emergency situation that has occurred based on text data, determines appropriate emergency measures, and generates those instructions.

[0009] "Means for encoding the generated instructions as audio data" is processing power or software that converts the generated first aid instructions into audio format.

[0010] The "means for transmitting voice instruction data to a user terminal" is a function or device for transmitting encoded voice data to a terminal used by a user.

[0011] The "means for reproducing the voice instruction data encoded on the user terminal" is a function or program that reproduces the voice data received on the user terminal and conveys the instruction content to the user as voice.

[0012] The "means for monitoring the user's actions in accordance with the first aid instructions and generating additional instructions based thereon" refers to a function or program that monitors the first aid actions taken by the user and generates any necessary additional instructions based on the results.

[0013] "Means for analyzing voice data relating to multiple situations and identifying and instructing first aid measures with the highest priority" refers to a function or software that analyzes multiple emergency situations from received voice data, identifies the measure that requires the most urgent response, and generates instructions for that measure. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention relates to a system that uses voice recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input.

[0036] First, the user launches the "RescueVoice" application on their smartphone. The system starts up when the user says "Help me!" The user's device captures the voice using the built-in microphone and generates voice data. The generated voice data is immediately sent to the server.

[0037] On the server side, a voice recognition engine analyzes the received voice data and converts it into text data. Based on this text data, an AI model analyzes the situation. For example, if someone says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[0038] Next, the first aid instructions generated by the server are encoded as voice data. The encoded voice data is immediately sent to the user's device. The user's device plays back the received voice data and delivers it to the user. The user begins first aid according to the instructions. For example, the instructions might be, "Start chest compressions. Place your hands together in the center of the chest and compress at a rate of 100-120 times per minute."

[0039] The system can also monitor the progress of first aid and provide additional instructions as needed. For example, if a user says, "I went to get the AED," new voice data is sent to the server, which immediately analyzes it. As a result, additional instructions such as "Prepare to use the AED. Attach the pads to the patient's chest" are generated and encoded as voice data. The user device plays back the voice data and provides the user with additional instructions.

[0040] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice and provides additional instructions as needed. If an AED is available, the system also provides specific instructions on how to use it. As a result, the chances of appropriate first aid being administered and saving a life are significantly increased before an ambulance arrives.

[0041] This system allows even ordinary users without specialized knowledge to provide first aid quickly and accurately, enabling important life-saving measures to be taken before an ambulance arrives.

[0042] The processing flow will be explained below.

[0043] Step 1:

[0044] The user launches the "RescueVoice" app on their smartphone, which puts the app into a voice input standby state.

[0045] Step 2:

[0046] The user says "Help!" The device uses the built-in microphone to capture the user's voice and generate digital audio data.

[0047] Step 3:

[0048] The audio data generated by the device is sent to the server using the secure HTTP(S) protocol.

[0049] Step 4:

[0050] The server passes the received voice data to a speech recognition engine, which converts the voice data into text data. This speech recognition engine uses a highly accurate generative AI model.

[0051] Step 5:

[0052] The server passes the text data to an AI model, which analyzes the situation. For example, it analyzes text such as "My friend has collapsed" and determines whether the situation is urgent.

[0053] Step 6:

[0054] The server generates appropriate first aid instructions based on the analysis results. For example, if cardiac arrest is suspected, it generates instructions such as, "Start chest compressions immediately. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[0055] Step 7:

[0056] The server encodes the generated instructions as voice data and transmits the voice data to the user terminal.

[0057] Step 8:

[0058] The device plays the received voice data and delivers it to the user, who then follows the instructions to begin providing first aid.

[0059] Step 9:

[0060] The device continuously monitors the user's voice input. For example, if the user says, "Help isn't here yet," the device retransmits the voice to the server.

[0061] Step 10:

[0062] The server receives the new voice data, analyzes it again, and generates additional first aid instructions based on the new situation, such as "Continue chest compressions and use an AED if one is available."

[0063] Step 11:

[0064] The server then transmits the encoded audio data back to the user terminal, which then plays the audio back to the user.

[0065] Step 12:

[0066] The user follows additional instructions to administer any necessary first aid, such as attaching pads to the patient's chest to use an AED.

[0067] Through this series of processing steps, the user receives appropriate first aid instructions in real time, enabling them to take effective life-saving measures until an ambulance arrives.

[0068] Example 1

[0069] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0070] In modern society, emergencies can occur due to sudden illness or accidents. Responding to these emergencies requires prompt and appropriate first aid, but the general public often does not know how to provide it. To solve this problem, a system is needed that allows anyone to provide first aid easily, even without specialized knowledge. It also needs to be able to respond to multiple situations and determine the appropriate course of action.

[0071] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0072] In this invention, the server includes means for analyzing voice data and converting it into text data, means for analyzing the situation based on the text data and generating first aid instructions, and means for encoding the generated first aid instructions as voice data. This allows a user to receive appropriate first aid instructions in real time simply by calling for help by voice. The system can also monitor the progress of the first aid and provide additional instructions as needed, allowing it to respond to multiple situations and provide optimal treatment.

[0073] The "voice input means" is a device that has the function of taking in the user's voice and generating it as digital voice data.

[0074] The "means for generating audio data and transmitting it to a server" is a device or software that has the function of transmitting captured audio data to a server using an appropriate communication protocol.

[0075] The "means for analyzing voice data and converting it into text data" refers to a software algorithm or device for analyzing received voice data and generating the content of the voice data as text data.

[0076] "Means for analyzing a situation based on text data and generating first aid instructions" refers to software or a device that uses an AI model or rule-based system to analyze a situation based on text data and issue appropriate first aid instructions.

[0077] The "means for encoding the generated first aid instructions as audio data" is software or a device for converting the text-based first aid instructions back into audio data.

[0078] The "means for transmitting encoded voice data to a user terminal" is a device or software that has the function of transmitting encoded voice data to a user terminal using an appropriate communication protocol.

[0079] The "means for reproducing audio data received on a user terminal" refers to a device or software having the function of reproducing audio data received by a user terminal and providing audio instructions to the user.

[0080] This invention is a system that supports first aid in emergencies using voice recognition and an AI model. The system includes a voice input unit, a voice analysis unit, a situation analysis unit, a first aid instruction generation unit, and a voice output unit.

[0081] System configuration and operation

[0082] Hardware and Software Configuration

[0083] User terminal: A mobile information terminal such as a smartphone. It uses a built-in microphone to capture voice input and generate voice data. It also has a speaker to play back voice instruction data from the server.

[0084] Server: A powerful computer system that includes a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze and convert audio data into text, and a generative AI model (e.g., OpenAI's GPT-4) to analyze the situation and generate first aid instructions.

[0085] Program processing explanation

[0086] Voice Recognition

[0087] In an emergency, a user launches the "RescueVoice" application on their smartphone and utters the command "Help!" The user's device's built-in microphone captures this voice and converts it into digital audio data, which is then sent to a server via the Internet.

[0088] Analysis of audio data

[0089] The server uses a speech recognition engine to analyze the received voice data and convert it into text data. For example, if a user says, "Help me, my friend has fallen down," the voice data is converted into text data.

[0090] Analyzing the situation and generating first aid instructions

[0091] The converted text data is then analyzed by a generative AI model to understand the situation. The AI ​​model then references past data and medical knowledge to generate the most appropriate first aid instructions. For example, it might generate specific instructions such as, "If cardiac arrest is suspected, begin chest compressions."

[0092] Encoding and transmitting first aid instructions as audio data

[0093] The server encodes the generated emergency treatment instructions as voice data, which is then transmitted to the user terminal via the Internet.

[0094] Playing voice instructions

[0095] The user device plays the received voice data and conveys instructions to the user, such as "Please begin chest compressions. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[0096] Monitor progress and generate further instructions

[0097] The system continuously monitors the user's progress in administering first aid and generates additional instructions as needed. For example, if the user says, "I went to get the AED," new voice data is sent to the server for analysis. As a result, additional instructions are generated and encoded as voice data, such as, "Prepare the AED. Attach the pads to the patient's chest."

[0098] Examples of concrete examples and prompts

[0099] Specific examples

[0100] If a friend suddenly collapses:

[0101] 1. User: Launch the "RescueVoice" application on their smartphone.

[0102] 2. User: Says "Help!"

[0103] 3. Device: Captures audio with the built-in microphone and sends the audio data to the server.

[0104] 4. Server: The speech recognition engine converts the speech into text, and the AI ​​model determines the situation: "My friend has collapsed."

[0105] 5. Server: Generates the instruction "Please begin chest compressions" and encodes it as audio data.

[0106] 6. Terminal: Plays the received audio data and delivers it to the user.

[0107] 7. User: Begin chest compressions as instructed.

[0108] Prompt Sentence Examples

[0109] "If a user is in an emergency and needs first aid, design a system that provides appropriate instructions in real time based on voice input."

[0110] "Imagine a scenario in which a user uses a voice recognition system to provide first aid to a friend who has suddenly collapsed, and explain the specific instructions that would be given."

[0111] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0112] Step 1: The user launches the "RescueVoice" application on their smartphone. This launches the app and makes it ready to accept voice input. When the user says "Help!", the built-in microphone captures the voice. The input is the user's voice, and the output is voice data.

[0113] Step 2: The device generates the captured voice data and sends it to the server via the Internet. The specific operations at this stage are the process from voice capture to data transmission. The input is the voice data, and the output is the voice data sent to the server.

[0114] Step 3: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze the received voice data. This engine converts the voice data into text data. The specific data processing involves analyzing the voice data and converting it into text. The input is voice data, and the output is text data.

[0115] Step 4: The server performs situation analysis based on the generated text data. Here, a generative AI model (e.g., OpenAI's GPT-4) is used to determine the situation from the text. This analysis identifies an emergency situation and generates appropriate first aid instructions. The input is text data, and the output is first aid instructions.

[0116] Step 5: The server uses a text-to-speech engine to encode the generated first aid instructions as audio data. The specific operation is to convert the text data into an audio file. The input is the first aid text instructions, and the output is the audio data.

[0117] Step 6: The server sends the encoded voice data to the user terminal via the Internet. The specific operation at this stage is to send data from the server and receive it at the user terminal. The input is the voice data, and the output is the voice data received at the terminal.

[0118] Step 7: The device plays the received voice data and delivers instructions to the user. The user then performs the instructed first aid. Specifically, the instruction to the user is "Please begin chest compressions." The input is the voice data, and the output is the played voice instructions.

[0119] Step 8: The device monitors the user's progress in providing first aid and continuously captures new voice input. For example, when the user says, "I went to get the AED," the voice data is sent to the server again. The input is the user's new voice, and the output is the regenerated voice data.

[0120] Step 9: The server analyzes the newly received voice data and generates additional first aid instructions as needed. It then encodes these instructions as voice data and sends them to the user terminal. For example, the generated instruction might be, "Prepare to use the AED. Attach the pads to the patient's chest." The input is the new voice data, and the output is the voice data of the additional first aid instructions.

[0121] (Application example 1)

[0122] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0123] In the event of an emergency at a factory, it is difficult for employees to provide first aid quickly and accurately. For this reason, there is a need for a system that can immediately provide appropriate first aid instructions in an emergency and improve safety. In particular, a rapid response is required when a worker collapses, and there is a need for a method to solve this problem using robots.

[0124] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0125] In this invention, the server includes a voice input means, a means for analyzing voice data and converting it into text data, a means for analyzing the situation and generating first aid instructions, a means for encoding the generated instructions as voice data, a means for transmitting the voice instruction data to a user terminal, a means for playing the encoded voice instruction data on the user terminal, a means for the robot to capture voice using a built-in microphone when an emergency occurs and transmit the voice data to the server, and a means for the robot to receive and play the first aid instruction voice data transmitted from the server. This enables the robot to instantly provide appropriate first aid instructions in the event of an emergency at a factory site, enabling a fast and effective response.

[0126] "Audio input means" refers to hardware and software for capturing audio data.

[0127] "Means for analyzing voice data and converting it into text data" refers to the function of converting voice data into text using voice recognition technology.

[0128] The "means for analyzing the situation and generating first aid instructions" refers to a function for analyzing the emergency situation based on input voice data and generating appropriate first aid instructions.

[0129] "Means for encoding the generated instructions as voice data" refers to speech synthesis technology for converting text data into voice data.

[0130] The "means for transmitting voice instruction data to the user terminal" refers to a communication means for transmitting the generated voice data to the user terminal.

[0131] The "means for reproducing the encoded voice instruction data on the user terminal" refers to a function for reproducing the received voice instruction data on the user terminal.

[0132] "Means for the robot to capture audio using its built-in microphone in the event of an emergency and transmit that audio data to a server" refers to the function of transmitting audio data in an emergency to a server using the robot's microphone and communication function.

[0133] "Means for the robot to receive and play back the voice data of first aid instructions sent from the server" refers to the function of the robot to play back the voice data of first aid instructions received from the server.

[0134] The present invention relates to a system that provides effective first aid instructions for emergencies that occur at factory sites. This system aims to improve worker safety by using a robot to input voice and provide appropriate first aid in emergencies.

[0135] The robot uses a built-in microphone as a means of voice input. This microphone captures surrounding sounds and sends the data to a server. If the robot detects an emergency, for example, if a worker yells "Help!", the captured voice data is immediately sent to the server.

[0136] On the server side, speech recognition software (e.g., Google's speech recognition API) is used to convert the speech data into text data, which is then analyzed by an AI model (e.g., a generative AI model) to understand the emergency situation and generate appropriate first aid instructions.

[0137] The generated first aid instructions are encoded as voice data using speech synthesis technology, using speech synthesis software (e.g., the pydub library), and the encoded voice data is sent to the robot, which plays it back, providing the appropriate first aid instructions to the worker in real time.

[0138] As a concrete example, consider the case where a worker collapses in a factory. When the worker says "Help!", the robot's built-in microphone captures the voice and sends the voice data to the server. The server analyzes the voice and, if it determines that cardiopulmonary resuscitation is necessary, generates the instruction "Please begin chest compressions." This instruction is encoded as voice data and sent back to the robot. The robot then plays back this instruction and provides the worker with specific first aid instructions, enabling a rapid response.

[0139] As an example, the following prompt is presented:

[0140] "A worker falls and cries out, 'Help!' This voice is captured by the robot's microphone, and appropriate first aid instructions are provided in real time."

[0141] In this way, the present invention allows for a fast and effective response to emergencies on the factory floor, greatly improving worker safety.

[0142] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0143] Step 1:

[0144] The robot uses a built-in microphone to capture surrounding sounds. The input is the worker's speech, and the output is audio data. Specifically, the microphone captures the voice saying "Help!" and stores the audio data in the robot's memory.

[0145] Step 2:

[0146] The robot sends the captured audio data to the server. The input is the audio data obtained in step 1, and the output is the audio data sent to the server. Specifically, the robot uploads the audio data to the server via an internet connection.

[0147] Step 3:

[0148] The server analyzes the received voice data and converts it into text data. The input is voice data and the output is text data. Specifically, the server uses voice recognition software (for example, Google's voice recognition API) to analyze the voice data and convert it into text data such as "Help!"

[0149] Step 4:

[0150] The server analyzes the situation based on the text data and generates appropriate first aid instructions. The input is text data, and the output is first aid instructions. Specifically, the server analyzes the text data using a generative AI model and generates the instruction "Please begin chest compressions."

[0151] Step 5:

[0152] The server encodes the generated first aid instructions as audio data. The input is the first aid instructions, and the output is the encoded audio data. Specifically, the server converts the instructions into audio data using speech synthesis software (e.g., the pydub library).

[0153] Step 6:

[0154] The server sends the encoded voice data to the robot. The input is the encoded voice data, and the output is the voice data sent to the robot. Specifically, the server uploads the encoded voice data to the robot via an internet connection.

[0155] Step 7:

[0156] The robot receives the transmitted voice data and plays it back to the worker. The input is encoded voice data, and the output is the played-back voice instruction. Specifically, the robot plays the received voice data through its built-in speaker and tells the worker to "start chest compressions."

[0157] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0158] This invention relates to a system that uses voice recognition and emotion recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input and emotion recognition.

[0159] First, the user launches the "RescueVoice" application on their smartphone. The system starts up when the user says "Help!" The user's device captures the voice using a built-in microphone and generates digital voice data, which is then immediately sent to the server.

[0160] On the server side, a voice recognition engine analyzes the received voice data and converts it into text data. Based on this text data, an AI model analyzes the situation. For example, if someone says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[0161] Next, the first aid instructions generated by the server are encoded as voice data. The encoded voice data is immediately sent to the user's device. The user's device plays back the received voice data and delivers it to the user. The user begins first aid according to the instructions. For example, the instructions might be, "Start chest compressions. Place your hands together in the center of the chest and compress at a rate of 100-120 times per minute."

[0162] Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions. The emotion engine analyzes voice data and user behavior data to identify the user's emotional state. For example, if the user is very anxious, the emotion engine will determine that the user is in a stressed state.

[0163] Based on the user's emotional state recognized by the emotion engine, the server adjusts the instructions. For example, if a user is under stress, the server can provide additional encouragement and advice to encourage them to remain calm and administer first aid. A voice instruction such as "Please stay calm. It's okay. I'll give you instructions, so please follow them," is generated, encoded, and sent to the user's device.

[0164] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice. If the emotion engine determines that the user is under stress, it provides additional encouragement such as "Please stay calm. Your actions will help."

[0165] Thus, by combining voice input and emotion recognition, the system of the present invention helps users administer first aid more calmly and effectively, providing vital life-saving assistance until an ambulance arrives.

[0166] The processing flow will be explained below.

[0167] Step 1:

[0168] The user launches the "RescueVoice" app on their smartphone, which puts the app into a voice input standby state.

[0169] Step 2:

[0170] The user says "Help!" The device uses the built-in microphone to capture the user's voice and generate digital audio data.

[0171] Step 3:

[0172] The audio data generated by the device is sent to the server using the secure HTTP(S) protocol.

[0173] Step 4:

[0174] The server passes the received voice data to a speech recognition engine, which converts the voice data into text data. This speech recognition engine uses a highly accurate generative AI model.

[0175] Step 5:

[0176] The server passes the text data to an AI model, which analyzes the situation. For example, it analyzes text such as "My friend has collapsed" and determines whether the situation is urgent.

[0177] Step 6:

[0178] The server generates appropriate first aid instructions based on the analysis results. For example, if cardiac arrest is suspected, it generates instructions such as, "Start chest compressions immediately. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[0179] Step 7:

[0180] The server encodes the generated instructions as voice data and transmits the voice data to the user terminal.

[0181] Step 8:

[0182] The device plays the received voice data and delivers it to the user, who then follows the instructions to begin providing first aid.

[0183] Step 9:

[0184] The device continuously monitors the user's voice input. For example, if the user says, "Help isn't here yet," the device retransmits the voice to the server.

[0185] Step 10:

[0186] The server receives the new voice data, analyzes it again, and generates additional first aid instructions based on the new situation, such as "Continue chest compressions and use an AED if one is available."

[0187] Step 11:

[0188] The server then transmits the encoded audio data back to the user terminal, which then plays the audio back to the user.

[0189] Step 12:

[0190] The user follows additional instructions to administer any necessary first aid, such as attaching pads to the patient's chest to use an AED.

[0191] Step 13:

[0192] The server sends the voice data and behavioral data to the emotion engine, which analyzes the user's emotional state and determines whether the user is stressed or anxious.

[0193] Step 14:

[0194] The server adjusts the instructions based on the analysis results of the emotion engine. For example, if it determines that the user is under stress, it adds encouragement and advice to stabilize the user, such as "Please stay calm. It's okay. I'll give you instructions, so please act as instructed."

[0195] Step 15:

[0196] The server encodes the adjusted instructions as voice data and transmits it to the user terminal.

[0197] Step 16:

[0198] The device then plays back the received audio data and delivers it to the user, who then follows the instructions to continue providing first aid.

[0199] Through this series of processing steps, the user receives appropriate first aid instructions in real time, as well as encouragement and advice tailored to their emotional state, enabling them to carry out effective life-saving measures until an ambulance arrives.

[0200] Example 2

[0201] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] Conventional first aid support systems use voice input and output instructions, but they are unable to consider the user's emotional state, which can lead to users becoming impatient or anxious and being unable to act appropriately in an emergency. In particular, when a user is under stress, there is a high risk that they will lose their composure and make a serious, life-threatening mistake. In addition, when multiple situations overlap, it is sometimes difficult to properly determine priorities.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0204] In this invention, the server includes a voice input unit, a unit for analyzing voice data and converting it into text data, a unit for analyzing a situation based on the text data and generating appropriate first aid instructions, a unit for encoding the generated first aid instructions as voice data, a unit for transmitting the encoded voice instruction data to a user terminal, a unit for playing the encoded voice instruction data on the user terminal, a unit for analyzing the voice data and behavioral data to recognize the user's emotional state, and a unit for adjusting the first aid instructions based on the user's emotional state and generating additional instructions or encouragement. This allows the user to provide first aid appropriately and calmly in an emergency. Furthermore, even when multiple situations arise, the server can determine priorities and provide optimal instructions.

[0205] "Audio input means" refers to a device or function that captures a user's voice and converts it into digital audio data.

[0206] The "means for analyzing voice data and converting it into text data" refers to software or hardware for analyzing acquired voice data and converting it into character string data.

[0207] The "means for analyzing the situation and generating appropriate first aid instructions" is a device or function that judges the situation based on text data and automatically generates optimal first aid instructions.

[0208] The "means for encoding the generated first aid instructions as audio data" refers to software or hardware for converting the generated text format first aid instructions into audio data.

[0209] The "means for transmitting encoded voice instruction data to a user terminal" is a function or system that transmits encoded voice data to a device held by a user via a network.

[0210] The "means for reproducing encoded voice instruction data on the user terminal" refers to a system or function for reproducing voice data sent to the user terminal and conveying instructions to the user.

[0211] The "means for recognizing the emotional state of a user by analyzing voice data and behavioral data" refers to software or a device for analyzing the voice data and behavioral data of a user and automatically recognizing the emotional state.

[0212] "Means for adjusting first aid instructions and generating additional instructions or encouragement based on emotional state" refers to a system or function for adjusting the content of first aid instructions based on the user's emotional state and generating further instructions or encouragement as needed.

[0213] This invention relates to a system that uses voice recognition and emotion recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input and emotion recognition.

[0214] First, the user launches a dedicated application on their smartphone. The system starts up when the user says "Help!" The user's device captures the voice using a built-in microphone and generates digital voice data. This generated voice data is immediately sent to the server.

[0215] On the server side, a speech recognition engine (e.g., Google Speech-to-Text API) analyzes the received voice data and converts it into text data. Based on this text data, an AI model (e.g., OpenAI GPT-3) analyzes the situation. For example, if a voice input says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[0216] Next, a speech synthesis engine (e.g., Amazon Polly) is used to encode the first aid instructions generated by the server as voice data. This encoded voice data is immediately sent to the user's device. The user's device plays the received voice data and delivers it to the user. The user can start first aid by following the instructions. For example, the voice instruction is "Start chest compressions. Place your hands together at the center of the chest and compress at a rate of 100-120 times per minute."

[0217] Furthermore, this system incorporates an emotion engine (e.g., Affectiva's emotion recognition API) that recognizes the user's emotions. The emotion engine analyzes voice data and user behavior data to identify the user's emotional state. For example, if the user is very anxious, the emotion engine will determine that the user is in a state of stress.

[0218] Based on the user's emotional state recognized by the emotion engine, the server adjusts the instructions. For example, if a user is under stress, the server can provide additional encouragement and advice to encourage them to remain calm and administer first aid. A voice instruction such as "Please stay calm. It's okay. I'll give you instructions, so please follow them," is generated, encoded, and sent to the user's device.

[0219] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the user device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice. If the emotion engine determines that the user is under stress, it provides additional encouragement such as "Please stay calm. Your actions will help."

[0220] Examples of prompts for the generative AI model include, "The user says that their friend has collapsed. Please suggest the best first aid for this situation." or "The user seems very anxious. Please think of some encouraging words to calm the user in this situation."

[0221] In this way, the combination of voice input and real-time emotion recognition technology provides a system that allows users to respond calmly and effectively in an emergency, providing vital life-saving assistance until an ambulance arrives.

[0222] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0223] Step 1:

[0224] The user launches a dedicated application on their smartphone and says, "Help!"

[0225] Input: User's voice

[0226] Output: Captured digital audio data

[0227] Specific actions: The user taps the icon on their smartphone to launch the app and utters "Help!" in an emergency.

[0228] Step 2:

[0229] The device uses a built-in microphone to capture the user's voice and convert it into digital audio data.

[0230] Input: User's voice

[0231] Output: Digital audio data

[0232] Specific operation: The smartphone's microphone picks up the user's speech and digitizes it as voice data.

[0233] Step 3:

[0234] The terminal transmits the generated digital voice data to the server via the network.

[0235] Input: Digital audio data

[0236] Output: Audio data sent to the server

[0237] What happens: The device sends audio data to the server via an internet connection.

[0238] Step 4:

[0239] The server uses a speech recognition engine (e.g., Google Speech-to-Text API) to analyze the received voice data and convert it into text data.

[0240] Input: Audio data sent to the server

[0241] Output: Text data

[0242] Specific operation: The server calls the Google Speech-to-Text API and converts the audio data into text.

[0243] Step 5:

[0244] The server analyzes the situation using an AI model (e.g., OpenAI GPT-3) based on the text data and generates first aid instructions.

[0245] Input: Text data

[0246] Output: First aid instruction text

[0247] How it works: The server inputs text data into the GPT-3 model and generates first aid instructions such as "Begin chest compressions."

[0248] Step 6:

[0249] Use a speech synthesis engine (e.g., Amazon Polly) to encode the server-generated first aid instructions as audio data.

[0250] Input: First Aid Instructions Text

[0251] Output: Audio data

[0252] What happens: The server calls Amazon Polly to convert the text data into speech.

[0253] Step 7:

[0254] The server transmits the encoded voice instruction data to the user terminal via the network.

[0255] Input: Audio data

[0256] Output: Audio data sent to the user's device

[0257] Specific operation: The server sends encoded voice data to the user's terminal via the Internet.

[0258] Step 8:

[0259] The terminal plays back the received audio data and conveys emergency instructions to the user.

[0260] Input: Audio data sent to the user's device

[0261] Output: The audio instructions played

[0262] Specific operation: The device speaker plays audio data and gives the user the voice instruction, "Please begin chest compressions."

[0263] Step 9:

[0264] The server uses an emotion engine (e.g., Affectiva's emotion recognition API) to analyze the user's voice data and behavioral data to identify the user's emotional state.

[0265] Input: User voice data, behavioral data

[0266] Output: User's emotional state data

[0267] How it works: The server calls Affectiva's API, analyzes the user's voice and behavioral data, and identifies their stress level.

[0268] Step 10:

[0269] The server adjusts first aid instructions and generates additional instructions and encouragement based on the perceived emotional state.

[0270] Input: User emotional state data

[0271] Output: Adjusted first aid instruction text, additional encouragement text

[0272] Specific actions: The AI ​​model generates additional instructions such as, "Please stay calm. It's okay. I'll give you instructions, so please follow them," and encodes them as voice data and sends them to the user's device.

[0273] Step 11:

[0274] The terminal plays back the received additional audio data to provide encouragement or advice to the user.

[0275] Input: Adjusted first aid instruction audio data, additional encouragement audio data

[0276] Output: Played back audio instructions of encouragement and advice

[0277] Specific action: The device speaker plays audio data and conveys an encouraging message to the user to stay calm.

[0278] By using the specific processing steps described above, this system supports the user in providing appropriate and calm first aid in an emergency.

[0279] (Application example 2)

[0280] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0281] Conventional first aid support systems can sometimes make it difficult for users to respond appropriately in an emergency using only voice instructions. This is especially true when users are stressed or anxious. Furthermore, systems that rely solely on voice data cannot take into account the user's emotional state and lack the support necessary to help the user remain calm. This can reduce the effectiveness of first aid and potentially waste valuable time waiting for an ambulance to arrive.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means, a means for analyzing voice data and converting it into text data, a means for analyzing the situation and generating first aid instructions, a means for encoding the generated instructions as voice data, a means for transmitting the voice instruction data to the user terminal, a means for playing the encoded voice instruction data on the user terminal, an emotion recognition means for recognizing the emotional state, and a means for adjusting the instruction content based on the emotion recognition result. As a result, appropriate first aid instructions according to the user's voice and emotional state are provided in real time, allowing the user to respond to emergencies more calmly and effectively.

[0283] The "voice input means" is a device for capturing the user's voice and recording it as digital voice data.

[0284] The "means for analyzing voice data and converting it into text data" refers to a system or software for analyzing captured voice data and converting it into character string data.

[0285] The "means for analyzing the situation and generating first aid instructions" refers to a system or software for determining appropriate first aid based on speech recognition and the situation, and generating the instructions.

[0286] The "means for encoding the generated instructions as audio data" is a system or software for converting the generated first aid instructions into audio data.

[0287] The "means for transmitting voice instruction data to a user terminal" is a communication means for transferring the generated voice data to the user's device.

[0288] The "means for playing back encoded voice instruction data on a user terminal" refers to hardware or software for playing back encoded voice data on a device held by the user.

[0289] The "emotion recognition means for recognizing an emotional state" is an algorithm or software for analyzing and identifying the emotional state of a user from voice data.

[0290] The "means for adjusting instructions based on emotion recognition results" refers to a system or software for modifying and optimizing first aid instructions according to the user's emotional state.

[0291] This invention is a system for responding to emergencies, which supports first aid by utilizing voice input and emotion recognition technology. The specific configuration and operation procedure are shown below.

[0292] The server captures the user's voice through a voice input means. This voice input is performed using the built-in microphone of the user's device, such as a smartphone. The captured voice data is recorded in digital format and sent to the server. The server analyzes the received voice data using voice recognition software (for example, the speech_recognition library) and converts it into text data.

[0293] Based on the text data, the server analyzes the situation and generates appropriate first aid instructions. The AI ​​model used here references historical data and medical knowledge to determine the most appropriate first aid. Based on this determination, the server encodes the instructions into audio data (using a text-to-speech system, for example, the TextToSpeech class).

[0294] The generated voice instruction data is immediately transmitted to the user terminal, which plays back the voice data and provides the user with appropriate first aid instructions.

[0295] Furthermore, an emotion recognition engine (e.g., using the EmotionRecognition module) that recognizes the user's emotional state analyzes the voice data and determines whether the user is feeling stressed. Based on the emotion recognition result, the server adjusts the instructions. For example, if the user is determined to be in a stressful state, the server will include additional encouragement or advice in the voice instructions.

[0296] As a concrete example, consider a scenario in which a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends the data to the server. The server analyzes the voice and generates an instruction to "Start chest compressions" as an emergency. The instruction is encoded as voice data and sent to the user device. When the user starts chest compressions as instructed, the emotion engine continuously monitors the user's voice and, if it determines that the user is in a stressed state, provides additional voice instructions such as "Please stay calm. It's okay. I'm giving you instructions, so please follow them."

[0297] An example of a prompt is "When a user is in an extremely urgent situation and needs help, please generate voice instructions that utilize emotion recognition." This will help users provide first aid as calmly and effectively as possible.

[0298] The present invention reduces stress when a user performs first aid in an emergency, and enables the user to perform correct procedures more reliably.

[0299] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0300] Step 1:

[0301] A user speaks "Help!" into a smartphone. The user's input is voice data, which is captured using the smartphone's built-in microphone. The output is digital voice data.

[0302] Step 2:

[0303] The device transmits the captured audio data to the server. The input is digital audio data, and the output is the audio data transmitted to the server. The data transmission is performed using the device's communication module.

[0304] Step 3:

[0305] The server analyzes the received voice data using a voice recognition engine (for example, the speech_recognition library) and converts it into text data. The input is voice data, and the output is text data. Specifically, the waveform of the voice data is analyzed and data processing is performed to express it as a string of characters.

[0306] Step 4:

[0307] The server analyzes the situation based on the converted text data. The AI ​​model used is a generative AI model, which refers to past data and medical knowledge to determine the most appropriate first aid treatment. The input is text data, and the output is first aid instruction data. For data calculation, the generative AI model performs an analytical process on the text data and generates first aid instructions.

[0308] Step 5:

[0309] The server encodes the generated first aid instructions as audio data. The input is the first aid instructions and the output is the encoded audio data. The software used is a text-to-speech system (e.g., TextToSpeech class). At this stage, the instructions are prepared to be transmitted to the user as audio.

[0310] Step 6:

[0311] The server transmits the encoded voice instruction data to the user terminal. The input is the encoded voice data, and the output is the voice data transmitted to the user terminal. The data transmission is performed using a communication module.

[0312] Step 7:

[0313] The device plays the received audio data and provides the user with appropriate first aid instructions. The input is the encoded audio data, and the output is audio instructions that the user can hear. The audio instructions are played using the device's audio playback capabilities.

[0314] Step 8:

[0315] The server analyzes the user's emotional state from the voice data using an emotion recognition engine to recognize the user's emotional state. The input is the newly captured voice data and the output is the emotional state data. The voice analysis is performed using an emotion recognition engine (e.g., EmotionRecognition module).

[0316] Step 9:

[0317] The server adjusts the instructions based on the emotion recognition results. The input is emotional state data and first aid instruction data, and the output is the adjusted first aid instruction data. Specifically, if the user is in a stressful state, the server includes encouragement and advice in addition to the first aid instruction data.

[0318] Step 10:

[0319] The server re-encodes the adjusted first aid instruction data as voice data and transmits it to the user terminal. The input is the adjusted instruction data, and the output is the re-encoded voice data. The data is again transmitted using the communication module, and the user terminal plays back the voice data.

[0320] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0321] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0322] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0323] [Second embodiment]

[0324] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0325] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0326] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0327] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0328] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0329] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0330] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0331] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0332] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0333] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0334] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0335] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0336] This invention relates to a system that uses voice recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input.

[0337] First, the user launches the "RescueVoice" application on their smartphone. The system starts up when the user says "Help me!" The user's device captures the voice using the built-in microphone and generates voice data. The generated voice data is immediately sent to the server.

[0338] On the server side, a voice recognition engine analyzes the received voice data and converts it into text data. Based on this text data, an AI model analyzes the situation. For example, if someone says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[0339] Next, the first aid instructions generated by the server are encoded as voice data. The encoded voice data is immediately sent to the user's device. The user's device plays back the received voice data and delivers it to the user. The user begins first aid according to the instructions. For example, the instructions might be, "Start chest compressions. Place your hands together in the center of the chest and compress at a rate of 100-120 times per minute."

[0340] The system can also monitor the progress of first aid and provide additional instructions as needed. For example, if a user says, "I went to get the AED," new voice data is sent to the server, which immediately analyzes it. As a result, additional instructions such as "Prepare to use the AED. Attach the pads to the patient's chest" are generated and encoded as voice data. The user device plays back the voice data and provides the user with additional instructions.

[0341] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice and provides additional instructions as needed. If an AED is available, the system also provides specific instructions on how to use it. As a result, the chances of appropriate first aid being administered and saving a life are significantly increased before an ambulance arrives.

[0342] This system allows even ordinary users without specialized knowledge to provide first aid quickly and accurately, enabling important life-saving measures to be taken before an ambulance arrives.

[0343] The processing flow will be explained below.

[0344] Step 1:

[0345] The user launches the "RescueVoice" app on their smartphone, which puts the app into a voice input standby state.

[0346] Step 2:

[0347] The user says "Help!" The device uses the built-in microphone to capture the user's voice and generate digital audio data.

[0348] Step 3:

[0349] The audio data generated by the device is sent to the server using the secure HTTP(S) protocol.

[0350] Step 4:

[0351] The server passes the received voice data to a speech recognition engine, which converts the voice data into text data. This speech recognition engine uses a highly accurate generative AI model.

[0352] Step 5:

[0353] The server passes the text data to an AI model, which analyzes the situation. For example, it analyzes text such as "My friend has collapsed" and determines whether the situation is urgent.

[0354] Step 6:

[0355] The server generates appropriate first aid instructions based on the analysis results. For example, if cardiac arrest is suspected, it generates instructions such as, "Start chest compressions immediately. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[0356] Step 7:

[0357] The server encodes the generated instructions as voice data and transmits the voice data to the user terminal.

[0358] Step 8:

[0359] The device plays the received voice data and delivers it to the user, who then follows the instructions to begin providing first aid.

[0360] Step 9:

[0361] The device continuously monitors the user's voice input. For example, if the user says, "Help isn't here yet," the device retransmits the voice to the server.

[0362] Step 10:

[0363] The server receives the new voice data, analyzes it again, and generates additional first aid instructions based on the new situation, such as "Continue chest compressions and use an AED if one is available."

[0364] Step 11:

[0365] The server then transmits the encoded audio data back to the user terminal, which then plays the audio back to the user.

[0366] Step 12:

[0367] The user follows additional instructions to administer any necessary first aid, such as attaching pads to the patient's chest to use an AED.

[0368] Through this series of processing steps, the user receives appropriate first aid instructions in real time, enabling them to take effective life-saving measures until an ambulance arrives.

[0369] Example 1

[0370] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0371] In modern society, emergencies can occur due to sudden illness or accidents. Responding to these emergencies requires prompt and appropriate first aid, but the general public often does not know how to provide it. To solve this problem, a system is needed that allows anyone to provide first aid easily, even without specialized knowledge. It also needs to be able to respond to multiple situations and determine the appropriate course of action.

[0372] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0373] In this invention, the server includes means for analyzing voice data and converting it into text data, means for analyzing the situation based on the text data and generating first aid instructions, and means for encoding the generated first aid instructions as voice data. This allows a user to receive appropriate first aid instructions in real time simply by calling for help by voice. The system can also monitor the progress of the first aid and provide additional instructions as needed, allowing it to respond to multiple situations and provide optimal treatment.

[0374] The "voice input means" is a device that has the function of taking in the user's voice and generating it as digital voice data.

[0375] The "means for generating audio data and transmitting it to a server" is a device or software that has the function of transmitting captured audio data to a server using an appropriate communication protocol.

[0376] The "means for analyzing voice data and converting it into text data" refers to a software algorithm or device for analyzing received voice data and generating the content of the voice data as text data.

[0377] "Means for analyzing a situation based on text data and generating first aid instructions" refers to software or a device that uses an AI model or rule-based system to analyze a situation based on text data and issue appropriate first aid instructions.

[0378] The "means for encoding the generated first aid instructions as audio data" is software or a device for converting the text-based first aid instructions back into audio data.

[0379] The "means for transmitting encoded voice data to a user terminal" is a device or software that has the function of transmitting encoded voice data to a user terminal using an appropriate communication protocol.

[0380] The "means for reproducing audio data received on a user terminal" refers to a device or software having the function of reproducing audio data received by a user terminal and providing audio instructions to the user.

[0381] This invention is a system that supports first aid in emergencies using voice recognition and an AI model. The system includes a voice input unit, a voice analysis unit, a situation analysis unit, a first aid instruction generation unit, and a voice output unit.

[0382] System configuration and operation

[0383] Hardware and Software Configuration

[0384] User terminal: A mobile information terminal such as a smartphone. It uses a built-in microphone to capture voice input and generate voice data. It also has a speaker to play back voice instruction data from the server.

[0385] Server: A powerful computer system that includes a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze and convert audio data into text, and a generative AI model (e.g., OpenAI's GPT-4) to analyze the situation and generate first aid instructions.

[0386] Program processing explanation

[0387] Voice Recognition

[0388] In an emergency, a user launches the "RescueVoice" application on their smartphone and utters the command "Help!" The user's device's built-in microphone captures this voice and converts it into digital audio data, which is then sent to a server via the Internet.

[0389] Analysis of audio data

[0390] The server uses a speech recognition engine to analyze the received voice data and convert it into text data. For example, if a user says, "Help me, my friend has fallen down," the voice data is converted into text data.

[0391] Analyzing the situation and generating first aid instructions

[0392] The converted text data is then analyzed by a generative AI model to understand the situation. The AI ​​model then references past data and medical knowledge to generate the most appropriate first aid instructions. For example, it might generate specific instructions such as, "If cardiac arrest is suspected, begin chest compressions."

[0393] Encoding and transmitting first aid instructions as audio data

[0394] The server encodes the generated emergency treatment instructions as voice data, which is then transmitted to the user terminal via the Internet.

[0395] Playing voice instructions

[0396] The user device plays the received voice data and conveys instructions to the user, such as "Please begin chest compressions. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[0397] Monitor progress and generate further instructions

[0398] The system continuously monitors the user's progress in administering first aid and generates additional instructions as needed. For example, if the user says, "I went to get the AED," new voice data is sent to the server for analysis. As a result, additional instructions are generated and encoded as voice data, such as, "Prepare the AED. Attach the pads to the patient's chest."

[0399] Examples of concrete examples and prompts

[0400] Specific examples

[0401] If a friend suddenly collapses:

[0402] 1. User: Launch the "RescueVoice" application on their smartphone.

[0403] 2. User: Says "Help!"

[0404] 3. Device: Captures audio with the built-in microphone and sends the audio data to the server.

[0405] 4. Server: The speech recognition engine converts the speech into text, and the AI ​​model determines the situation: "My friend has collapsed."

[0406] 5. Server: Generates the instruction "Please begin chest compressions" and encodes it as audio data.

[0407] 6. Terminal: Plays the received audio data and delivers it to the user.

[0408] 7. User: Begin chest compressions as instructed.

[0409] Prompt Sentence Examples

[0410] "If a user is in an emergency and needs first aid, design a system that provides appropriate instructions in real time based on voice input."

[0411] "Imagine a scenario in which a user uses a voice recognition system to provide first aid to a friend who has suddenly collapsed, and explain the specific instructions that would be given."

[0412] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0413] Step 1: The user launches the "RescueVoice" application on their smartphone. This launches the app and makes it ready to accept voice input. When the user says "Help!", the built-in microphone captures the voice. The input is the user's voice, and the output is voice data.

[0414] Step 2: The device generates the captured voice data and sends it to the server via the Internet. The specific operations at this stage are the process from voice capture to data transmission. The input is the voice data, and the output is the voice data sent to the server.

[0415] Step 3: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze the received voice data. This engine converts the voice data into text data. The specific data processing involves analyzing the voice data and converting it into text. The input is voice data, and the output is text data.

[0416] Step 4: The server performs situation analysis based on the generated text data. Here, a generative AI model (e.g., OpenAI's GPT-4) is used to determine the situation from the text. This analysis identifies an emergency situation and generates appropriate first aid instructions. The input is text data, and the output is first aid instructions.

[0417] Step 5: The server uses a text-to-speech engine to encode the generated first aid instructions as audio data. The specific operation is to convert the text data into an audio file. The input is the first aid text instructions, and the output is the audio data.

[0418] Step 6: The server sends the encoded voice data to the user terminal via the Internet. The specific operation at this stage is to send data from the server and receive it at the user terminal. The input is the voice data, and the output is the voice data received at the terminal.

[0419] Step 7: The device plays the received voice data and delivers instructions to the user. The user then performs the instructed first aid. Specifically, the instruction to the user is "Please begin chest compressions." The input is the voice data, and the output is the played voice instructions.

[0420] Step 8: The device monitors the user's progress in providing first aid and continuously captures new voice input. For example, when the user says, "I went to get the AED," the voice data is sent to the server again. The input is the user's new voice, and the output is the regenerated voice data.

[0421] Step 9: The server analyzes the newly received voice data and generates additional first aid instructions as needed. It then encodes these instructions as voice data and sends them to the user terminal. For example, the generated instruction might be, "Prepare to use the AED. Attach the pads to the patient's chest." The input is the new voice data, and the output is the voice data of the additional first aid instructions.

[0422] (Application example 1)

[0423] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0424] In the event of an emergency at a factory, it is difficult for employees to provide first aid quickly and accurately. For this reason, there is a need for a system that can immediately provide appropriate first aid instructions in an emergency and improve safety. In particular, a rapid response is required when a worker collapses, and there is a need for a method to solve this problem using robots.

[0425] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0426] In this invention, the server includes a voice input means, a means for analyzing voice data and converting it into text data, a means for analyzing the situation and generating first aid instructions, a means for encoding the generated instructions as voice data, a means for transmitting the voice instruction data to a user terminal, a means for playing the encoded voice instruction data on the user terminal, a means for the robot to capture voice using a built-in microphone when an emergency occurs and transmit the voice data to the server, and a means for the robot to receive and play the first aid instruction voice data transmitted from the server. This enables the robot to instantly provide appropriate first aid instructions in the event of an emergency at a factory site, enabling a fast and effective response.

[0427] "Audio input means" refers to hardware and software for capturing audio data.

[0428] "Means for analyzing voice data and converting it into text data" refers to the function of converting voice data into text using voice recognition technology.

[0429] The "means for analyzing the situation and generating first aid instructions" refers to a function for analyzing the emergency situation based on input voice data and generating appropriate first aid instructions.

[0430] "Means for encoding the generated instructions as voice data" refers to speech synthesis technology for converting text data into voice data.

[0431] The "means for transmitting voice instruction data to the user terminal" refers to a communication means for transmitting the generated voice data to the user terminal.

[0432] The "means for reproducing the encoded voice instruction data on the user terminal" refers to a function for reproducing the received voice instruction data on the user terminal.

[0433] "Means for the robot to capture audio using its built-in microphone in the event of an emergency and transmit that audio data to a server" refers to the function of transmitting audio data in an emergency to a server using the robot's microphone and communication function.

[0434] "Means for the robot to receive and play back the voice data of first aid instructions sent from the server" refers to the function of the robot to play back the voice data of first aid instructions received from the server.

[0435] The present invention relates to a system that provides effective first aid instructions for emergencies that occur at factory sites. This system aims to improve worker safety by using a robot to input voice and provide appropriate first aid in emergencies.

[0436] The robot uses a built-in microphone as a means of voice input. This microphone captures surrounding sounds and sends the data to a server. If the robot detects an emergency, for example, if a worker yells "Help!", the captured voice data is immediately sent to the server.

[0437] On the server side, speech recognition software (e.g., Google's speech recognition API) is used to convert the speech data into text data, which is then analyzed by an AI model (e.g., a generative AI model) to understand the emergency situation and generate appropriate first aid instructions.

[0438] The generated first aid instructions are encoded as voice data using speech synthesis technology, using speech synthesis software (e.g., the pydub library), and the encoded voice data is sent to the robot, which plays it back, providing the appropriate first aid instructions to the worker in real time.

[0439] As a concrete example, consider the case where a worker collapses in a factory. When the worker says "Help!", the robot's built-in microphone captures the voice and sends the voice data to the server. The server analyzes the voice and, if it determines that cardiopulmonary resuscitation is necessary, generates the instruction "Please begin chest compressions." This instruction is encoded as voice data and sent back to the robot. The robot then plays back this instruction and provides the worker with specific first aid instructions, enabling a rapid response.

[0440] As an example, the following prompt is presented:

[0441] "A worker falls and cries out, 'Help!' This voice is captured by the robot's microphone, and appropriate first aid instructions are provided in real time."

[0442] In this way, the present invention allows for a fast and effective response to emergencies on the factory floor, greatly improving worker safety.

[0443] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0444] Step 1:

[0445] The robot uses a built-in microphone to capture surrounding sounds. The input is the worker's speech, and the output is audio data. Specifically, the microphone captures the voice saying "Help!" and stores the audio data in the robot's memory.

[0446] Step 2:

[0447] The robot sends the captured audio data to the server. The input is the audio data obtained in step 1, and the output is the audio data sent to the server. Specifically, the robot uploads the audio data to the server via an internet connection.

[0448] Step 3:

[0449] The server analyzes the received voice data and converts it into text data. The input is voice data and the output is text data. Specifically, the server uses voice recognition software (for example, Google's voice recognition API) to analyze the voice data and convert it into text data such as "Help!"

[0450] Step 4:

[0451] The server analyzes the situation based on the text data and generates appropriate first aid instructions. The input is text data, and the output is first aid instructions. Specifically, the server analyzes the text data using a generative AI model and generates the instruction "Please begin chest compressions."

[0452] Step 5:

[0453] The server encodes the generated first aid instructions as audio data. The input is the first aid instructions, and the output is the encoded audio data. Specifically, the server converts the instructions into audio data using speech synthesis software (e.g., the pydub library).

[0454] Step 6:

[0455] The server sends the encoded voice data to the robot. The input is the encoded voice data, and the output is the voice data sent to the robot. Specifically, the server uploads the encoded voice data to the robot via an internet connection.

[0456] Step 7:

[0457] The robot receives the transmitted voice data and plays it back to the worker. The input is encoded voice data, and the output is the played-back voice instruction. Specifically, the robot plays the received voice data through its built-in speaker and tells the worker to "start chest compressions."

[0458] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0459] This invention relates to a system that uses voice recognition and emotion recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input and emotion recognition.

[0460] First, the user launches the "RescueVoice" application on their smartphone. The system starts up when the user says "Help!" The user's device captures the voice using a built-in microphone and generates digital voice data, which is then immediately sent to the server.

[0461] On the server side, a voice recognition engine analyzes the received voice data and converts it into text data. Based on this text data, an AI model analyzes the situation. For example, if someone says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[0462] Next, the first aid instructions generated by the server are encoded as voice data. The encoded voice data is immediately sent to the user's device. The user's device plays back the received voice data and delivers it to the user. The user begins first aid according to the instructions. For example, the instructions might be, "Start chest compressions. Place your hands together in the center of the chest and compress at a rate of 100-120 times per minute."

[0463] Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions. The emotion engine analyzes voice data and user behavior data to identify the user's emotional state. For example, if the user is very anxious, the emotion engine will determine that the user is in a stressed state.

[0464] Based on the user's emotional state recognized by the emotion engine, the server adjusts the instructions. For example, if a user is under stress, the server can provide additional encouragement and advice to encourage them to remain calm and administer first aid. A voice instruction such as "Please stay calm. It's okay. I'll give you instructions, so please follow them," is generated, encoded, and sent to the user's device.

[0465] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice. If the emotion engine determines that the user is under stress, it provides additional encouragement such as "Please stay calm. Your actions will help."

[0466] Thus, by combining voice input and emotion recognition, the system of the present invention helps users administer first aid more calmly and effectively, providing vital life-saving assistance until an ambulance arrives.

[0467] The processing flow will be explained below.

[0468] Step 1:

[0469] The user launches the "RescueVoice" app on their smartphone, which puts the app into a voice input standby state.

[0470] Step 2:

[0471] The user says "Help!" The device uses the built-in microphone to capture the user's voice and generate digital audio data.

[0472] Step 3:

[0473] The audio data generated by the device is sent to the server using the secure HTTP(S) protocol.

[0474] Step 4:

[0475] The server passes the received voice data to a speech recognition engine, which converts the voice data into text data. This speech recognition engine uses a highly accurate generative AI model.

[0476] Step 5:

[0477] The server passes the text data to an AI model, which analyzes the situation. For example, it analyzes text such as "My friend has collapsed" and determines whether the situation is urgent.

[0478] Step 6:

[0479] The server generates appropriate first aid instructions based on the analysis results. For example, if cardiac arrest is suspected, it generates instructions such as, "Start chest compressions immediately. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[0480] Step 7:

[0481] The server encodes the generated instructions as voice data and transmits the voice data to the user terminal.

[0482] Step 8:

[0483] The device plays the received voice data and delivers it to the user, who then follows the instructions to begin providing first aid.

[0484] Step 9:

[0485] The device continuously monitors the user's voice input. For example, if the user says, "Help isn't here yet," the device retransmits the voice to the server.

[0486] Step 10:

[0487] The server receives the new voice data, analyzes it again, and generates additional first aid instructions based on the new situation, such as "Continue chest compressions and use an AED if one is available."

[0488] Step 11:

[0489] The server then transmits the encoded audio data back to the user terminal, which then plays the audio back to the user.

[0490] Step 12:

[0491] The user follows additional instructions to administer any necessary first aid, such as attaching pads to the patient's chest to use an AED.

[0492] Step 13:

[0493] The server sends the voice data and behavioral data to the emotion engine, which analyzes the user's emotional state and determines whether the user is stressed or anxious.

[0494] Step 14:

[0495] The server adjusts the instructions based on the analysis results of the emotion engine. For example, if it determines that the user is under stress, it adds encouragement and advice to stabilize the user, such as "Please stay calm. It's okay. I'll give you instructions, so please act as instructed."

[0496] Step 15:

[0497] The server encodes the adjusted instructions as voice data and transmits it to the user terminal.

[0498] Step 16:

[0499] The device then plays back the received audio data and delivers it to the user, who then follows the instructions to continue providing first aid.

[0500] Through this series of processing steps, the user receives appropriate first aid instructions in real time, as well as encouragement and advice tailored to their emotional state, enabling them to carry out effective life-saving measures until an ambulance arrives.

[0501] Example 2

[0502] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0503] Conventional first aid support systems use voice input and output instructions, but they are unable to consider the user's emotional state, which can lead to users becoming impatient or anxious and being unable to act appropriately in an emergency. In particular, when a user is under stress, there is a high risk that they will lose their composure and make a serious, life-threatening mistake. In addition, when multiple situations overlap, it is sometimes difficult to properly determine priorities.

[0504] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0505] In this invention, the server includes a voice input unit, a unit for analyzing voice data and converting it into text data, a unit for analyzing a situation based on the text data and generating appropriate first aid instructions, a unit for encoding the generated first aid instructions as voice data, a unit for transmitting the encoded voice instruction data to a user terminal, a unit for playing the encoded voice instruction data on the user terminal, a unit for analyzing the voice data and behavioral data to recognize the user's emotional state, and a unit for adjusting the first aid instructions based on the user's emotional state and generating additional instructions or encouragement. This allows the user to provide first aid appropriately and calmly in an emergency. Furthermore, even when multiple situations arise, the server can determine priorities and provide optimal instructions.

[0506] "Audio input means" refers to a device or function that captures a user's voice and converts it into digital audio data.

[0507] The "means for analyzing voice data and converting it into text data" refers to software or hardware for analyzing acquired voice data and converting it into character string data.

[0508] The "means for analyzing the situation and generating appropriate first aid instructions" is a device or function that judges the situation based on text data and automatically generates optimal first aid instructions.

[0509] The "means for encoding the generated first aid instructions as audio data" refers to software or hardware for converting the generated text format first aid instructions into audio data.

[0510] The "means for transmitting encoded voice instruction data to a user terminal" is a function or system that transmits encoded voice data to a device held by a user via a network.

[0511] The "means for reproducing encoded voice instruction data on the user terminal" refers to a system or function for reproducing voice data sent to the user terminal and conveying instructions to the user.

[0512] The "means for recognizing the emotional state of a user by analyzing voice data and behavioral data" refers to software or a device for analyzing the voice data and behavioral data of a user and automatically recognizing the emotional state.

[0513] "Means for adjusting first aid instructions and generating additional instructions or encouragement based on emotional state" refers to a system or function for adjusting the content of first aid instructions based on the user's emotional state and generating further instructions or encouragement as needed.

[0514] This invention relates to a system that uses voice recognition and emotion recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input and emotion recognition.

[0515] First, the user launches a dedicated application on their smartphone. The system starts up when the user says "Help!" The user's device captures the voice using a built-in microphone and generates digital voice data. This generated voice data is immediately sent to the server.

[0516] On the server side, a speech recognition engine (e.g., Google Speech-to-Text API) analyzes the received voice data and converts it into text data. Based on this text data, an AI model (e.g., OpenAI GPT-3) analyzes the situation. For example, if a voice input says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[0517] Next, a speech synthesis engine (e.g., Amazon Polly) is used to encode the first aid instructions generated by the server as voice data. This encoded voice data is immediately sent to the user's device. The user's device plays the received voice data and delivers it to the user. The user can start first aid by following the instructions. For example, the voice instruction is "Start chest compressions. Place your hands together at the center of the chest and compress at a rate of 100-120 times per minute."

[0518] Furthermore, this system incorporates an emotion engine (e.g., Affectiva's emotion recognition API) that recognizes the user's emotions. The emotion engine analyzes voice data and user behavior data to identify the user's emotional state. For example, if the user is very anxious, the emotion engine will determine that the user is in a state of stress.

[0519] Based on the user's emotional state recognized by the emotion engine, the server adjusts the instructions. For example, if a user is under stress, the server can provide additional encouragement and advice to encourage them to remain calm and administer first aid. A voice instruction such as "Please stay calm. It's okay. I'll give you instructions, so please follow them," is generated, encoded, and sent to the user's device.

[0520] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the user device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice. If the emotion engine determines that the user is under stress, it provides additional encouragement such as "Please stay calm. Your actions will help."

[0521] Examples of prompts for the generative AI model include, "The user says that their friend has collapsed. Please suggest the best first aid for this situation." or "The user seems very anxious. Please think of some encouraging words to calm the user in this situation."

[0522] In this way, the combination of voice input and real-time emotion recognition technology provides a system that allows users to respond calmly and effectively in an emergency, providing vital life-saving assistance until an ambulance arrives.

[0523] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0524] Step 1:

[0525] The user launches a dedicated application on their smartphone and says, "Help!"

[0526] Input: User's voice

[0527] Output: Captured digital audio data

[0528] Specific actions: The user taps the icon on their smartphone to launch the app and utters "Help!" in an emergency.

[0529] Step 2:

[0530] The device uses a built-in microphone to capture the user's voice and convert it into digital audio data.

[0531] Input: User's voice

[0532] Output: Digital audio data

[0533] Specific operation: The smartphone's microphone picks up the user's speech and digitizes it as voice data.

[0534] Step 3:

[0535] The terminal transmits the generated digital voice data to the server via the network.

[0536] Input: Digital audio data

[0537] Output: Audio data sent to the server

[0538] What happens: The device sends audio data to the server via an internet connection.

[0539] Step 4:

[0540] The server uses a speech recognition engine (e.g., Google Speech-to-Text API) to analyze the received voice data and convert it into text data.

[0541] Input: Audio data sent to the server

[0542] Output: Text data

[0543] Specific operation: The server calls the Google Speech-to-Text API and converts the audio data into text.

[0544] Step 5:

[0545] The server analyzes the situation using an AI model (e.g., OpenAI GPT-3) based on the text data and generates first aid instructions.

[0546] Input: Text data

[0547] Output: First aid instruction text

[0548] How it works: The server inputs text data into the GPT-3 model and generates first aid instructions such as "Begin chest compressions."

[0549] Step 6:

[0550] Use a speech synthesis engine (e.g., Amazon Polly) to encode the server-generated first aid instructions as audio data.

[0551] Input: First Aid Instructions Text

[0552] Output: Audio data

[0553] What happens: The server calls Amazon Polly to convert the text data into speech.

[0554] Step 7:

[0555] The server transmits the encoded voice instruction data to the user terminal via the network.

[0556] Input: Audio data

[0557] Output: Audio data sent to the user's device

[0558] Specific operation: The server sends encoded voice data to the user's terminal via the Internet.

[0559] Step 8:

[0560] The terminal plays back the received audio data and conveys emergency instructions to the user.

[0561] Input: Audio data sent to the user's device

[0562] Output: The audio instructions played

[0563] Specific operation: The device speaker plays audio data and gives the user the voice instruction, "Please begin chest compressions."

[0564] Step 9:

[0565] The server uses an emotion engine (e.g., Affectiva's emotion recognition API) to analyze the user's voice data and behavioral data to identify the user's emotional state.

[0566] Input: User voice data, behavioral data

[0567] Output: User's emotional state data

[0568] How it works: The server calls Affectiva's API, analyzes the user's voice and behavioral data, and identifies their stress level.

[0569] Step 10:

[0570] The server adjusts first aid instructions and generates additional instructions and encouragement based on the perceived emotional state.

[0571] Input: User emotional state data

[0572] Output: Adjusted first aid instruction text, additional encouragement text

[0573] Specific actions: The AI ​​model generates additional instructions such as, "Please stay calm. It's okay. I'll give you instructions, so please follow them," and encodes them as voice data and sends them to the user's device.

[0574] Step 11:

[0575] The terminal plays back the received additional audio data to provide encouragement or advice to the user.

[0576] Input: Adjusted first aid instruction audio data, additional encouragement audio data

[0577] Output: Played back audio instructions of encouragement and advice

[0578] Specific action: The device speaker plays audio data and conveys an encouraging message to the user to stay calm.

[0579] By using the specific processing steps described above, this system supports the user in providing appropriate and calm first aid in an emergency.

[0580] (Application example 2)

[0581] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0582] Conventional first aid support systems can sometimes make it difficult for users to respond appropriately in an emergency using only voice instructions. This is especially true when users are stressed or anxious. Furthermore, systems that rely solely on voice data cannot take into account the user's emotional state and lack the support necessary to help the user remain calm. This can reduce the effectiveness of first aid and potentially waste valuable time waiting for an ambulance to arrive.

[0583] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means, a means for analyzing voice data and converting it into text data, a means for analyzing the situation and generating first aid instructions, a means for encoding the generated instructions as voice data, a means for transmitting the voice instruction data to the user terminal, a means for playing the encoded voice instruction data on the user terminal, an emotion recognition means for recognizing the emotional state, and a means for adjusting the instruction content based on the emotion recognition result. As a result, appropriate first aid instructions according to the user's voice and emotional state are provided in real time, allowing the user to respond to emergencies more calmly and effectively.

[0584] The "voice input means" is a device for capturing the user's voice and recording it as digital voice data.

[0585] The "means for analyzing voice data and converting it into text data" refers to a system or software for analyzing captured voice data and converting it into character string data.

[0586] The "means for analyzing the situation and generating first aid instructions" refers to a system or software for determining appropriate first aid based on speech recognition and the situation, and generating the instructions.

[0587] The "means for encoding the generated instructions as audio data" is a system or software for converting the generated first aid instructions into audio data.

[0588] The "means for transmitting voice instruction data to a user terminal" is a communication means for transferring the generated voice data to the user's device.

[0589] The "means for playing back encoded voice instruction data on a user terminal" refers to hardware or software for playing back encoded voice data on a device held by the user.

[0590] The "emotion recognition means for recognizing an emotional state" is an algorithm or software for analyzing and identifying the emotional state of a user from voice data.

[0591] The "means for adjusting instructions based on emotion recognition results" refers to a system or software for modifying and optimizing first aid instructions according to the user's emotional state.

[0592] This invention is a system for responding to emergencies, which supports first aid by utilizing voice input and emotion recognition technology. The specific configuration and operation procedure are shown below.

[0593] The server captures the user's voice through a voice input means. This voice input is performed using the built-in microphone of the user's device, such as a smartphone. The captured voice data is recorded in digital format and sent to the server. The server analyzes the received voice data using voice recognition software (for example, the speech_recognition library) and converts it into text data.

[0594] Based on the text data, the server analyzes the situation and generates appropriate first aid instructions. The AI ​​model used here references historical data and medical knowledge to determine the most appropriate first aid. Based on this determination, the server encodes the instructions into audio data (using a text-to-speech system, for example, the TextToSpeech class).

[0595] The generated voice instruction data is immediately transmitted to the user terminal, which plays back the voice data and provides the user with appropriate first aid instructions.

[0596] Furthermore, an emotion recognition engine (e.g., using the EmotionRecognition module) that recognizes the user's emotional state analyzes the voice data and determines whether the user is feeling stressed. Based on the emotion recognition result, the server adjusts the instructions. For example, if the user is determined to be in a stressful state, the server will include additional encouragement or advice in the voice instructions.

[0597] As a concrete example, consider a scenario in which a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends the data to the server. The server analyzes the voice and generates an instruction to "Start chest compressions" as an emergency. The instruction is encoded as voice data and sent to the user device. When the user starts chest compressions as instructed, the emotion engine continuously monitors the user's voice and, if it determines that the user is in a stressed state, provides additional voice instructions such as "Please stay calm. It's okay. I'm giving you instructions, so please follow them."

[0598] An example of a prompt is "When a user is in an extremely urgent situation and needs help, please generate voice instructions that utilize emotion recognition." This will help users provide first aid as calmly and effectively as possible.

[0599] The present invention reduces stress when a user performs first aid in an emergency, and enables the user to perform correct procedures more reliably.

[0600] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0601] Step 1:

[0602] A user speaks "Help!" into a smartphone. The user's input is voice data, which is captured using the smartphone's built-in microphone. The output is digital voice data.

[0603] Step 2:

[0604] The device transmits the captured audio data to the server. The input is digital audio data, and the output is the audio data transmitted to the server. The data transmission is performed using the device's communication module.

[0605] Step 3:

[0606] The server analyzes the received voice data using a voice recognition engine (for example, the speech_recognition library) and converts it into text data. The input is voice data, and the output is text data. Specifically, the waveform of the voice data is analyzed and data processing is performed to express it as a string of characters.

[0607] Step 4:

[0608] The server analyzes the situation based on the converted text data. The AI ​​model used is a generative AI model, which refers to past data and medical knowledge to determine the most appropriate first aid treatment. The input is text data, and the output is first aid instruction data. For data calculation, the generative AI model performs an analytical process on the text data and generates first aid instructions.

[0609] Step 5:

[0610] The server encodes the generated first aid instructions as audio data. The input is the first aid instructions and the output is the encoded audio data. The software used is a text-to-speech system (e.g., TextToSpeech class). At this stage, the instructions are prepared to be transmitted to the user as audio.

[0611] Step 6:

[0612] The server transmits the encoded voice instruction data to the user terminal. The input is the encoded voice data, and the output is the voice data transmitted to the user terminal. The data transmission is performed using a communication module.

[0613] Step 7:

[0614] The device plays the received audio data and provides the user with appropriate first aid instructions. The input is the encoded audio data, and the output is audio instructions that the user can hear. The audio instructions are played using the device's audio playback capabilities.

[0615] Step 8:

[0616] The server analyzes the user's emotional state from the voice data using an emotion recognition engine to recognize the user's emotional state. The input is the newly captured voice data and the output is the emotional state data. The voice analysis is performed using an emotion recognition engine (e.g., EmotionRecognition module).

[0617] Step 9:

[0618] The server adjusts the instructions based on the emotion recognition results. The input is emotional state data and first aid instruction data, and the output is the adjusted first aid instruction data. Specifically, if the user is in a stressful state, the server includes encouragement and advice in addition to the first aid instruction data.

[0619] Step 10:

[0620] The server re-encodes the adjusted first aid instruction data as voice data and transmits it to the user terminal. The input is the adjusted instruction data, and the output is the re-encoded voice data. The data is again transmitted using the communication module, and the user terminal plays back the voice data.

[0621] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0622] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0623] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0624] [Third embodiment]

[0625] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0626] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0627] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0628] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0629] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0630] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0631] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0632] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0633] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0634] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0635] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0636] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0637] This invention relates to a system that uses voice recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input.

[0638] First, the user launches the "RescueVoice" application on their smartphone. The system starts up when the user says "Help me!" The user's device captures the voice using the built-in microphone and generates voice data. The generated voice data is immediately sent to the server.

[0639] On the server side, a voice recognition engine analyzes the received voice data and converts it into text data. Based on this text data, an AI model analyzes the situation. For example, if someone says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[0640] Next, the first aid instructions generated by the server are encoded as voice data. The encoded voice data is immediately sent to the user's device. The user's device plays back the received voice data and delivers it to the user. The user begins first aid according to the instructions. For example, the instructions might be, "Start chest compressions. Place your hands together in the center of the chest and compress at a rate of 100-120 times per minute."

[0641] The system can also monitor the progress of first aid and provide additional instructions as needed. For example, if a user says, "I went to get the AED," new voice data is sent to the server, which immediately analyzes it. As a result, additional instructions such as "Prepare to use the AED. Attach the pads to the patient's chest" are generated and encoded as voice data. The user device plays back the voice data and provides the user with additional instructions.

[0642] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice and provides additional instructions as needed. If an AED is available, the system also provides specific instructions on how to use it. As a result, the chances of appropriate first aid being administered and saving a life are significantly increased before an ambulance arrives.

[0643] This system allows even ordinary users without specialized knowledge to provide first aid quickly and accurately, enabling important life-saving measures to be taken before an ambulance arrives.

[0644] The processing flow will be explained below.

[0645] Step 1:

[0646] The user launches the "RescueVoice" app on their smartphone, which puts the app into a voice input standby state.

[0647] Step 2:

[0648] The user says "Help!" The device uses the built-in microphone to capture the user's voice and generate digital audio data.

[0649] Step 3:

[0650] The audio data generated by the device is sent to the server using the secure HTTP(S) protocol.

[0651] Step 4:

[0652] The server passes the received voice data to a speech recognition engine, which converts the voice data into text data. This speech recognition engine uses a highly accurate generative AI model.

[0653] Step 5:

[0654] The server passes the text data to an AI model, which analyzes the situation. For example, it analyzes text such as "My friend has collapsed" and determines whether the situation is urgent.

[0655] Step 6:

[0656] The server generates appropriate first aid instructions based on the analysis results. For example, if cardiac arrest is suspected, it generates instructions such as, "Start chest compressions immediately. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[0657] Step 7:

[0658] The server encodes the generated instructions as voice data and transmits the voice data to the user terminal.

[0659] Step 8:

[0660] The device plays the received voice data and delivers it to the user, who then follows the instructions to begin providing first aid.

[0661] Step 9:

[0662] The device continuously monitors the user's voice input. For example, if the user says, "Help isn't here yet," the device retransmits the voice to the server.

[0663] Step 10:

[0664] The server receives the new voice data, analyzes it again, and generates additional first aid instructions based on the new situation, such as "Continue chest compressions and use an AED if one is available."

[0665] Step 11:

[0666] The server then transmits the encoded audio data back to the user terminal, which then plays the audio back to the user.

[0667] Step 12:

[0668] The user follows additional instructions to administer any necessary first aid, such as attaching pads to the patient's chest to use an AED.

[0669] Through this series of processing steps, the user receives appropriate first aid instructions in real time, enabling them to take effective life-saving measures until an ambulance arrives.

[0670] Example 1

[0671] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0672] In modern society, emergencies can occur due to sudden illness or accidents. Responding to these emergencies requires prompt and appropriate first aid, but the general public often does not know how to provide it. To solve this problem, a system is needed that allows anyone to provide first aid easily, even without specialized knowledge. It also needs to be able to respond to multiple situations and determine the appropriate course of action.

[0673] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0674] In this invention, the server includes means for analyzing voice data and converting it into text data, means for analyzing the situation based on the text data and generating first aid instructions, and means for encoding the generated first aid instructions as voice data. This allows a user to receive appropriate first aid instructions in real time simply by calling for help by voice. The system can also monitor the progress of the first aid and provide additional instructions as needed, allowing it to respond to multiple situations and provide optimal treatment.

[0675] The "voice input means" is a device that has the function of taking in the user's voice and generating it as digital voice data.

[0676] The "means for generating audio data and transmitting it to a server" is a device or software that has the function of transmitting captured audio data to a server using an appropriate communication protocol.

[0677] The "means for analyzing voice data and converting it into text data" refers to a software algorithm or device for analyzing received voice data and generating the content of the voice data as text data.

[0678] "Means for analyzing a situation based on text data and generating first aid instructions" refers to software or a device that uses an AI model or rule-based system to analyze a situation based on text data and issue appropriate first aid instructions.

[0679] The "means for encoding the generated first aid instructions as audio data" is software or a device for converting the text-based first aid instructions back into audio data.

[0680] The "means for transmitting encoded voice data to a user terminal" is a device or software that has the function of transmitting encoded voice data to a user terminal using an appropriate communication protocol.

[0681] The "means for reproducing audio data received on a user terminal" refers to a device or software having the function of reproducing audio data received by a user terminal and providing audio instructions to the user.

[0682] This invention is a system that supports first aid in emergencies using voice recognition and an AI model. The system includes a voice input unit, a voice analysis unit, a situation analysis unit, a first aid instruction generation unit, and a voice output unit.

[0683] System configuration and operation

[0684] Hardware and Software Configuration

[0685] User terminal: A mobile information terminal such as a smartphone. It uses a built-in microphone to capture voice input and generate voice data. It also has a speaker to play back voice instruction data from the server.

[0686] Server: A powerful computer system that includes a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze and convert audio data into text, and a generative AI model (e.g., OpenAI's GPT-4) to analyze the situation and generate first aid instructions.

[0687] Program processing explanation

[0688] Voice Recognition

[0689] In an emergency, a user launches the "RescueVoice" application on their smartphone and utters the command "Help!" The user's device's built-in microphone captures this voice and converts it into digital audio data, which is then sent to a server via the Internet.

[0690] Analysis of audio data

[0691] The server uses a speech recognition engine to analyze the received voice data and convert it into text data. For example, if a user says, "Help me, my friend has fallen down," the voice data is converted into text data.

[0692] Analyzing the situation and generating first aid instructions

[0693] The converted text data is then analyzed by a generative AI model to understand the situation. The AI ​​model then references past data and medical knowledge to generate the most appropriate first aid instructions. For example, it might generate specific instructions such as, "If cardiac arrest is suspected, begin chest compressions."

[0694] Encoding and transmitting first aid instructions as audio data

[0695] The server encodes the generated emergency treatment instructions as voice data, which is then transmitted to the user terminal via the Internet.

[0696] Playing voice instructions

[0697] The user device plays the received voice data and conveys instructions to the user, such as "Please begin chest compressions. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[0698] Monitor progress and generate further instructions

[0699] The system continuously monitors the user's progress in administering first aid and generates additional instructions as needed. For example, if the user says, "I went to get the AED," new voice data is sent to the server for analysis. As a result, additional instructions are generated and encoded as voice data, such as, "Prepare the AED. Attach the pads to the patient's chest."

[0700] Examples of concrete examples and prompts

[0701] Specific examples

[0702] If a friend suddenly collapses:

[0703] 1. User: Launch the "RescueVoice" application on their smartphone.

[0704] 2. User: Says "Help!"

[0705] 3. Device: Captures audio with the built-in microphone and sends the audio data to the server.

[0706] 4. Server: The speech recognition engine converts the speech into text, and the AI ​​model determines the situation: "My friend has collapsed."

[0707] 5. Server: Generates the instruction "Please begin chest compressions" and encodes it as audio data.

[0708] 6. Terminal: Plays the received audio data and delivers it to the user.

[0709] 7. User: Begin chest compressions as instructed.

[0710] Prompt Sentence Examples

[0711] "If a user is in an emergency and needs first aid, design a system that provides appropriate instructions in real time based on voice input."

[0712] "Imagine a scenario in which a user uses a voice recognition system to provide first aid to a friend who has suddenly collapsed, and explain the specific instructions that would be given."

[0713] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0714] Step 1: The user launches the "RescueVoice" application on their smartphone. This launches the app and makes it ready to accept voice input. When the user says "Help!", the built-in microphone captures the voice. The input is the user's voice, and the output is voice data.

[0715] Step 2: The device generates the captured voice data and sends it to the server via the Internet. The specific operations at this stage are the process from voice capture to data transmission. The input is the voice data, and the output is the voice data sent to the server.

[0716] Step 3: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze the received voice data. This engine converts the voice data into text data. The specific data processing involves analyzing the voice data and converting it into text. The input is voice data, and the output is text data.

[0717] Step 4: The server performs situation analysis based on the generated text data. Here, a generative AI model (e.g., OpenAI's GPT-4) is used to determine the situation from the text. This analysis identifies an emergency situation and generates appropriate first aid instructions. The input is text data, and the output is first aid instructions.

[0718] Step 5: The server uses a text-to-speech engine to encode the generated first aid instructions as audio data. The specific operation is to convert the text data into an audio file. The input is the first aid text instructions, and the output is the audio data.

[0719] Step 6: The server sends the encoded voice data to the user terminal via the Internet. The specific operation at this stage is to send data from the server and receive it at the user terminal. The input is the voice data, and the output is the voice data received at the terminal.

[0720] Step 7: The device plays the received voice data and delivers instructions to the user. The user then performs the instructed first aid. Specifically, the instruction to the user is "Please begin chest compressions." The input is the voice data, and the output is the played voice instructions.

[0721] Step 8: The device monitors the user's progress in providing first aid and continuously captures new voice input. For example, when the user says, "I went to get the AED," the voice data is sent to the server again. The input is the user's new voice, and the output is the regenerated voice data.

[0722] Step 9: The server analyzes the newly received voice data and generates additional first aid instructions as needed. It then encodes these instructions as voice data and sends them to the user terminal. For example, the generated instruction might be, "Prepare to use the AED. Attach the pads to the patient's chest." The input is the new voice data, and the output is the voice data of the additional first aid instructions.

[0723] (Application example 1)

[0724] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0725] In the event of an emergency at a factory, it is difficult for employees to provide first aid quickly and accurately. For this reason, there is a need for a system that can immediately provide appropriate first aid instructions in an emergency and improve safety. In particular, a rapid response is required when a worker collapses, and there is a need for a method to solve this problem using robots.

[0726] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0727] In this invention, the server includes a voice input means, a means for analyzing voice data and converting it into text data, a means for analyzing the situation and generating first aid instructions, a means for encoding the generated instructions as voice data, a means for transmitting the voice instruction data to a user terminal, a means for playing the encoded voice instruction data on the user terminal, a means for the robot to capture voice using a built-in microphone when an emergency occurs and transmit the voice data to the server, and a means for the robot to receive and play the first aid instruction voice data transmitted from the server. This enables the robot to instantly provide appropriate first aid instructions in the event of an emergency at a factory site, enabling a fast and effective response.

[0728] "Audio input means" refers to hardware and software for capturing audio data.

[0729] "Means for analyzing voice data and converting it into text data" refers to the function of converting voice data into text using voice recognition technology.

[0730] The "means for analyzing the situation and generating first aid instructions" refers to a function for analyzing the emergency situation based on input voice data and generating appropriate first aid instructions.

[0731] "Means for encoding the generated instructions as voice data" refers to speech synthesis technology for converting text data into voice data.

[0732] The "means for transmitting voice instruction data to the user terminal" refers to a communication means for transmitting the generated voice data to the user terminal.

[0733] The "means for reproducing the encoded voice instruction data on the user terminal" refers to a function for reproducing the received voice instruction data on the user terminal.

[0734] "Means for the robot to capture audio using its built-in microphone in the event of an emergency and transmit that audio data to a server" refers to the function of transmitting audio data in an emergency to a server using the robot's microphone and communication function.

[0735] "Means for the robot to receive and play back the voice data of first aid instructions sent from the server" refers to the function of the robot to play back the voice data of first aid instructions received from the server.

[0736] The present invention relates to a system that provides effective first aid instructions for emergencies that occur at factory sites. This system aims to improve worker safety by using a robot to input voice and provide appropriate first aid in emergencies.

[0737] The robot uses a built-in microphone as a means of voice input. This microphone captures surrounding sounds and sends the data to a server. If the robot detects an emergency, for example, if a worker yells "Help!", the captured voice data is immediately sent to the server.

[0738] On the server side, speech recognition software (e.g., Google's speech recognition API) is used to convert the speech data into text data, which is then analyzed by an AI model (e.g., a generative AI model) to understand the emergency situation and generate appropriate first aid instructions.

[0739] The generated first aid instructions are encoded as voice data using speech synthesis technology, using speech synthesis software (e.g., the pydub library), and the encoded voice data is sent to the robot, which plays it back, providing the appropriate first aid instructions to the worker in real time.

[0740] As a concrete example, consider the case where a worker collapses in a factory. When the worker says "Help!", the robot's built-in microphone captures the voice and sends the voice data to the server. The server analyzes the voice and, if it determines that cardiopulmonary resuscitation is necessary, generates the instruction "Please begin chest compressions." This instruction is encoded as voice data and sent back to the robot. The robot then plays back this instruction and provides the worker with specific first aid instructions, enabling a rapid response.

[0741] As an example, the following prompt is presented:

[0742] "A worker falls and cries out, 'Help!' This voice is captured by the robot's microphone, and appropriate first aid instructions are provided in real time."

[0743] In this way, the present invention allows for a fast and effective response to emergencies on the factory floor, greatly improving worker safety.

[0744] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0745] Step 1:

[0746] The robot uses a built-in microphone to capture surrounding sounds. The input is the worker's speech, and the output is audio data. Specifically, the microphone captures the voice saying "Help!" and stores the audio data in the robot's memory.

[0747] Step 2:

[0748] The robot sends the captured audio data to the server. The input is the audio data obtained in step 1, and the output is the audio data sent to the server. Specifically, the robot uploads the audio data to the server via an internet connection.

[0749] Step 3:

[0750] The server analyzes the received voice data and converts it into text data. The input is voice data and the output is text data. Specifically, the server uses voice recognition software (for example, Google's voice recognition API) to analyze the voice data and convert it into text data such as "Help!"

[0751] Step 4:

[0752] The server analyzes the situation based on the text data and generates appropriate first aid instructions. The input is text data, and the output is first aid instructions. Specifically, the server analyzes the text data using a generative AI model and generates the instruction "Please begin chest compressions."

[0753] Step 5:

[0754] The server encodes the generated first aid instructions as audio data. The input is the first aid instructions, and the output is the encoded audio data. Specifically, the server converts the instructions into audio data using speech synthesis software (e.g., the pydub library).

[0755] Step 6:

[0756] The server sends the encoded voice data to the robot. The input is the encoded voice data, and the output is the voice data sent to the robot. Specifically, the server uploads the encoded voice data to the robot via an internet connection.

[0757] Step 7:

[0758] The robot receives the transmitted voice data and plays it back to the worker. The input is encoded voice data, and the output is the played-back voice instruction. Specifically, the robot plays the received voice data through its built-in speaker and tells the worker to "start chest compressions."

[0759] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0760] This invention relates to a system that uses voice recognition and emotion recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input and emotion recognition.

[0761] First, the user launches the "RescueVoice" application on their smartphone. The system starts up when the user says "Help!" The user's device captures the voice using a built-in microphone and generates digital voice data, which is then immediately sent to the server.

[0762] On the server side, a voice recognition engine analyzes the received voice data and converts it into text data. Based on this text data, an AI model analyzes the situation. For example, if someone says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[0763] Next, the first aid instructions generated by the server are encoded as voice data. The encoded voice data is immediately sent to the user's device. The user's device plays back the received voice data and delivers it to the user. The user begins first aid according to the instructions. For example, the instructions might be, "Start chest compressions. Place your hands together in the center of the chest and compress at a rate of 100-120 times per minute."

[0764] Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions. The emotion engine analyzes voice data and user behavior data to identify the user's emotional state. For example, if the user is very anxious, the emotion engine will determine that the user is in a stressed state.

[0765] Based on the user's emotional state recognized by the emotion engine, the server adjusts the instructions. For example, if a user is under stress, the server can provide additional encouragement and advice to encourage them to remain calm and administer first aid. A voice instruction such as "Please stay calm. It's okay. I'll give you instructions, so please follow them," is generated, encoded, and sent to the user's device.

[0766] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice. If the emotion engine determines that the user is under stress, it provides additional encouragement such as "Please stay calm. Your actions will help."

[0767] Thus, by combining voice input and emotion recognition, the system of the present invention helps users administer first aid more calmly and effectively, providing vital life-saving assistance until an ambulance arrives.

[0768] The processing flow will be explained below.

[0769] Step 1:

[0770] The user launches the "RescueVoice" app on their smartphone, which puts the app into a voice input standby state.

[0771] Step 2:

[0772] The user says "Help!" The device uses the built-in microphone to capture the user's voice and generate digital audio data.

[0773] Step 3:

[0774] The audio data generated by the device is sent to the server using the secure HTTP(S) protocol.

[0775] Step 4:

[0776] The server passes the received voice data to a speech recognition engine, which converts the voice data into text data. This speech recognition engine uses a highly accurate generative AI model.

[0777] Step 5:

[0778] The server passes the text data to an AI model, which analyzes the situation. For example, it analyzes text such as "My friend has collapsed" and determines whether the situation is urgent.

[0779] Step 6:

[0780] The server generates appropriate first aid instructions based on the analysis results. For example, if cardiac arrest is suspected, it generates instructions such as, "Start chest compressions immediately. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[0781] Step 7:

[0782] The server encodes the generated instructions as voice data and transmits the voice data to the user terminal.

[0783] Step 8:

[0784] The device plays the received voice data and delivers it to the user, who then follows the instructions to begin providing first aid.

[0785] Step 9:

[0786] The device continuously monitors the user's voice input. For example, if the user says, "Help isn't here yet," the device retransmits the voice to the server.

[0787] Step 10:

[0788] The server receives the new voice data, analyzes it again, and generates additional first aid instructions based on the new situation, such as "Continue chest compressions and use an AED if one is available."

[0789] Step 11:

[0790] The server then transmits the encoded audio data back to the user terminal, which then plays the audio back to the user.

[0791] Step 12:

[0792] The user follows additional instructions to administer any necessary first aid, such as attaching pads to the patient's chest to use an AED.

[0793] Step 13:

[0794] The server sends the voice data and behavioral data to the emotion engine, which analyzes the user's emotional state and determines whether the user is stressed or anxious.

[0795] Step 14:

[0796] The server adjusts the instructions based on the analysis results of the emotion engine. For example, if it determines that the user is under stress, it adds encouragement and advice to stabilize the user, such as "Please stay calm. It's okay. I'll give you instructions, so please act as instructed."

[0797] Step 15:

[0798] The server encodes the adjusted instructions as voice data and transmits it to the user terminal.

[0799] Step 16:

[0800] The device then plays back the received audio data and delivers it to the user, who then follows the instructions to continue providing first aid.

[0801] Through this series of processing steps, the user receives appropriate first aid instructions in real time, as well as encouragement and advice tailored to their emotional state, enabling them to carry out effective life-saving measures until an ambulance arrives.

[0802] Example 2

[0803] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0804] Conventional first aid support systems use voice input and output instructions, but they are unable to consider the user's emotional state, which can lead to users becoming impatient or anxious and being unable to act appropriately in an emergency. In particular, when a user is under stress, there is a high risk that they will lose their composure and make a serious, life-threatening mistake. In addition, when multiple situations overlap, it is sometimes difficult to properly determine priorities.

[0805] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0806] In this invention, the server includes a voice input unit, a unit for analyzing voice data and converting it into text data, a unit for analyzing a situation based on the text data and generating appropriate first aid instructions, a unit for encoding the generated first aid instructions as voice data, a unit for transmitting the encoded voice instruction data to a user terminal, a unit for playing the encoded voice instruction data on the user terminal, a unit for analyzing the voice data and behavioral data to recognize the user's emotional state, and a unit for adjusting the first aid instructions based on the user's emotional state and generating additional instructions or encouragement. This allows the user to provide first aid appropriately and calmly in an emergency. Furthermore, even when multiple situations arise, the server can determine priorities and provide optimal instructions.

[0807] "Audio input means" refers to a device or function that captures a user's voice and converts it into digital audio data.

[0808] The "means for analyzing voice data and converting it into text data" refers to software or hardware for analyzing acquired voice data and converting it into character string data.

[0809] The "means for analyzing the situation and generating appropriate first aid instructions" is a device or function that judges the situation based on text data and automatically generates optimal first aid instructions.

[0810] The "means for encoding the generated first aid instructions as audio data" refers to software or hardware for converting the generated text format first aid instructions into audio data.

[0811] The "means for transmitting encoded voice instruction data to a user terminal" is a function or system that transmits encoded voice data to a device held by a user via a network.

[0812] The "means for reproducing encoded voice instruction data on the user terminal" refers to a system or function for reproducing voice data sent to the user terminal and conveying instructions to the user.

[0813] The "means for recognizing the emotional state of a user by analyzing voice data and behavioral data" refers to software or a device for analyzing the voice data and behavioral data of a user and automatically recognizing the emotional state.

[0814] "Means for adjusting first aid instructions and generating additional instructions or encouragement based on emotional state" refers to a system or function for adjusting the content of first aid instructions based on the user's emotional state and generating further instructions or encouragement as needed.

[0815] This invention relates to a system that uses voice recognition and emotion recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input and emotion recognition.

[0816] First, the user launches a dedicated application on their smartphone. The system starts up when the user says "Help!" The user's device captures the voice using a built-in microphone and generates digital voice data. This generated voice data is immediately sent to the server.

[0817] On the server side, a speech recognition engine (e.g., Google Speech-to-Text API) analyzes the received voice data and converts it into text data. Based on this text data, an AI model (e.g., OpenAI GPT-3) analyzes the situation. For example, if a voice input says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[0818] Next, a speech synthesis engine (e.g., Amazon Polly) is used to encode the first aid instructions generated by the server as voice data. This encoded voice data is immediately sent to the user's device. The user's device plays the received voice data and delivers it to the user. The user can start first aid by following the instructions. For example, the voice instruction is "Start chest compressions. Place your hands together at the center of the chest and compress at a rate of 100-120 times per minute."

[0819] Furthermore, this system incorporates an emotion engine (e.g., Affectiva's emotion recognition API) that recognizes the user's emotions. The emotion engine analyzes voice data and user behavior data to identify the user's emotional state. For example, if the user is very anxious, the emotion engine will determine that the user is in a state of stress.

[0820] Based on the user's emotional state recognized by the emotion engine, the server adjusts the instructions. For example, if a user is under stress, the server can provide additional encouragement and advice to encourage them to remain calm and administer first aid. A voice instruction such as "Please stay calm. It's okay. I'll give you instructions, so please follow them," is generated, encoded, and sent to the user's device.

[0821] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the user device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice. If the emotion engine determines that the user is under stress, it provides additional encouragement such as "Please stay calm. Your actions will help."

[0822] Examples of prompts for the generative AI model include, "The user says that their friend has collapsed. Please suggest the best first aid for this situation." or "The user seems very anxious. Please think of some encouraging words to calm the user in this situation."

[0823] In this way, the combination of voice input and real-time emotion recognition technology provides a system that allows users to respond calmly and effectively in an emergency, providing vital life-saving assistance until an ambulance arrives.

[0824] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0825] Step 1:

[0826] The user launches a dedicated application on their smartphone and says, "Help!"

[0827] Input: User's voice

[0828] Output: Captured digital audio data

[0829] Specific actions: The user taps the icon on their smartphone to launch the app and utters "Help!" in an emergency.

[0830] Step 2:

[0831] The device uses a built-in microphone to capture the user's voice and convert it into digital audio data.

[0832] Input: User's voice

[0833] Output: Digital audio data

[0834] Specific operation: The smartphone's microphone picks up the user's speech and digitizes it as voice data.

[0835] Step 3:

[0836] The terminal transmits the generated digital voice data to the server via the network.

[0837] Input: Digital audio data

[0838] Output: Audio data sent to the server

[0839] What happens: The device sends audio data to the server via an internet connection.

[0840] Step 4:

[0841] The server uses a speech recognition engine (e.g., Google Speech-to-Text API) to analyze the received voice data and convert it into text data.

[0842] Input: Audio data sent to the server

[0843] Output: Text data

[0844] Specific operation: The server calls the Google Speech-to-Text API and converts the audio data into text.

[0845] Step 5:

[0846] The server analyzes the situation using an AI model (e.g., OpenAI GPT-3) based on the text data and generates first aid instructions.

[0847] Input: Text data

[0848] Output: First aid instruction text

[0849] How it works: The server inputs text data into the GPT-3 model and generates first aid instructions such as "Begin chest compressions."

[0850] Step 6:

[0851] Use a speech synthesis engine (e.g., Amazon Polly) to encode the server-generated first aid instructions as audio data.

[0852] Input: First Aid Instructions Text

[0853] Output: Audio data

[0854] What happens: The server calls Amazon Polly to convert the text data into speech.

[0855] Step 7:

[0856] The server transmits the encoded voice instruction data to the user terminal via the network.

[0857] Input: Audio data

[0858] Output: Audio data sent to the user's device

[0859] Specific operation: The server sends encoded voice data to the user's terminal via the Internet.

[0860] Step 8:

[0861] The terminal plays back the received audio data and conveys emergency instructions to the user.

[0862] Input: Audio data sent to the user's device

[0863] Output: The audio instructions played

[0864] Specific operation: The device speaker plays audio data and gives the user the voice instruction, "Please begin chest compressions."

[0865] Step 9:

[0866] The server uses an emotion engine (e.g., Affectiva's emotion recognition API) to analyze the user's voice data and behavioral data to identify the user's emotional state.

[0867] Input: User voice data, behavioral data

[0868] Output: User's emotional state data

[0869] How it works: The server calls Affectiva's API, analyzes the user's voice and behavioral data, and identifies their stress level.

[0870] Step 10:

[0871] The server adjusts first aid instructions and generates additional instructions and encouragement based on the perceived emotional state.

[0872] Input: User emotional state data

[0873] Output: Adjusted first aid instruction text, additional encouragement text

[0874] Specific actions: The AI ​​model generates additional instructions such as, "Please stay calm. It's okay. I'll give you instructions, so please follow them," and encodes them as voice data and sends them to the user's device.

[0875] Step 11:

[0876] The terminal plays back the received additional audio data to provide encouragement or advice to the user.

[0877] Input: Adjusted first aid instruction audio data, additional encouragement audio data

[0878] Output: Played back audio instructions of encouragement and advice

[0879] Specific action: The device speaker plays audio data and conveys an encouraging message to the user to stay calm.

[0880] By using the specific processing steps described above, this system supports the user in providing appropriate and calm first aid in an emergency.

[0881] (Application example 2)

[0882] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0883] Conventional first aid support systems can sometimes make it difficult for users to respond appropriately in an emergency using only voice instructions. This is especially true when users are stressed or anxious. Furthermore, systems that rely solely on voice data cannot take into account the user's emotional state and lack the support necessary to help the user remain calm. This can reduce the effectiveness of first aid and potentially waste valuable time waiting for an ambulance to arrive.

[0884] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means, a means for analyzing voice data and converting it into text data, a means for analyzing the situation and generating first aid instructions, a means for encoding the generated instructions as voice data, a means for transmitting the voice instruction data to the user terminal, a means for playing the encoded voice instruction data on the user terminal, an emotion recognition means for recognizing the emotional state, and a means for adjusting the instruction content based on the emotion recognition result. As a result, appropriate first aid instructions according to the user's voice and emotional state are provided in real time, allowing the user to respond to emergencies more calmly and effectively.

[0885] The "voice input means" is a device for capturing the user's voice and recording it as digital voice data.

[0886] The "means for analyzing voice data and converting it into text data" refers to a system or software for analyzing captured voice data and converting it into character string data.

[0887] The "means for analyzing the situation and generating first aid instructions" refers to a system or software for determining appropriate first aid based on speech recognition and the situation, and generating the instructions.

[0888] The "means for encoding the generated instructions as audio data" is a system or software for converting the generated first aid instructions into audio data.

[0889] The "means for transmitting voice instruction data to a user terminal" is a communication means for transferring the generated voice data to the user's device.

[0890] The "means for playing back encoded voice instruction data on a user terminal" refers to hardware or software for playing back encoded voice data on a device held by the user.

[0891] The "emotion recognition means for recognizing an emotional state" is an algorithm or software for analyzing and identifying the emotional state of a user from voice data.

[0892] The "means for adjusting instructions based on emotion recognition results" refers to a system or software for modifying and optimizing first aid instructions according to the user's emotional state.

[0893] This invention is a system for responding to emergencies, which supports first aid by utilizing voice input and emotion recognition technology. The specific configuration and operation procedure are shown below.

[0894] The server captures the user's voice through a voice input means. This voice input is performed using the built-in microphone of the user's device, such as a smartphone. The captured voice data is recorded in digital format and sent to the server. The server analyzes the received voice data using voice recognition software (for example, the speech_recognition library) and converts it into text data.

[0895] Based on the text data, the server analyzes the situation and generates appropriate first aid instructions. The AI ​​model used here references historical data and medical knowledge to determine the most appropriate first aid. Based on this determination, the server encodes the instructions into audio data (using a text-to-speech system, for example, the TextToSpeech class).

[0896] The generated voice instruction data is immediately transmitted to the user terminal, which plays back the voice data and provides the user with appropriate first aid instructions.

[0897] Furthermore, an emotion recognition engine (e.g., using the EmotionRecognition module) that recognizes the user's emotional state analyzes the voice data and determines whether the user is feeling stressed. Based on the emotion recognition result, the server adjusts the instructions. For example, if the user is determined to be in a stressful state, the server will include additional encouragement or advice in the voice instructions.

[0898] As a concrete example, consider a scenario in which a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends the data to the server. The server analyzes the voice and generates an instruction to "Start chest compressions" as an emergency. The instruction is encoded as voice data and sent to the user device. When the user starts chest compressions as instructed, the emotion engine continuously monitors the user's voice and, if it determines that the user is in a stressed state, provides additional voice instructions such as "Please stay calm. It's okay. I'm giving you instructions, so please follow them."

[0899] An example of a prompt is "When a user is in an extremely urgent situation and needs help, please generate voice instructions that utilize emotion recognition." This will help users provide first aid as calmly and effectively as possible.

[0900] The present invention reduces stress when a user performs first aid in an emergency, and enables the user to perform correct procedures more reliably.

[0901] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0902] Step 1:

[0903] A user speaks "Help!" into a smartphone. The user's input is voice data, which is captured using the smartphone's built-in microphone. The output is digital voice data.

[0904] Step 2:

[0905] The device transmits the captured audio data to the server. The input is digital audio data, and the output is the audio data transmitted to the server. The data transmission is performed using the device's communication module.

[0906] Step 3:

[0907] The server analyzes the received voice data using a voice recognition engine (for example, the speech_recognition library) and converts it into text data. The input is voice data, and the output is text data. Specifically, the waveform of the voice data is analyzed and data processing is performed to express it as a string of characters.

[0908] Step 4:

[0909] The server analyzes the situation based on the converted text data. The AI ​​model used is a generative AI model, which refers to past data and medical knowledge to determine the most appropriate first aid treatment. The input is text data, and the output is first aid instruction data. For data calculation, the generative AI model performs an analytical process on the text data and generates first aid instructions.

[0910] Step 5:

[0911] The server encodes the generated first aid instructions as audio data. The input is the first aid instructions and the output is the encoded audio data. The software used is a text-to-speech system (e.g., TextToSpeech class). At this stage, the instructions are prepared to be transmitted to the user as audio.

[0912] Step 6:

[0913] The server transmits the encoded voice instruction data to the user terminal. The input is the encoded voice data, and the output is the voice data transmitted to the user terminal. The data transmission is performed using a communication module.

[0914] Step 7:

[0915] The device plays the received audio data and provides the user with appropriate first aid instructions. The input is the encoded audio data, and the output is audio instructions that the user can hear. The audio instructions are played using the device's audio playback capabilities.

[0916] Step 8:

[0917] The server analyzes the user's emotional state from the voice data using an emotion recognition engine to recognize the user's emotional state. The input is the newly captured voice data and the output is the emotional state data. The voice analysis is performed using an emotion recognition engine (e.g., EmotionRecognition module).

[0918] Step 9:

[0919] The server adjusts the instructions based on the emotion recognition results. The input is emotional state data and first aid instruction data, and the output is the adjusted first aid instruction data. Specifically, if the user is in a stressful state, the server includes encouragement and advice in addition to the first aid instruction data.

[0920] Step 10:

[0921] The server re-encodes the adjusted first aid instruction data as voice data and transmits it to the user terminal. The input is the adjusted instruction data, and the output is the re-encoded voice data. The data is again transmitted using the communication module, and the user terminal plays back the voice data.

[0922] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0923] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0924] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0925] [Fourth embodiment]

[0926] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0927] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0928] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0929] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0930] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0931] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0932] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0933] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0934] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0935] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0936] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0937] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0938] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0939] This invention relates to a system that uses voice recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input.

[0940] First, the user launches the "RescueVoice" application on their smartphone. The system starts up when the user says "Help me!" The user's device captures the voice using the built-in microphone and generates voice data. The generated voice data is immediately sent to the server.

[0941] On the server side, a voice recognition engine analyzes the received voice data and converts it into text data. Based on this text data, an AI model analyzes the situation. For example, if someone says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[0942] Next, the first aid instructions generated by the server are encoded as voice data. The encoded voice data is immediately sent to the user's device. The user's device plays back the received voice data and delivers it to the user. The user begins first aid according to the instructions. For example, the instructions might be, "Start chest compressions. Place your hands together in the center of the chest and compress at a rate of 100-120 times per minute."

[0943] The system can also monitor the progress of first aid and provide additional instructions as needed. For example, if a user says, "I went to get the AED," new voice data is sent to the server, which immediately analyzes it. As a result, additional instructions such as "Prepare to use the AED. Attach the pads to the patient's chest" are generated and encoded as voice data. The user device plays back the voice data and provides the user with additional instructions.

[0944] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice and provides additional instructions as needed. If an AED is available, the system also provides specific instructions on how to use it. As a result, the chances of appropriate first aid being administered and saving a life are significantly increased before an ambulance arrives.

[0945] This system allows even ordinary users without specialized knowledge to provide first aid quickly and accurately, enabling important life-saving measures to be taken before an ambulance arrives.

[0946] The processing flow will be explained below.

[0947] Step 1:

[0948] The user launches the "RescueVoice" app on their smartphone, which puts the app into a voice input standby state.

[0949] Step 2:

[0950] The user says "Help!" The device uses the built-in microphone to capture the user's voice and generate digital audio data.

[0951] Step 3:

[0952] The audio data generated by the device is sent to the server using the secure HTTP(S) protocol.

[0953] Step 4:

[0954] The server passes the received voice data to a speech recognition engine, which converts the voice data into text data. This speech recognition engine uses a highly accurate generative AI model.

[0955] Step 5:

[0956] The server passes the text data to an AI model, which analyzes the situation. For example, it analyzes text such as "My friend has collapsed" and determines whether the situation is urgent.

[0957] Step 6:

[0958] The server generates appropriate first aid instructions based on the analysis results. For example, if cardiac arrest is suspected, it generates instructions such as, "Start chest compressions immediately. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[0959] Step 7:

[0960] The server encodes the generated instructions as voice data and transmits the voice data to the user terminal.

[0961] Step 8:

[0962] The device plays the received voice data and delivers it to the user, who then follows the instructions to begin providing first aid.

[0963] Step 9:

[0964] The device continuously monitors the user's voice input. For example, if the user says, "Help isn't here yet," the device retransmits the voice to the server.

[0965] Step 10:

[0966] The server receives the new voice data, analyzes it again, and generates additional first aid instructions based on the new situation, such as "Continue chest compressions and use an AED if one is available."

[0967] Step 11:

[0968] The server then transmits the encoded audio data back to the user terminal, which then plays the audio back to the user.

[0969] Step 12:

[0970] The user follows additional instructions to administer any necessary first aid, such as attaching pads to the patient's chest to use an AED.

[0971] Through this series of processing steps, the user receives appropriate first aid instructions in real time, enabling them to take effective life-saving measures until an ambulance arrives.

[0972] Example 1

[0973] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0974] In modern society, emergencies can occur due to sudden illness or accidents. Responding to these emergencies requires prompt and appropriate first aid, but the general public often does not know how to provide it. To solve this problem, a system is needed that allows anyone to provide first aid easily, even without specialized knowledge. It also needs to be able to respond to multiple situations and determine the appropriate course of action.

[0975] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0976] In this invention, the server includes means for analyzing voice data and converting it into text data, means for analyzing the situation based on the text data and generating first aid instructions, and means for encoding the generated first aid instructions as voice data. This allows a user to receive appropriate first aid instructions in real time simply by calling for help by voice. The system can also monitor the progress of the first aid and provide additional instructions as needed, allowing it to respond to multiple situations and provide optimal treatment.

[0977] The "voice input means" is a device that has the function of taking in the user's voice and generating it as digital voice data.

[0978] The "means for generating audio data and transmitting it to a server" is a device or software that has the function of transmitting captured audio data to a server using an appropriate communication protocol.

[0979] The "means for analyzing voice data and converting it into text data" refers to a software algorithm or device for analyzing received voice data and generating the content of the voice data as text data.

[0980] "Means for analyzing a situation based on text data and generating first aid instructions" refers to software or a device that uses an AI model or rule-based system to analyze a situation based on text data and issue appropriate first aid instructions.

[0981] The "means for encoding the generated first aid instructions as audio data" is software or a device for converting the text-based first aid instructions back into audio data.

[0982] The "means for transmitting encoded voice data to a user terminal" is a device or software that has the function of transmitting encoded voice data to a user terminal using an appropriate communication protocol.

[0983] The "means for reproducing audio data received on a user terminal" refers to a device or software having the function of reproducing audio data received by a user terminal and providing audio instructions to the user.

[0984] This invention is a system that supports first aid in emergencies using voice recognition and an AI model. The system includes a voice input unit, a voice analysis unit, a situation analysis unit, a first aid instruction generation unit, and a voice output unit.

[0985] System configuration and operation

[0986] Hardware and Software Configuration

[0987] User terminal: A mobile information terminal such as a smartphone. It uses a built-in microphone to capture voice input and generate voice data. It also has a speaker to play back voice instruction data from the server.

[0988] Server: A powerful computer system that includes a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze and convert audio data into text, and a generative AI model (e.g., OpenAI's GPT-4) to analyze the situation and generate first aid instructions.

[0989] Program processing explanation

[0990] Voice Recognition

[0991] In an emergency, a user launches the "RescueVoice" application on their smartphone and utters the command "Help!" The user's device's built-in microphone captures this voice and converts it into digital audio data, which is then sent to a server via the Internet.

[0992] Analysis of audio data

[0993] The server uses a speech recognition engine to analyze the received voice data and convert it into text data. For example, if a user says, "Help me, my friend has fallen down," the voice data is converted into text data.

[0994] Analyzing the situation and generating first aid instructions

[0995] The converted text data is then analyzed by a generative AI model to understand the situation. The AI ​​model then references past data and medical knowledge to generate the most appropriate first aid instructions. For example, it might generate specific instructions such as, "If cardiac arrest is suspected, begin chest compressions."

[0996] Encoding and transmitting first aid instructions as audio data

[0997] The server encodes the generated emergency treatment instructions as voice data, which is then transmitted to the user terminal via the Internet.

[0998] Playing voice instructions

[0999] The user device plays the received voice data and conveys instructions to the user, such as "Please begin chest compressions. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[1000] Monitor progress and generate further instructions

[1001] The system continuously monitors the user's progress in administering first aid and generates additional instructions as needed. For example, if the user says, "I went to get the AED," new voice data is sent to the server for analysis. As a result, additional instructions are generated and encoded as voice data, such as, "Prepare the AED. Attach the pads to the patient's chest."

[1002] Examples of concrete examples and prompts

[1003] Specific examples

[1004] If a friend suddenly collapses:

[1005] 1. User: Launch the "RescueVoice" application on their smartphone.

[1006] 2. User: Says "Help!"

[1007] 3. Device: Captures audio with the built-in microphone and sends the audio data to the server.

[1008] 4. Server: The speech recognition engine converts the speech into text, and the AI ​​model determines the situation: "My friend has collapsed."

[1009] 5. Server: Generates the instruction "Please begin chest compressions" and encodes it as audio data.

[1010] 6. Terminal: Plays the received audio data and delivers it to the user.

[1011] 7. User: Begin chest compressions as instructed.

[1012] Prompt Sentence Examples

[1013] "If a user is in an emergency and needs first aid, design a system that provides appropriate instructions in real time based on voice input."

[1014] "Imagine a scenario in which a user uses a voice recognition system to provide first aid to a friend who has suddenly collapsed, and explain the specific instructions that would be given."

[1015] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1016] Step 1: The user launches the "RescueVoice" application on their smartphone. This launches the app and makes it ready to accept voice input. When the user says "Help!", the built-in microphone captures the voice. The input is the user's voice, and the output is voice data.

[1017] Step 2: The device generates the captured voice data and sends it to the server via the Internet. The specific operations at this stage are the process from voice capture to data transmission. The input is the voice data, and the output is the voice data sent to the server.

[1018] Step 3: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze the received voice data. This engine converts the voice data into text data. The specific data processing involves analyzing the voice data and converting it into text. The input is voice data, and the output is text data.

[1019] Step 4: The server performs situation analysis based on the generated text data. Here, a generative AI model (e.g., OpenAI's GPT-4) is used to determine the situation from the text. This analysis identifies an emergency situation and generates appropriate first aid instructions. The input is text data, and the output is first aid instructions.

[1020] Step 5: The server uses a text-to-speech engine to encode the generated first aid instructions as audio data. The specific operation is to convert the text data into an audio file. The input is the first aid text instructions, and the output is the audio data.

[1021] Step 6: The server sends the encoded voice data to the user terminal via the Internet. The specific operation at this stage is to send data from the server and receive it at the user terminal. The input is the voice data, and the output is the voice data received at the terminal.

[1022] Step 7: The device plays the received voice data and delivers instructions to the user. The user then performs the instructed first aid. Specifically, the instruction to the user is "Please begin chest compressions." The input is the voice data, and the output is the played voice instructions.

[1023] Step 8: The device monitors the user's progress in providing first aid and continuously captures new voice input. For example, when the user says, "I went to get the AED," the voice data is sent to the server again. The input is the user's new voice, and the output is the regenerated voice data.

[1024] Step 9: The server analyzes the newly received voice data and generates additional first aid instructions as needed. It then encodes these instructions as voice data and sends them to the user terminal. For example, the generated instruction might be, "Prepare to use the AED. Attach the pads to the patient's chest." The input is the new voice data, and the output is the voice data of the additional first aid instructions.

[1025] (Application example 1)

[1026] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1027] In the event of an emergency at a factory, it is difficult for employees to provide first aid quickly and accurately. For this reason, there is a need for a system that can immediately provide appropriate first aid instructions in an emergency and improve safety. In particular, a rapid response is required when a worker collapses, and there is a need for a method to solve this problem using robots.

[1028] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1029] In this invention, the server includes a voice input means, a means for analyzing voice data and converting it into text data, a means for analyzing the situation and generating first aid instructions, a means for encoding the generated instructions as voice data, a means for transmitting the voice instruction data to a user terminal, a means for playing the encoded voice instruction data on the user terminal, a means for the robot to capture voice using a built-in microphone when an emergency occurs and transmit the voice data to the server, and a means for the robot to receive and play the first aid instruction voice data transmitted from the server. This enables the robot to instantly provide appropriate first aid instructions in the event of an emergency at a factory site, enabling a fast and effective response.

[1030] "Audio input means" refers to hardware and software for capturing audio data.

[1031] "Means for analyzing voice data and converting it into text data" refers to the function of converting voice data into text using voice recognition technology.

[1032] The "means for analyzing the situation and generating first aid instructions" refers to a function for analyzing the emergency situation based on input voice data and generating appropriate first aid instructions.

[1033] "Means for encoding the generated instructions as voice data" refers to speech synthesis technology for converting text data into voice data.

[1034] The "means for transmitting voice instruction data to the user terminal" refers to a communication means for transmitting the generated voice data to the user terminal.

[1035] The "means for reproducing the encoded voice instruction data on the user terminal" refers to a function for reproducing the received voice instruction data on the user terminal.

[1036] "Means for the robot to capture audio using its built-in microphone in the event of an emergency and transmit that audio data to a server" refers to the function of transmitting audio data in an emergency to a server using the robot's microphone and communication function.

[1037] "Means for the robot to receive and play back the voice data of first aid instructions sent from the server" refers to the function of the robot to play back the voice data of first aid instructions received from the server.

[1038] The present invention relates to a system that provides effective first aid instructions for emergencies that occur at factory sites. This system aims to improve worker safety by using a robot to input voice and provide appropriate first aid in emergencies.

[1039] The robot uses a built-in microphone as a means of voice input. This microphone captures surrounding sounds and sends the data to a server. If the robot detects an emergency, for example, if a worker yells "Help!", the captured voice data is immediately sent to the server.

[1040] On the server side, speech recognition software (e.g., Google's speech recognition API) is used to convert the speech data into text data, which is then analyzed by an AI model (e.g., a generative AI model) to understand the emergency situation and generate appropriate first aid instructions.

[1041] The generated first aid instructions are encoded as voice data using speech synthesis technology, using speech synthesis software (e.g., the pydub library), and the encoded voice data is sent to the robot, which plays it back, providing the appropriate first aid instructions to the worker in real time.

[1042] As a concrete example, consider the case where a worker collapses in a factory. When the worker says "Help!", the robot's built-in microphone captures the voice and sends the voice data to the server. The server analyzes the voice and, if it determines that cardiopulmonary resuscitation is necessary, generates the instruction "Please begin chest compressions." This instruction is encoded as voice data and sent back to the robot. The robot then plays back this instruction and provides the worker with specific first aid instructions, enabling a rapid response.

[1043] As an example, the following prompt is presented:

[1044] "A worker falls and cries out, 'Help!' This voice is captured by the robot's microphone, and appropriate first aid instructions are provided in real time."

[1045] In this way, the present invention allows for a fast and effective response to emergencies on the factory floor, greatly improving worker safety.

[1046] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1047] Step 1:

[1048] The robot uses a built-in microphone to capture surrounding sounds. The input is the worker's speech, and the output is audio data. Specifically, the microphone captures the voice saying "Help!" and stores the audio data in the robot's memory.

[1049] Step 2:

[1050] The robot sends the captured audio data to the server. The input is the audio data obtained in step 1, and the output is the audio data sent to the server. Specifically, the robot uploads the audio data to the server via an internet connection.

[1051] Step 3:

[1052] The server analyzes the received voice data and converts it into text data. The input is voice data and the output is text data. Specifically, the server uses voice recognition software (for example, Google's voice recognition API) to analyze the voice data and convert it into text data such as "Help!"

[1053] Step 4:

[1054] The server analyzes the situation based on the text data and generates appropriate first aid instructions. The input is text data, and the output is first aid instructions. Specifically, the server analyzes the text data using a generative AI model and generates the instruction "Please begin chest compressions."

[1055] Step 5:

[1056] The server encodes the generated first aid instructions as audio data. The input is the first aid instructions, and the output is the encoded audio data. Specifically, the server converts the instructions into audio data using speech synthesis software (e.g., the pydub library).

[1057] Step 6:

[1058] The server sends the encoded voice data to the robot. The input is the encoded voice data, and the output is the voice data sent to the robot. Specifically, the server uploads the encoded voice data to the robot via an internet connection.

[1059] Step 7:

[1060] The robot receives the transmitted voice data and plays it back to the worker. The input is encoded voice data, and the output is the played-back voice instruction. Specifically, the robot plays the received voice data through its built-in speaker and tells the worker to "start chest compressions."

[1061] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1062] This invention relates to a system that uses voice recognition and emotion recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input and emotion recognition.

[1063] First, the user launches the "RescueVoice" application on their smartphone. The system starts up when the user says "Help!" The user's device captures the voice using a built-in microphone and generates digital voice data, which is then immediately sent to the server.

[1064] On the server side, a voice recognition engine analyzes the received voice data and converts it into text data. Based on this text data, an AI model analyzes the situation. For example, if someone says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[1065] Next, the first aid instructions generated by the server are encoded as voice data. The encoded voice data is immediately sent to the user's device. The user's device plays back the received voice data and delivers it to the user. The user begins first aid according to the instructions. For example, the instructions might be, "Start chest compressions. Place your hands together in the center of the chest and compress at a rate of 100-120 times per minute."

[1066] Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions. The emotion engine analyzes voice data and user behavior data to identify the user's emotional state. For example, if the user is very anxious, the emotion engine will determine that the user is in a stressed state.

[1067] Based on the user's emotional state recognized by the emotion engine, the server adjusts the instructions. For example, if a user is under stress, the server can provide additional encouragement and advice to encourage them to remain calm and administer first aid. A voice instruction such as "Please stay calm. It's okay. I'll give you instructions, so please follow them," is generated, encoded, and sent to the user's device.

[1068] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice. If the emotion engine determines that the user is under stress, it provides additional encouragement such as "Please stay calm. Your actions will help."

[1069] Thus, by combining voice input and emotion recognition, the system of the present invention helps users administer first aid more calmly and effectively, providing vital life-saving assistance until an ambulance arrives.

[1070] The processing flow will be explained below.

[1071] Step 1:

[1072] The user launches the "RescueVoice" app on their smartphone, which puts the app into a voice input standby state.

[1073] Step 2:

[1074] The user says "Help!" The device uses the built-in microphone to capture the user's voice and generate digital audio data.

[1075] Step 3:

[1076] The audio data generated by the device is sent to the server using the secure HTTP(S) protocol.

[1077] Step 4:

[1078] The server passes the received voice data to a speech recognition engine, which converts the voice data into text data. This speech recognition engine uses a highly accurate generative AI model.

[1079] Step 5:

[1080] The server passes the text data to an AI model, which analyzes the situation. For example, it analyzes text such as "My friend has collapsed" and determines whether the situation is urgent.

[1081] Step 6:

[1082] The server generates appropriate first aid instructions based on the analysis results. For example, if cardiac arrest is suspected, it generates instructions such as, "Start chest compressions immediately. Place your hands together in the center of the chest and press at a rate of 100-120 times per minute."

[1083] Step 7:

[1084] The server encodes the generated instructions as voice data and transmits the voice data to the user terminal.

[1085] Step 8:

[1086] The device plays the received voice data and delivers it to the user, who then follows the instructions to begin providing first aid.

[1087] Step 9:

[1088] The device continuously monitors the user's voice input. For example, if the user says, "Help isn't here yet," the device retransmits the voice to the server.

[1089] Step 10:

[1090] The server receives the new voice data, analyzes it again, and generates additional first aid instructions based on the new situation, such as "Continue chest compressions and use an AED if one is available."

[1091] Step 11:

[1092] The server then transmits the encoded audio data back to the user terminal, which then plays the audio back to the user.

[1093] Step 12:

[1094] The user follows additional instructions to administer any necessary first aid, such as attaching pads to the patient's chest to use an AED.

[1095] Step 13:

[1096] The server sends the voice data and behavioral data to the emotion engine, which analyzes the user's emotional state and determines whether the user is stressed or anxious.

[1097] Step 14:

[1098] The server adjusts the instructions based on the analysis results of the emotion engine. For example, if it determines that the user is under stress, it adds encouragement and advice to stabilize the user, such as "Please stay calm. It's okay. I'll give you instructions, so please act as instructed."

[1099] Step 15:

[1100] The server encodes the adjusted instructions as voice data and transmits it to the user terminal.

[1101] Step 16:

[1102] The device then plays back the received audio data and delivers it to the user, who then follows the instructions to continue providing first aid.

[1103] Through this series of processing steps, the user receives appropriate first aid instructions in real time, as well as encouragement and advice tailored to their emotional state, enabling them to carry out effective life-saving measures until an ambulance arrives.

[1104] Example 2

[1105] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1106] Conventional first aid support systems use voice input and output instructions, but they are unable to consider the user's emotional state, which can lead to users becoming impatient or anxious and being unable to act appropriately in an emergency. In particular, when a user is under stress, there is a high risk that they will lose their composure and make a serious, life-threatening mistake. In addition, when multiple situations overlap, it is sometimes difficult to properly determine priorities.

[1107] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1108] In this invention, the server includes a voice input unit, a unit for analyzing voice data and converting it into text data, a unit for analyzing a situation based on the text data and generating appropriate first aid instructions, a unit for encoding the generated first aid instructions as voice data, a unit for transmitting the encoded voice instruction data to a user terminal, a unit for playing the encoded voice instruction data on the user terminal, a unit for analyzing the voice data and behavioral data to recognize the user's emotional state, and a unit for adjusting the first aid instructions based on the user's emotional state and generating additional instructions or encouragement. This allows the user to provide first aid appropriately and calmly in an emergency. Furthermore, even when multiple situations arise, the server can determine priorities and provide optimal instructions.

[1109] "Audio input means" refers to a device or function that captures a user's voice and converts it into digital audio data.

[1110] The "means for analyzing voice data and converting it into text data" refers to software or hardware for analyzing acquired voice data and converting it into character string data.

[1111] The "means for analyzing the situation and generating appropriate first aid instructions" is a device or function that judges the situation based on text data and automatically generates optimal first aid instructions.

[1112] The "means for encoding the generated first aid instructions as audio data" refers to software or hardware for converting the generated text format first aid instructions into audio data.

[1113] The "means for transmitting encoded voice instruction data to a user terminal" is a function or system that transmits encoded voice data to a device held by a user via a network.

[1114] The "means for reproducing encoded voice instruction data on the user terminal" refers to a system or function for reproducing voice data sent to the user terminal and conveying instructions to the user.

[1115] The "means for recognizing the emotional state of a user by analyzing voice data and behavioral data" refers to software or a device for analyzing the voice data and behavioral data of a user and automatically recognizing the emotional state.

[1116] "Means for adjusting first aid instructions and generating additional instructions or encouragement based on emotional state" refers to a system or function for adjusting the content of first aid instructions based on the user's emotional state and generating further instructions or encouragement as needed.

[1117] This invention relates to a system that uses voice recognition and emotion recognition to support first aid in emergencies. This system uses a user terminal such as a smartphone to provide appropriate first aid instructions in real time based on voice input and emotion recognition.

[1118] First, the user launches a dedicated application on their smartphone. The system starts up when the user says "Help!" The user's device captures the voice using a built-in microphone and generates digital voice data. This generated voice data is immediately sent to the server.

[1119] On the server side, a speech recognition engine (e.g., Google Speech-to-Text API) analyzes the received voice data and converts it into text data. Based on this text data, an AI model (e.g., OpenAI GPT-3) analyzes the situation. For example, if a voice input says, "My friend has collapsed," it could be an emergency situation such as cardiac arrest or fainting. The AI ​​model refers to past data and medical knowledge to determine the most appropriate first aid.

[1120] Next, a speech synthesis engine (e.g., Amazon Polly) is used to encode the first aid instructions generated by the server as voice data. This encoded voice data is immediately sent to the user's device. The user's device plays the received voice data and delivers it to the user. The user can start first aid by following the instructions. For example, the voice instruction is "Start chest compressions. Place your hands together at the center of the chest and compress at a rate of 100-120 times per minute."

[1121] Furthermore, this system incorporates an emotion engine (e.g., Affectiva's emotion recognition API) that recognizes the user's emotions. The emotion engine analyzes voice data and user behavior data to identify the user's emotional state. For example, if the user is very anxious, the emotion engine will determine that the user is in a state of stress.

[1122] Based on the user's emotional state recognized by the emotion engine, the server adjusts the instructions. For example, if a user is under stress, the server can provide additional encouragement and advice to encourage them to remain calm and administer first aid. A voice instruction such as "Please stay calm. It's okay. I'll give you instructions, so please follow them," is generated, encoded, and sent to the user's device.

[1123] As a concrete example, consider the case where a friend suddenly collapses. When the user utters "Help!", the user device captures the voice and sends it to the server. The server analyzes the voice, recognizes the emergency situation, and generates the instruction "Please begin chest compressions." As the user performs chest compressions as instructed, the system continuously monitors the user's voice. If the emotion engine determines that the user is under stress, it provides additional encouragement such as "Please stay calm. Your actions will help."

[1124] Examples of prompts for the generative AI model include, "The user says that their friend has collapsed. Please suggest the best first aid for this situation." or "The user seems very anxious. Please think of some encouraging words to calm the user in this situation."

[1125] In this way, the combination of voice input and real-time emotion recognition technology provides a system that allows users to respond calmly and effectively in an emergency, providing vital life-saving assistance until an ambulance arrives.

[1126] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1127] Step 1:

[1128] The user launches a dedicated application on their smartphone and says, "Help!"

[1129] Input: User's voice

[1130] Output: Captured digital audio data

[1131] Specific actions: The user taps the icon on their smartphone to launch the app and utters "Help!" in an emergency.

[1132] Step 2:

[1133] The device uses a built-in microphone to capture the user's voice and convert it into digital audio data.

[1134] Input: User's voice

[1135] Output: Digital audio data

[1136] Specific operation: The smartphone's microphone picks up the user's speech and digitizes it as voice data.

[1137] Step 3:

[1138] The terminal transmits the generated digital voice data to the server via the network.

[1139] Input: Digital audio data

[1140] Output: Audio data sent to the server

[1141] What happens: The device sends audio data to the server via an internet connection.

[1142] Step 4:

[1143] The server uses a speech recognition engine (e.g., Google Speech-to-Text API) to analyze the received voice data and convert it into text data.

[1144] Input: Audio data sent to the server

[1145] Output: Text data

[1146] Specific operation: The server calls the Google Speech-to-Text API and converts the audio data into text.

[1147] Step 5:

[1148] The server analyzes the situation using an AI model (e.g., OpenAI GPT-3) based on the text data and generates first aid instructions.

[1149] Input: Text data

[1150] Output: First aid instruction text

[1151] How it works: The server inputs text data into the GPT-3 model and generates first aid instructions such as "Begin chest compressions."

[1152] Step 6:

[1153] Use a speech synthesis engine (e.g., Amazon Polly) to encode the server-generated first aid instructions as audio data.

[1154] Input: First Aid Instructions Text

[1155] Output: Audio data

[1156] What happens: The server calls Amazon Polly to convert the text data into speech.

[1157] Step 7:

[1158] The server transmits the encoded voice instruction data to the user terminal via the network.

[1159] Input: Audio data

[1160] Output: Audio data sent to the user's device

[1161] Specific operation: The server sends encoded voice data to the user's terminal via the Internet.

[1162] Step 8:

[1163] The terminal plays back the received audio data and conveys emergency instructions to the user.

[1164] Input: Audio data sent to the user's device

[1165] Output: The audio instructions played

[1166] Specific operation: The device speaker plays audio data and gives the user the voice instruction, "Please begin chest compressions."

[1167] Step 9:

[1168] The server uses an emotion engine (e.g., Affectiva's emotion recognition API) to analyze the user's voice data and behavioral data to identify the user's emotional state.

[1169] Input: User voice data, behavioral data

[1170] Output: User's emotional state data

[1171] How it works: The server calls Affectiva's API, analyzes the user's voice and behavioral data, and identifies their stress level.

[1172] Step 10:

[1173] The server adjusts first aid instructions and generates additional instructions and encouragement based on the perceived emotional state.

[1174] Input: User emotional state data

[1175] Output: Adjusted first aid instruction text, additional encouragement text

[1176] Specific actions: The AI ​​model generates additional instructions such as, "Please stay calm. It's okay. I'll give you instructions, so please follow them," and encodes them as voice data and sends them to the user's device.

[1177] Step 11:

[1178] The terminal plays back the received additional audio data to provide encouragement or advice to the user.

[1179] Input: Adjusted first aid instruction audio data, additional encouragement audio data

[1180] Output: Played back audio instructions of encouragement and advice

[1181] Specific action: The device speaker plays audio data and conveys an encouraging message to the user to stay calm.

[1182] By using the specific processing steps described above, this system supports the user in providing appropriate and calm first aid in an emergency.

[1183] (Application example 2)

[1184] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1185] Conventional first aid support systems can sometimes make it difficult for users to respond appropriately in an emergency using only voice instructions. This is especially true when users are stressed or anxious. Furthermore, systems that rely solely on voice data cannot take into account the user's emotional state and lack the support necessary to help the user remain calm. This can reduce the effectiveness of first aid and potentially waste valuable time waiting for an ambulance to arrive.

[1186] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means, a means for analyzing voice data and converting it into text data, a means for analyzing the situation and generating first aid instructions, a means for encoding the generated instructions as voice data, a means for transmitting the voice instruction data to the user terminal, a means for playing the encoded voice instruction data on the user terminal, an emotion recognition means for recognizing the emotional state, and a means for adjusting the instruction content based on the emotion recognition result. As a result, appropriate first aid instructions according to the user's voice and emotional state are provided in real time, allowing the user to respond to emergencies more calmly and effectively.

[1187] The "voice input means" is a device for capturing the user's voice and recording it as digital voice data.

[1188] The "means for analyzing voice data and converting it into text data" refers to a system or software for analyzing captured voice data and converting it into character string data.

[1189] The "means for analyzing the situation and generating first aid instructions" refers to a system or software for determining appropriate first aid based on speech recognition and the situation, and generating the instructions.

[1190] The "means for encoding the generated instructions as audio data" is a system or software for converting the generated first aid instructions into audio data.

[1191] The "means for transmitting voice instruction data to a user terminal" is a communication means for transferring the generated voice data to the user's device.

[1192] The "means for playing back encoded voice instruction data on a user terminal" refers to hardware or software for playing back encoded voice data on a device held by the user.

[1193] The "emotion recognition means for recognizing an emotional state" is an algorithm or software for analyzing and identifying the emotional state of a user from voice data.

[1194] The "means for adjusting instructions based on emotion recognition results" refers to a system or software for modifying and optimizing first aid instructions according to the user's emotional state.

[1195] This invention is a system for responding to emergencies, which supports first aid by utilizing voice input and emotion recognition technology. The specific configuration and operation procedure are shown below.

[1196] The server captures the user's voice through a voice input means. This voice input is performed using the built-in microphone of the user's device, such as a smartphone. The captured voice data is recorded in digital format and sent to the server. The server analyzes the received voice data using voice recognition software (for example, the speech_recognition library) and converts it into text data.

[1197] Based on the text data, the server analyzes the situation and generates appropriate first aid instructions. The AI ​​model used here references historical data and medical knowledge to determine the most appropriate first aid. Based on this determination, the server encodes the instructions into audio data (using a text-to-speech system, for example, the TextToSpeech class).

[1198] The generated voice instruction data is immediately transmitted to the user terminal, which plays back the voice data and provides the user with appropriate first aid instructions.

[1199] Furthermore, an emotion recognition engine (e.g., using the EmotionRecognition module) that recognizes the user's emotional state analyzes the voice data and determines whether the user is feeling stressed. Based on the emotion recognition result, the server adjusts the instructions. For example, if the user is determined to be in a stressful state, the server will include additional encouragement or advice in the voice instructions.

[1200] As a concrete example, consider a scenario in which a friend suddenly collapses. When the user utters "Help!", the device captures the voice and sends the data to the server. The server analyzes the voice and generates an instruction to "Start chest compressions" as an emergency. The instruction is encoded as voice data and sent to the user device. When the user starts chest compressions as instructed, the emotion engine continuously monitors the user's voice and, if it determines that the user is in a stressed state, provides additional voice instructions such as "Please stay calm. It's okay. I'm giving you instructions, so please follow them."

[1201] An example of a prompt is "When a user is in an extremely urgent situation and needs help, please generate voice instructions that utilize emotion recognition." This will help users provide first aid as calmly and effectively as possible.

[1202] The present invention reduces stress when a user performs first aid in an emergency, and enables the user to perform correct procedures more reliably.

[1203] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1204] Step 1:

[1205] A user speaks "Help!" into a smartphone. The user's input is voice data, which is captured using the smartphone's built-in microphone. The output is digital voice data.

[1206] Step 2:

[1207] The device transmits the captured audio data to the server. The input is digital audio data, and the output is the audio data transmitted to the server. The data transmission is performed using the device's communication module.

[1208] Step 3:

[1209] The server analyzes the received voice data using a voice recognition engine (for example, the speech_recognition library) and converts it into text data. The input is voice data, and the output is text data. Specifically, the waveform of the voice data is analyzed and data processing is performed to express it as a string of characters.

[1210] Step 4:

[1211] The server analyzes the situation based on the converted text data. The AI ​​model used is a generative AI model, which refers to past data and medical knowledge to determine the most appropriate first aid treatment. The input is text data, and the output is first aid instruction data. For data calculation, the generative AI model performs an analytical process on the text data and generates first aid instructions.

[1212] Step 5:

[1213] The server encodes the generated first aid instructions as audio data. The input is the first aid instructions and the output is the encoded audio data. The software used is a text-to-speech system (e.g., TextToSpeech class). At this stage, the instructions are prepared to be transmitted to the user as audio.

[1214] Step 6:

[1215] The server transmits the encoded voice instruction data to the user terminal. The input is the encoded voice data, and the output is the voice data transmitted to the user terminal. The data transmission is performed using a communication module.

[1216] Step 7:

[1217] The device plays the received audio data and provides the user with appropriate first aid instructions. The input is the encoded audio data, and the output is audio instructions that the user can hear. The audio instructions are played using the device's audio playback capabilities.

[1218] Step 8:

[1219] The server analyzes the user's emotional state from the voice data using an emotion recognition engine to recognize the user's emotional state. The input is the newly captured voice data and the output is the emotional state data. The voice analysis is performed using an emotion recognition engine (e.g., EmotionRecognition module).

[1220] Step 9:

[1221] The server adjusts the instructions based on the emotion recognition results. The input is emotional state data and first aid instruction data, and the output is the adjusted first aid instruction data. Specifically, if the user is in a stressful state, the server includes encouragement and advice in addition to the first aid instruction data.

[1222] Step 10:

[1223] The server re-encodes the adjusted first aid instruction data as voice data and transmits it to the user terminal. The input is the adjusted instruction data, and the output is the re-encoded voice data. The data is again transmitted using the communication module, and the user terminal plays back the voice data.

[1224] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1225] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1226] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1227] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1228] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1229] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1230] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1231] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1232] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1233] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1234] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1235] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1236] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1237] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1238] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1239] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1240] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1241] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1242] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1243] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1244] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1245] The following is further disclosed regarding the above embodiment.

[1246] (Claim 1)

[1247] A voice input means;

[1248] A means for analyzing the voice data and converting it into text data;

[1249] means for conducting a situation analysis and generating first aid instructions;

[1250] means for encoding the generated instructions as audio data;

[1251] means for transmitting voice instruction data to a user terminal;

[1252] means for playing the encoded audio instruction data on the user terminal;

[1253] A system including:

[1254] (Claim 2)

[1255] 10. The system of claim 1, further comprising means for monitoring user behavior in accordance with the first aid instructions and generating additional instructions based thereon.

[1256] (Claim 3)

[1257] 2. The system of claim 1, further comprising means for analyzing the situations and identifying and instructing first aid with a high priority when the received voice data relates to multiple situations.

[1258] "Example 1"

[1259] (Claim 1)

[1260] A voice input means;

[1261] means for generating and transmitting audio data to a server;

[1262] A means for analyzing the voice data and converting it into text data;

[1263] a means for analyzing the situation based on the text data and generating first aid instructions;

[1264] means for encoding the generated first aid instructions as audio data;

[1265] means for transmitting the encoded voice data to a user terminal;

[1266] means for playing the received audio data on the user terminal;

[1267] A system including:

[1268] (Claim 2)

[1269] 10. The system of claim 1, further comprising means for monitoring user behavior in accordance with the first aid instructions and generating additional instructions based thereon.

[1270] (Claim 3)

[1271] 2. The system of claim 1, further comprising means for analyzing the situations and identifying and instructing first aid with a high priority when the received voice data relates to multiple situations.

[1272] "Application Example 1"

[1273] (Claim 1)

[1274] A voice input means;

[1275] A means for analyzing the voice data and converting it into text data;

[1276] means for conducting a situation analysis and generating first aid instructions;

[1277] means for encoding the generated instructions as audio data;

[1278] means for transmitting voice instruction data to a user terminal;

[1279] means for playing the encoded audio instruction data on the user terminal;

[1280] A means for the robot to capture voice using a built-in microphone when an emergency occurs and transmit the voice data to a server;

[1281] a means for the robot to receive and play back the voice data of instructions for first aid transmitted from the server;

[1282] A system including:

[1283] (Claim 2)

[1284] 10. The system of claim 1, further comprising means for monitoring user behavior in accordance with the first aid instructions and generating additional instructions based thereon.

[1285] (Claim 3)

[1286] 2. The system of claim 1, further comprising means for analyzing the situations and identifying and instructing first aid with a high priority when the received voice data relates to multiple situations.

[1287] "Example 2: Combining Emotion Engines"

[1288] (Claim 1)

[1289] A voice input means;

[1290] A means for analyzing the voice data and converting it into text data;

[1291] A means for analyzing a situation based on the text data and generating appropriate first aid instructions;

[1292] means for encoding the generated first aid instructions as audio data;

[1293] means for transmitting the encoded voice instruction data to a user terminal;

[1294] means for playing the encoded audio instruction data on the user terminal;

[1295] means for analyzing voice data and behavioral data to recognize the emotional state of a user;

[1296] means for adjusting first aid instructions and generating additional instructions or encouragement based on the emotional state;

[1297] A system including:

[1298] (Claim 2)

[1299] 10. The system of claim 1, further comprising means for monitoring user behavior in accordance with the first aid instructions and generating additional instructions based thereon.

[1300] (Claim 3)

[1301] 2. The system according to claim 1, further comprising means for, when the received voice data includes a plurality of situations, analyzing the situations to identify and instruct first aid with a high priority.

[1302] "Application example 2 when combining emotion engines"

[1303] (Claim 1)

[1304] A voice input means;

[1305] A means for analyzing the voice data and converting it into text data;

[1306] means for conducting a situation analysis and generating first aid instructions;

[1307] means for encoding the generated instructions as audio data;

[1308] means for transmitting voice instruction data to a user terminal;

[1309] means for playing the encoded audio instruction data on the user terminal;

[1310] an emotion recognition means for recognizing an emotional state;

[1311] means for adjusting instruction content based on emotion recognition results;

[1312] A system including:

[1313] (Claim 2)

[1314] 10. The system of claim 1, further comprising means for monitoring user behavior in accordance with the first aid instructions and generating additional instructions based thereon.

[1315] (Claim 3)

[1316] 2. The system of claim 1, further comprising means for analyzing the situations and identifying and instructing first aid with a high priority when the received voice data relates to multiple situations.

[1317] (Claim 4)

[1318] 10. The system of claim 1, further comprising means for performing real-time voice recognition and emotion recognition when retransmitting the generated voice instruction data to the user terminal.

[1319] (Claim 5)

[1320] 10. The system of claim 1, further comprising means for providing encouragement or advice to the user based on the emotion recognition result. [Explanation of symbols]

[1321] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A voice input means; A means for analyzing the voice data and converting it into text data; means for conducting a situation analysis and generating first aid instructions; means for encoding the generated instructions as audio data; means for transmitting voice instruction data to a user terminal; means for playing the encoded audio instruction data on the user terminal; A system including:

2. 10. The system of claim 1, further comprising means for monitoring user behavior in accordance with first aid instructions and generating additional instructions based thereon.

3. 2. The system according to claim 1, further comprising means for analyzing the situations when the received voice data relates to a plurality of situations, and for identifying and instructing first aid with a high priority.

Images