system

Abstract

We provide the system. [Solution] A voice acquisition means for acquiring voice information from users, A speech conversion means that converts acquired speech information into text data, A means for analyzing the user's purpose from the aforementioned text data, Information transmission means for sending information to contacts selected based on the analyzed purpose, A speech generation means that converts the generated response into speech and outputs it, The aforementioned objective analysis means includes a command generation means that issues a command to prompt a specific action at a specified time, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When elderly people conduct daily communication due to vision decline and memory problems, there are problems such as being unable to remember phone numbers and the characters in the phone book being too small to see. They also suffer from feelings of loneliness and the inability to respond promptly in case of emergencies. As a result, the quality of life of the elderly has declined, and it has become difficult for them to live safely.

Means for Solving the Problems

[0005] This invention provides a means for receiving voice input from a user and converting the received voice into text using speech recognition. It also provides a means for analyzing the user's intent from the converted text data and communicating with a designated contact based on the analysis results. Furthermore, by outputting the generated response as voice using speech synthesis, it is made easy for elderly people with visual or memory impairments to use. In addition, the intent analysis means is equipped with a means for identifying keywords indicating an emergency and automatically notifying relevant external organizations, thereby ensuring the safety of the elderly. This realizes a communication support system that allows elderly people to live their daily lives with peace of mind.

[0006] "Voice receiving means" refers to a device or function that receives voice input from a user in digital format.

[0007] "Speech recognition means" refers to a technology or process that analyzes received speech and converts it into corresponding text data.

[0008] An "intent analysis system" is a system that understands and analyzes user requests and objectives from text data obtained by a speech recognition system.

[0009] "Communication means" refers to a device or function for sending a phone call or message to a designated contact based on the analyzed intent.

[0010] A "speech synthesis system" is a system that converts generated text responses into speech data and outputs it in a format that can be heard by the user.

[0011] A "notification system" is a system that has the function of automatically sending notifications to relevant external organizations in the event of an emergency.

[0012] A "memory device" is a data storage system that stores user contact information and related data, and allows them to be retrieved as needed. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] In order to implement this invention, it is necessary to construct a system that includes the following components.

[0035] The system begins when a user makes a phone call. First, the terminal receives the user's voice and sends it to the server. The server uses speech recognition to convert the voice data into text data, and this text data is analyzed by intent analysis. Intent analysis identifies the user's intent, and an appropriate action is determined accordingly.

[0036] As an example, consider a case where a user says, "Call my son." The server analyzes the intent of "Call my son" from the text data and uses a memory device to identify the son's contact information, which is registered in advance. Then, the communication device uses that phone number to actually make the call.

[0037] On the other hand, if a user encounters an emergency and says "Help," the server analyzes this as an emergency keyword. The notification system is activated, automatically sending notifications to the user's pre-configured emergency contacts and external emergency services. In this case, the user's current location information and voice message may be included in the notification.

[0038] Furthermore, the results of communications and notifications, or the system's response, are fed back to the user using speech synthesis. The speech synthesis converts the generated response into natural-sounding speech and sends it to the terminal, which then plays it back to the user.

[0039] In this system, each function is designed to be intuitively operable by the user, taking into consideration that elderly people can use it despite limitations in vision and memory. As a result, users can communicate with peace of mind, easily obtain necessary information, and respond quickly in emergencies. This embodiment effectively achieves the objective of the invention.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The user makes a call to a landline phone number. The device receives this incoming call and records the audio as digital audio data.

[0043] Step 2:

[0044] The terminal sends the recorded audio data to the server. The server receives the transmitted audio data and uses a speech recognition engine to convert it into text data.

[0045] Step 3:

[0046] The server uses the converted text data to analyze the user's intent. Intent analysis tools are used to interpret the content of the call, such as "who to call" or "what to do."

[0047] Step 4:

[0048] Once the user's intent is analyzed, the server uses a communication method to make a phone call to the specified contact. For example, if the instruction is "Call my son," the server will search for the son's phone number from the pre-registered contact information and make the call.

[0049] Step 5:

[0050] The server generates processing results and responses, and uses a speech synthesis engine to convert text responses into speech data.

[0051] Step 6:

[0052] The server sends the generated audio data to the terminal. The terminal plays this audio data and provides feedback to the user. For example, it might play a response like, "I'm calling your son."

[0053] Step 7:

[0054] If a user says "emergency" or "help," the server will recognize this as an emergency. The server will then use notification methods to automatically send notifications to registered emergency contacts and emergency services. These notifications may include the user's location information and a voice message.

[0055] (Example 1)

[0056] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0057] There is a need to provide a system that allows elderly and visually impaired users to communicate intuitively without complex operations and to respond quickly in emergencies. Therefore, technology is required that facilitates a smooth process from voice input to decision-making and provides user-friendly feedback.

[0058] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0059] In this invention, the server includes voice receiving means for receiving voice input from a user, voice recognition means for converting the received voice into text data, and intent analysis means for analyzing the user's intent from the text data. This makes it possible for the user to easily communicate their intent to the system by voice and to quickly perform appropriate actions based on that intent.

[0060] "Voice receiving means" refers to a device or technology that has the function of detecting voice input from a user, converting its content into digital data, and transmitting it to subsequent processing.

[0061] "Speech recognition means" refers to a technology that analyzes speech data acquired from a speech receiving means and converts its content into text data.

[0062] "Intention analysis means" refers to a technology that analyzes text data obtained by speech recognition means and identifies the user's intended actions or requests based on that data.

[0063] "Communication means" refers to technology or devices that provide the function of establishing a connection with a specific communication partner based on an analyzed intent and sending and receiving information.

[0064] "Speech synthesis means" refers to a technology that converts responses generated by a system into a speech format and provides them to the user in an easily understandable way.

[0065] A "notification mechanism" is a technology that has the function of automatically notifying relevant external organizations or pre-configured recipients under specific conditions, particularly in emergency situations.

[0066] "Storage means" refers to a device or technology that has the function of storing a user's contact information and other necessary data, and making it quickly accessible and available as needed.

[0067] To implement this invention, it is necessary to construct an information processing system having a voice input interface and multiple technical components for processing voice. The user initiates an operation by voice, the terminal receives the voice, and the server processes it.

[0068] The server primarily functions as a means of receiving audio. When a user gives voice commands via a phone or smart device, the device captures this audio. This audio is converted into digital audio data and sent to the server. In this process, a typical mobile device or headset equipped with a microphone is often used as the hardware.

[0069] Next, the server operates as a "speech recognition tool," converting digital speech data into text data. This process can utilize, for example, a "speech recognition API." This conversion allows the user's speech to be obtained in text format, enabling subsequent intent analysis.

[0070] Next, the server analyzes the text data using "intent analysis tools" to understand the user's intent. This process utilizes "generative AI models" and "language understanding APIs" that perform natural language processing. This allows the server to identify specific action requests and inquiries from the user's utterances.

[0071] As a practical example, if a user says, "Call my son," the server recognizes this instruction from the text data, retrieves the appropriate phone number by referring to pre-registered contacts, and then activates the server's "communication means" to place a call to the identified contact. This process is achieved through interfaces with "communication APIs" and "telephone service providers."

[0072] If a user utters an emergency phrase such as "Help," the server utilizes its "notification system" to automatically send location information and a voice message to designated recipients in an emergency. Notifications are also sent to pre-configured emergency contacts.

[0073] Finally, the server uses a "speech synthesis method" to convert the system response into a natural-sounding voice format. This allows the user to receive feedback from the system in voice through their terminal. It is expected that a "speech synthesis API" or similar will be used for this speech synthesis.

[0074] An example of a prompt might be, "Find my son's number in my phone book and make a call." This system allows users to intuitively operate the device with their voice, enabling smooth information retrieval and emergency response.

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] The user initiates the operation using voice commands.

[0078] Input: User voice command (e.g., "Call my son")

[0079] Output: Audio data

[0080] Specific operation: The user speaks a specified instruction into the microphone of their smartphone or other voice-enabled device. This voice is captured by the device's microphone and recorded as digital audio data.

[0081] Step 2:

[0082] The device sends audio to the server.

[0083] Input: Digital audio data

[0084] Output: Transfer of audio data

[0085] Specific operation: The terminal sends the captured digital audio data to the server via the network. This enables data processing on the server side.

[0086] Step 3:

[0087] The server converts the audio to text.

[0088] Input: Digital audio data

[0089] Output: Character data

[0090] Specific operation: The server uses speech recognition technology (e.g., "Speech Recognition API") to convert the received audio data into text data. This conversion process involves analyzing sound waves to convert spoken words into text format.

[0091] Step 4:

[0092] The server analyzes the intent.

[0093] Input: Text data

[0094] Output: User intent (e.g., "Make a phone call")

[0095] Specific operation: The server uses generative AI models and language understanding APIs to analyze text data and identify the actions and information the user is seeking. This allows for a clear understanding of what the user wants.

[0096] Step 5:

[0097] The server initiates communication.

[0098] Input: User intent and contact information

[0099] Output: Communication execution

[0100] Specific operation: Based on the user's intent, the server refers to pre-stored contact information and performs the necessary communication. This process involves using communication methods to perform actions such as making phone calls or sending messages.

[0101] Step 6:

[0102] The server will send notifications (if necessary).

[0103] Input: Emergency phrases or conditions

[0104] Output: Notification to external organizations and contacts

[0105] Specific operation: When a user utters an emergency phrase such as "Help," the server activates an emergency notification system and sends a notification containing location information and the situation to pre-configured recipients. This enables a rapid response.

[0106] Step 7:

[0107] The server generates an audio response.

[0108] Input: System response information

[0109] Output: Audio data

[0110] Specific operation: The server uses speech synthesis technology to convert the generated response into natural-sounding speech. This speech data is then used as feedback to the user.

[0111] Step 8:

[0112] The terminal plays the response.

[0113] Input: Audio data

[0114] Output: Voice feedback to the user

[0115] Specific operation: The terminal receives audio data sent from the server and plays it back to the user through the speaker. This allows the user to confirm the system's response.

[0116] (Application Example 1)

[0117] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0118] There is a problem in that users who are unfamiliar with technology, such as the elderly, have difficulty using voice control to quickly and easily perform important communication and respond to emergencies. For example, they may forget when to take their medication or be unable to immediately access the appropriate contacts in an emergency. Therefore, there is a need to provide technology that supports users with tasks necessary for daily life intuitively and without burden, and that allows them to respond with confidence in emergencies.

[0119] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0120] In this invention, the server includes voice acquisition means for acquiring voice information from the user, voice conversion means for converting the acquired voice information into text data, and purpose analysis means for analyzing the user's purpose from the text data. This allows the user to operate intuitively through voice, prompt specific actions at pre-set times, and provide rapid notifications in emergencies.

[0121] A "voice acquisition means" is a device that has the function of receiving voice information from a user and transferring it to a server for analysis.

[0122] A "speech conversion means" is a device that converts acquired speech information into text data.

[0123] A "purpose analysis tool" is a device that analyzes the purpose of a user's utterance from text data and identifies their intent.

[0124] An "information transmission means" is a device that transmits appropriate information to selected contacts based on the results of an analysis conducted according to the user's objectives.

[0125] A "speech generation means" is a device that has the function of converting the generated response into speech and presenting it to the user audibly.

[0126] A "command generation means" is a device that has the function of generating commands to prompt the user to take a specific action based on the results of objective analysis.

[0127] An "information storage device" is a device that stores user contact information and related data and has the function of referencing it as needed.

[0128] To implement this invention, it is necessary to construct a series of systems including voice acquisition means, voice conversion means, purpose analysis means, information transmission means, voice generation means, command generation means, and information storage means. The server first receives voice data sent from the terminal. Then, it converts this voice data into text data using the voice conversion means. The Google® Cloud Speech-to-Text API can be used for voice conversion. Next, the purpose analysis means analyzes the text data using a BERT model to identify the user's intent. Once the intent is identified, the information transmission means sends the information to the user's designated contact via the Twilio API. In emergencies, the command generation means responds to the user with voice generated using Amazon Polly and instructs them on the necessary actions.

[0129] This system is designed to be intuitive and easy to use, especially for users unfamiliar with technology, such as the elderly. It provides not only daily support but also rapid response in emergencies. For example, if a user says, "Tell me when to take my medication," the system will refer to a pre-set schedule and notify them via voice to take their medication at the appropriate time. Similarly, if a user says, "Help me," an emergency contact will be automatically notified.

[0130] A concrete example of a prompt might be, "How to respond when the user says, 'Tell me when to take my medication'." Using this prompt, it becomes possible to generate more natural responses using a generative AI model.

[0131] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0132] Step 1:

[0133] The terminal receives voice input from the user. This voice input includes spoken information from the user. The terminal then prepares to send this voice data to the server.

[0134] Input: User's voice information

[0135] Output: Audio data ready to be sent to the server.

[0136] Step 2:

[0137] The server converts the audio data sent from the terminal into text data using a speech-to-text conversion method. Specifically, it uses the Google Cloud Speech-to-Text API to convert speech to text.

[0138] Input: Audio data

[0139] Output: Converted character data

[0140] Step 3:

[0141] The server analyzes the generated text data using a target analysis tool. This analysis utilizes the BERT model to perform data processing that identifies the user's intent from their utterance.

[0142] Input: Text data

[0143] Output: Analyzed user intent

[0144] Step 4:

[0145] The server decides its actions based on the results of the purpose analysis. For example, if the identified intention is "tell me when to take my medicine," it will access the information storage system to obtain the necessary schedule information.

[0146] Input: Analyzed user intent

[0147] Output: Decision made or information

[0148] Step 5:

[0149] Based on the decided action, the server provides necessary information to the user via an information transmission means. This information is output as voice using a voice generation means and Amazon Polly.

[0150] Input: Decision made or information

[0151] Output: Voice feedback to the user

[0152] Step 6:

[0153] The user takes necessary actions based on the audio feedback received from the server. For example, they might take medication according to the time notified.

[0154] Input: Voice feedback

[0155] Output: User behavior

[0156] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0157] This invention provides a dialogue system that incorporates an emotion engine in addition to the conventional process of receiving user voice input and converting voice data into text. The system is implemented according to the following procedure.

[0158] First, the user makes a call to a fixed phone number to input their voice. The terminal records the user's voice in digital format and sends it to the server. The server uses speech recognition to convert the received voice data into text data.

[0159] Next, the server analyzes the text data using intent analysis tools to understand the user's requests and objectives. This includes things like "who to call" and "how to handle emergencies." Furthermore, it analyzes the user's emotions using newly incorporated emotion recognition tools. This analysis allows the server to determine whether the emotion is positive, negative, or neutral.

[0160] For example, if a user says, "I feel very anxious today," the server will determine through intent analysis that a conversation is necessary and detect anxiety through sentiment analysis. Based on these results, the server will generate words of comfort and encouragement.

[0161] The generated response is converted into audio data using a speech synthesis engine and played back to the user through the device. This provides the user with a sense of security.

[0162] Furthermore, if the emotion recognition system identifies the user's emotions and detects excessively strong anxiety or stress, the server uses a notification system to send an alert to pre-registered family members or caregivers. This feature enables a quick response and support.

[0163] The overall processing of this system aims to provide more personalized care to elderly users and those who require emotional support. This can improve users' quality of life and reduce feelings of loneliness.

[0164] The following describes the processing flow.

[0165] Step 1:

[0166] The user makes a call to a landline phone number. The device receives this incoming call and records the user's voice as digital audio data.

[0167] Step 2:

[0168] The terminal sends the recorded audio data to the server. The server converts the received audio data into text data using speech recognition technology.

[0169] Step 3:

[0170] The server analyzes text data using intent analysis tools. It understands the user's requests and objectives and determines what the appropriate action is. In this process, it identifies voice content such as "who to call" or "to talk about anxieties."

[0171] Step 4:

[0172] The server uses emotion recognition technology to analyze the user's voice to determine their emotions. This analysis determines whether the user is anxious, happy, or in any other emotional state.

[0173] Step 5:

[0174] Based on the analyzed emotional state, the server determines the response. For example, if anxiety is detected, it generates words of comfort or encouragement and takes corresponding action.

[0175] Step 6:

[0176] The server converts the generated response into audio data using a speech synthesis system. The terminal receives this audio data and plays it back to the user to provide feedback. The user receives an appropriate response based on the situation.

[0177] Step 7:

[0178] If the emotion recognition system detects strong anxiety or stress, the server sends an emergency notification to family members or caregivers via a notification system. The notification will include information about the user's emotional state and the support needed.

[0179] (Example 2)

[0180] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0181] Conventional interactive information processing systems have struggled to accurately understand user intent and generate appropriate responses, particularly failing to provide responses that take user emotions into consideration. Furthermore, the lack of mechanisms for quickly notifying external organizations when users experience anxiety or stress makes immediate response difficult, especially in urgent situations.

[0182] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0183] In this invention, the server includes a device means for receiving voice input, a conversion means for converting voice to text, an analysis means for analyzing intent from text, an emotion recognition means for determining the user's emotions, a response generation means for generating a response using a generative AI model, and a voice output means for outputting the generated response as voice. This enables accurate understanding of the user's intent, the provision of natural responses that are in line with their emotions, and the ability to quickly notify external organizations if anxiety or stress is detected, thereby realizing appropriate support for the user.

[0184] A "user" refers to an individual or group that uses the system, and is the entity that provides or obtains information through voice input.

[0185] "Voice input" refers to the format in which the words spoken by the user are incorporated into an information processing system.

[0186] "Device means" refers to equipment or devices for inputting and outputting sound and information, and primarily functions as an interface with the user.

[0187] "Conversion means" refers to technologies and devices that convert audio data into text data, and involves the process of changing speech into text through speech recognition.

[0188] "Analysis methods" refer to technologies and algorithms used to understand user intent and content from text data, and play a role in extracting user requests and objectives.

[0189] "Emotion recognition means" refers to technology that determines the user's emotional state from input voice or text and uses that determination to provide an appropriate response.

[0190] A "generative AI model" refers to a generative model that utilizes artificial intelligence technology to automatically generate natural-sounding text based on user input data.

[0191] "Response generation means" refers to the process or technology of generating a message to respond to the user based on analysis results and emotion recognition data.

[0192] "Audio output means" refers to the technology or device that converts text generated by the system into speech and outputs it in a format that can be heard by the user.

[0193] An "information processing system" refers to a computer system that integrates the above means to enable interaction with the user, and is responsible for processing and outputting input data.

[0194] This invention is an interactive information processing system that utilizes user voice input. The user dials a specific telephone number using a communication terminal and inputs voice. The terminal records the user's voice in digital voice format and sends the data to a server. The server converts the received voice data into text data using speech recognition software. A general-purpose speech recognition engine is suitable for use as the software for this purpose.

[0195] Next, the server uses a generative AI model to analyze the text data, understand the user's intent, and perform sentiment recognition. This involves applying software that utilizes natural language processing (NLP) technology. This analysis allows the system to identify what requests or emotions are underlying the user's statements.

[0196] For example, if a user says, "I feel very anxious today," this text is converted by speech recognition, and the emotion of anxiety is extracted through sentiment analysis. Based on this, the server inputs a prompt into the generation AI model to generate words of comfort. An example of a prompt might be an instruction such as, "Generate an appropriate response for when the user feels anxious."

[0197] The generated response is converted back into voice data using a speech synthesis engine and returned to the user via the terminal. This allows the user to naturally receive feedback from the system. Furthermore, emotion recognition means that if the user shows particularly strong stress or anxiety, the server automatically sends a notification to a pre-registered external organization, enabling a rapid response. This entire process provides the user with personalized support and enhances their sense of security.

[0198] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0199] Step 1:

[0200] The user makes a call to a specific phone number and enters a message by voice. This voice is recorded digitally by the device. The input is an audio signal, and the output is digital audio data. The device performs the necessary encoding to capture the audio signal in high quality.

[0201] Step 2:

[0202] The terminal sends recorded digital audio data to the server. The server receives this data and converts it into text data using speech recognition software. The input is digital audio data, and the output is a text string. The server analyzes the phonemes from the audio data and generates the corresponding text.

[0203] Step 3:

[0204] The server analyzes the generated text data and performs natural language processing to understand the user's intent. In this process, intent analysis is used to determine what the user is requesting in their utterance. The input is text data, and the output is intent information. Intent analysis uses algorithms to extract specific keywords and phrases from the text.

[0205] Step 4:

[0206] The server further detects the user's emotional state by performing sentiment recognition through text analysis. Using sentiment analysis algorithms, it outputs sentiment labels (e.g., positive, anxious, negative) from the input text data. Sentiment recognition utilizes machine learning models to identify language patterns and tones.

[0207] Step 5:

[0208] The server uses a generative AI model to generate an appropriate response based on intent information and emotion labels. A prompt (e.g., "Generate an appropriate response when the user feels anxious.") is input to the AI ​​model, causing it to output a text response. In this process, the AI ​​model performs a complex language generation task.

[0209] Step 6:

[0210] The generated text response is converted into speech data using a speech synthesis engine. The input is the response text, and the output is the synthesized speech data. Speech synthesis involves the process of converting text into sounds with natural pronunciation.

[0211] Step 7:

[0212] The server sends the converted audio data to the terminal, which then plays it back to the user. The user can then hear the system's response. The input is synthesized speech data, and the output is the audible sound. The terminal provides high-quality sound output.

[0213] Step 8:

[0214] The server performs a function that sends a notification to a pre-registered external organization when the emotion recognition system detects an emotion exceeding a certain threshold (e.g., severe anxiety). The input is the emotion recognition result, and the output is the notification message. This function enables security and rapid response.

[0215] (Application Example 2)

[0216] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0217] A challenge exists in that elderly users and those requiring emotional support often struggle to receive immediate and appropriate assistance when they experience feelings of loneliness or anxiety. Furthermore, these emotional states can persist, potentially leading to a decline in their quality of life. While conventional technologies have succeeded in analyzing user intentions, they have been insufficient in rapidly detecting emotional changes and providing the necessary support.

[0218] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0219] In this invention, the server includes an input device for receiving voice input from the user, a recognition device for converting the received voice into text information, an analysis device for analyzing the user's intent, an emotion analysis device for identifying the user's emotions, a communication device for communicating with a designated recipient based on the analyzed intent and emotions, and an output device for outputting the generated response as voice. This makes it possible to quickly detect the emotional state of the user when they feel anxious or lonely and provide appropriate support. Furthermore, if excessive stress or anxiety is detected, an alert can be quickly sent to a supporter using the notification function.

[0220] An "input device" is a device that has the function of receiving voice data from the user and incorporating it into the system.

[0221] A "recognition device" is a device that has the function of analyzing received audio data and converting it into text information.

[0222] An "analysis device" is a device that analyzes user intent from textual information to understand their requests and objectives.

[0223] An "emotion analysis device" is a device that identifies emotions from a user's voice data and determines their emotional state, such as positive, negative, or neutral.

[0224] A "communication device" is a device that has the function of transmitting information to a designated recipient based on analyzed intentions and emotions.

[0225] An "output device" is a device that has the function of providing the generated response to the user as audio.

[0226] The "notification function" is a feature that detects emotional states during emergencies and quickly sends alerts to registered supporters and organizations.

[0227] This invention is primarily intended to be integrated into smartwatches and digital assistant devices as a system to support elderly users and those requiring emotional support. When a user speaks into the device, the smartwatch's microphone captures the audio data. The captured audio data is sent to a server using the PyAudio library. The audio is then converted into text data using the Google Cloud Speech-to-Text API.

[0228] The server analyzes the converted character data using the NLTK library to determine the user's intent and emotions. Sentiment analysis identifies emotional states such as positive, negative, and neutral. Based on the analysis results, the server generates an appropriate response and provides voice feedback to the user using a speech synthesis engine. This response is designed to alleviate the user's feelings of loneliness and anxiety.

[0229] Furthermore, if the emotion analysis detects excessive stress or anxiety, Firebase Cloud Messaging is used to send an alert to the user's pre-registered supporters or family members. This notification feature is an important means of receiving prompt support.

[0230] For example, if a user says in an everyday situation, "I feel like I've been getting tired more easily lately," the system will offer advice such as, "You must be tired. Try having some tea and taking a short break." Furthermore, the system will contact family members as needed, increasing the user's sense of security.

[0231] A concrete example of a prompt might be the instruction, "If the user says, 'I've been feeling down lately,' send a message to soothe their emotions." Based on such prompts, the generative AI model creates a flexible response.

[0232] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0233] Step 1:

[0234] When a user makes a voice input to their smartwatch, the device's microphone acquires the audio data. The input data is a raw audio signal and cannot be analyzed directly, so it is converted into a digital format for the next step.

[0235] Step 2:

[0236] The terminal uses the PyAudio library to convert the acquired audio data into a digital format and send it to the server. Here, the analog audio signal is processed as data packets and sent to the server.

[0237] Step 3:

[0238] The server uses the Google Cloud Speech-to-Text API to convert received digital audio data into text data. This process involves analyzing the audio waveform and converting the information contained in the audio into an alphabetical string. This results in a string that can be read by humans.

[0239] Step 4:

[0240] The server analyzes text data using the NLTK library to determine the user's intent. It extracts keywords from the input text and performs data analysis to identify the information and actions the user is seeking.

[0241] Step 5:

[0242] The server uses the NLTK library to recognize emotions from text data. It analyzes the frequency and context of specific words and performs data processing to classify emotional states as positive, negative, or neutral. This emotional information is then used in the next step.

[0243] Step 6:

[0244] The server generates appropriate responses using a prompt message generated by an AI model based on the analyzed intent and emotions. These responses include content designed to soothe the user's emotions. The responses are output in text format.

[0245] Step 7:

[0246] The server converts the generated text-based response into audio data using a speech synthesis engine and plays it back to the user through the terminal. Speech synthesis is the phase in which information is provided to the user in an auditory-friendly format using algorithms that mimic a more realistic human voice.

[0247] Step 8:

[0248] If excessive stress or anxiety is detected, the server uses Firebase Cloud Messaging to send an alert to pre-registered caregivers or family members. This step involves retrieving contact information from the database and sending the alert message.

[0249] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0250] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0251] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0252] [Second Embodiment]

[0253] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0254] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0255] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0256] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0257] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0258] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0259] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0260] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0261] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0262] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0263] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0264] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0265] In order to implement this invention, it is necessary to construct a system that includes the following components.

[0266] The system begins when a user makes a phone call. First, the terminal receives the user's voice and sends it to the server. The server uses speech recognition to convert the voice data into text data, and this text data is analyzed by intent analysis. Intent analysis identifies the user's intent, and an appropriate action is determined accordingly.

[0267] As an example, consider a case where a user says, "Call my son." The server analyzes the intent of "Call my son" from the text data and uses a memory device to identify the son's contact information, which is registered in advance. Then, the communication device uses that phone number to actually make the call.

[0268] On the other hand, if a user encounters an emergency and says "Help," the server analyzes this as an emergency keyword. The notification system is activated, automatically sending notifications to the user's pre-configured emergency contacts and external emergency services. In this case, the user's current location information and voice message may be included in the notification.

[0269] Furthermore, the results of communications and notifications, or the system's response, are fed back to the user using speech synthesis. The speech synthesis converts the generated response into natural-sounding speech and sends it to the terminal, which then plays it back to the user.

[0270] In this system, each function is designed to be intuitively operable by the user, taking into consideration that elderly people can use it despite limitations in vision and memory. As a result, users can communicate with peace of mind, easily obtain necessary information, and respond quickly in emergencies. This embodiment effectively achieves the objective of the invention.

[0271] The following describes the processing flow.

[0272] Step 1:

[0273] The user makes a call to a landline phone number. The device receives this incoming call and records the audio as digital audio data.

[0274] Step 2:

[0275] The terminal sends the recorded audio data to the server. The server receives the transmitted audio data and uses a speech recognition engine to convert it into text data.

[0276] Step 3:

[0277] The server analyzes the user's intention using the converted text data. Using intention analysis means, it interprets instructions such as "who to call" or "what to do" in the content of the call.

[0278] Step 4:

[0279] When the user's intention is analyzed, the server makes a call to the specified contact using the communication means. For example, when an instruction such as "call my son" is analyzed, the server searches for the son's phone number from the pre-registered contact information and makes a call.

[0280] Step 5:

[0281] The server generates a processing result and a response, and uses a speech synthesis engine to convert the text response into voice data.

[0282] Step 6:

[0283] The server transmits the generated voice data to the terminal. The terminal plays back this voice data and provides feedback to the user. For example, it makes the user hear a response such as "I'm calling your son".

[0284] Step 7:

[0285] When the user speaks "emergency" or "help", the server determines this as an emergency situation. The server automatically sends a notification using the notification means to the registered emergency contacts or emergency institutions. This notification may include the user's location information and voice message.

[0286] (Example 1)

[0287] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0288] There is a need to provide a system that allows elderly and visually impaired users to communicate intuitively without complex operations and to respond quickly in emergencies. Therefore, technology is required that facilitates a smooth process from voice input to decision-making and provides user-friendly feedback.

[0289] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0290] In this invention, the server includes voice receiving means for receiving voice input from a user, voice recognition means for converting the received voice into text data, and intent analysis means for analyzing the user's intent from the text data. This makes it possible for the user to easily communicate their intent to the system by voice and to quickly perform appropriate actions based on that intent.

[0291] "Voice receiving means" refers to a device or technology that has the function of detecting voice input from a user, converting its content into digital data, and transmitting it to subsequent processing.

[0292] "Speech recognition means" refers to a technology that analyzes speech data acquired from a speech receiving means and converts its content into text data.

[0293] "Intention analysis means" refers to a technology that analyzes text data obtained by speech recognition means and identifies the user's intended actions or requests based on that data.

[0294] "Communication means" refers to technology or devices that provide the function of establishing a connection with a specific communication partner based on an analyzed intent and sending and receiving information.

[0295] "Speech synthesis means" refers to a technology that converts responses generated by a system into a speech format and provides them to the user in an easily understandable way.

[0296] A "notification mechanism" is a technology that has the function of automatically notifying relevant external organizations or pre-configured recipients under specific conditions, particularly in emergency situations.

[0297] "Storage means" refers to a device or technology that has the function of storing a user's contact information and other necessary data, and making it quickly accessible and available as needed.

[0298] To implement this invention, it is necessary to construct an information processing system having a voice input interface and multiple technical components for processing voice. The user initiates an operation by voice, the terminal receives the voice, and the server processes it.

[0299] The server primarily functions as a means of receiving audio. When a user gives voice commands via a phone or smart device, the device captures this audio. This audio is converted into digital audio data and sent to the server. In this process, a typical mobile device or headset equipped with a microphone is often used as the hardware.

[0300] Next, the server operates as a "speech recognition tool," converting digital speech data into text data. This process can utilize, for example, a "speech recognition API." This conversion allows the user's speech to be obtained in text format, enabling subsequent intent analysis.

[0301] Next, the server analyzes the text data using "intent analysis tools" to understand the user's intent. This process utilizes "generative AI models" and "language understanding APIs" that perform natural language processing. This allows the server to identify specific action requests and inquiries from the user's utterances.

[0302] As a practical example, when a user says "Call my son", the server recognizes the "make a call" instruction from the character data and obtains an appropriate phone number by referring to the pre-registered contacts. Furthermore, the "communication means" of the server operates to make a call to the identified contact. This process is realized through the interface with the "communication API" and the "phone service provider".

[0303] When the user utters an emergency phrase such as "Help", the server utilizes the "notification means" to automatically send location information and voice messages to the recipients designated as emergency situations. Notifications to pre-set emergency contacts are made, etc.

[0304] Finally, the server uses the "voice synthesis means" to convert the system response into a natural voice format. As a result, the user can receive feedback from the system in voice through the terminal. It is assumed that the "voice synthesis API" etc. are used for this voice synthesis.

[0305] As an example of a prompt sentence, "Check the number of my son from my phone book and make a call" can be considered. With this system, the user can intuitively operate the device by voice, enabling smooth information acquisition and emergency response.

[0306] The flow of the specific process in Example 1 will be described using FIG. 11.

[0307] Step 1:

[0308] The user starts operating by voice.

[0309] Input: User's voice instruction (e.g., "Call my son")

[0310] Output: Voice data

[0311] Specific operation: The user speaks a specified instruction into the microphone of their smartphone or other voice-enabled device. This voice is captured by the device's microphone and recorded as digital audio data.

[0312] Step 2:

[0313] The device sends audio to the server.

[0314] Input: Digital audio data

[0315] Output: Transfer of audio data

[0316] Specific operation: The terminal sends the captured digital audio data to the server via the network. This enables data processing on the server side.

[0317] Step 3:

[0318] The server converts the audio to text.

[0319] Input: Digital audio data

[0320] Output: Character data

[0321] Specific operation: The server uses speech recognition technology (e.g., "Speech Recognition API") to convert the received audio data into text data. This conversion process involves analyzing sound waves to convert spoken words into text format.

[0322] Step 4:

[0323] The server analyzes the intent.

[0324] Input: Text data

[0325] Output: User intent (e.g., "Make a phone call")

[0326] Specific operation: The server uses generative AI models and language understanding APIs to analyze text data and identify the actions and information the user is seeking. This allows for a clear understanding of what the user wants.

[0327] Step 5:

[0328] The server initiates communication.

[0329] Input: User intent and contact information

[0330] Output: Communication execution

[0331] Specific operation: Based on the user's intent, the server refers to pre-stored contact information and performs the necessary communication. This process involves using communication methods to perform actions such as making phone calls or sending messages.

[0332] Step 6:

[0333] The server will send notifications (if necessary).

[0334] Input: Emergency phrases or conditions

[0335] Output: Notification to external organizations and contacts

[0336] Specific operation: When a user utters an emergency phrase such as "Help," the server activates an emergency notification system and sends a notification containing location information and the situation to pre-configured recipients. This enables a rapid response.

[0337] Step 7:

[0338] The server generates an audio response.

[0339] Input: System response information

[0340] Output: Audio data

[0341] Specific operation: The server uses speech synthesis technology to convert the generated response into natural-sounding speech. This speech data is then used as feedback to the user.

[0342] Step 8:

[0343] The terminal plays the response.

[0344] Input: Audio data

[0345] Output: Voice feedback to the user

[0346] Specific operation: The terminal receives audio data sent from the server and plays it back to the user through the speaker. This allows the user to confirm the system's response.

[0347] (Application Example 1)

[0348] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0349] There is a problem in that users who are unfamiliar with technology, such as the elderly, have difficulty using voice control to quickly and easily perform important communication and respond to emergencies. For example, they may forget when to take their medication or be unable to immediately access the appropriate contacts in an emergency. Therefore, there is a need to provide technology that supports users with tasks necessary for daily life intuitively and without burden, and that allows them to respond with confidence in emergencies.

[0350] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0351] In this invention, the server includes voice acquisition means for acquiring voice information from the user, voice conversion means for converting the acquired voice information into text data, and purpose analysis means for analyzing the user's purpose from the text data. This allows the user to operate intuitively through voice, prompt specific actions at pre-set times, and provide rapid notifications in emergencies.

[0352] A "voice acquisition means" is a device that has the function of receiving voice information from a user and transferring it to a server for analysis.

[0353] A "speech conversion means" is a device that converts acquired speech information into text data.

[0354] A "purpose analysis tool" is a device that analyzes the purpose of a user's utterance from text data and identifies their intent.

[0355] An "information transmission means" is a device that transmits appropriate information to selected contacts based on the results of an analysis conducted according to the user's objectives.

[0356] A "speech generation means" is a device that has the function of converting the generated response into speech and presenting it to the user audibly.

[0357] A "command generation means" is a device that has the function of generating commands to prompt the user to take a specific action based on the results of objective analysis.

[0358] An "information storage device" is a device that stores user contact information and related data and has the function of referencing it as needed.

[0359] To implement this invention, it is necessary to construct a series of systems including voice acquisition means, voice conversion means, purpose analysis means, information transmission means, voice generation means, command generation means, and information storage means. The server first receives voice data sent from the terminal. Then, it converts this voice data into text data using the voice conversion means. The Google Cloud Speech-to-Text API can be used for voice conversion. Next, the purpose analysis means analyzes the text data using a BERT model to identify the user's intent. Once the intent is identified, the information transmission means sends the information to the user's designated contact via the Twilio API. In emergencies, the command generation means responds to the user with voice generated using Amazon Polly and instructs them on the necessary actions.

[0360] This system is designed to be intuitive and easy to use, especially for users unfamiliar with technology, such as the elderly. It provides not only daily support but also rapid response in emergencies. For example, if a user says, "Tell me when to take my medication," the system will refer to a pre-set schedule and notify them via voice to take their medication at the appropriate time. Similarly, if a user says, "Help me," an emergency contact will be automatically notified.

[0361] A concrete example of a prompt might be, "How to respond when the user says, 'Tell me when to take my medication'." Using this prompt, it becomes possible to generate more natural responses using a generative AI model.

[0362] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0363] Step 1:

[0364] The terminal receives voice input from the user. This voice input includes spoken information from the user. The terminal then prepares to send this voice data to the server.

[0365] Input: User's voice information

[0366] Output: Audio data ready to be sent to the server.

[0367] Step 2:

[0368] The server converts the audio data sent from the terminal into text data using a speech-to-text conversion method. Specifically, it uses the Google Cloud Speech-to-Text API to convert speech to text.

[0369] Input: Audio data

[0370] Output: Converted character data

[0371] Step 3:

[0372] The server analyzes the generated text data using a target analysis tool. This analysis utilizes the BERT model to perform data processing that identifies the user's intent from their utterance.

[0373] Input: Text data

[0374] Output: Analyzed user intent

[0375] Step 4:

[0376] The server decides its actions based on the results of the purpose analysis. For example, if the identified intention is "tell me when to take my medicine," it will access the information storage system to obtain the necessary schedule information.

[0377] Input: Analyzed user intent

[0378] Output: Decision made or information

[0379] Step 5:

[0380] Based on the decided action, the server provides necessary information to the user via an information transmission means. This information is output as voice using a voice generation means and Amazon Polly.

[0381] Input: Decision made or information

[0382] Output: Voice feedback to the user

[0383] Step 6:

[0384] The user takes necessary actions based on the audio feedback received from the server. For example, they might take medication according to the time notified.

[0385] Input: Voice feedback

[0386] Output: User behavior

[0387] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0388] This invention provides a dialogue system that incorporates an emotion engine in addition to the conventional process of receiving user voice input and converting voice data into text. The system is implemented according to the following procedure.

[0389] First, the user makes a call to a fixed phone number to input their voice. The terminal records the user's voice in digital format and sends it to the server. The server uses speech recognition to convert the received voice data into text data.

[0390] Next, the server analyzes the text data using intent analysis tools to understand the user's requests and objectives. This includes things like "who to call" and "how to handle emergencies." Furthermore, it analyzes the user's emotions using newly incorporated emotion recognition tools. This analysis allows the server to determine whether the emotion is positive, negative, or neutral.

[0391] For example, if a user says, "I feel very anxious today," the server will determine through intent analysis that a conversation is necessary and detect anxiety through sentiment analysis. Based on these results, the server will generate words of comfort and encouragement.

[0392] The generated response is converted into audio data using a speech synthesis engine and played back to the user through the device. This provides the user with a sense of security.

[0393] Furthermore, if the emotion recognition system identifies the user's emotions and detects excessively strong anxiety or stress, the server uses a notification system to send an alert to pre-registered family members or caregivers. This feature enables a quick response and support.

[0394] The overall processing of this system aims to provide more personalized care to elderly users and those who require emotional support. This can improve users' quality of life and reduce feelings of loneliness.

[0395] The following describes the processing flow.

[0396] Step 1:

[0397] The user makes a call to a landline phone number. The device receives this incoming call and records the user's voice as digital audio data.

[0398] Step 2:

[0399] The terminal sends the recorded audio data to the server. The server converts the received audio data into text data using speech recognition technology.

[0400] Step 3:

[0401] The server analyzes text data using intent analysis tools. It understands the user's requests and objectives and determines what the appropriate action is. In this process, it identifies voice content such as "who to call" or "to talk about anxieties."

[0402] Step 4:

[0403] The server uses emotion recognition technology to analyze the user's voice to determine their emotions. This analysis determines whether the user is anxious, happy, or in any other emotional state.

[0404] Step 5:

[0405] Based on the analyzed emotional state, the server determines the response. For example, if anxiety is detected, it generates words of comfort or encouragement and takes corresponding action.

[0406] Step 6:

[0407] The server converts the generated response into audio data using a speech synthesis system. The terminal receives this audio data and plays it back to the user to provide feedback. The user receives an appropriate response based on the situation.

[0408] Step 7:

[0409] If the emotion recognition system detects strong anxiety or stress, the server sends an emergency notification to family members or caregivers via a notification system. The notification will include information about the user's emotional state and the support needed.

[0410] (Example 2)

[0411] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0412] Conventional interactive information processing systems have struggled to accurately understand user intent and generate appropriate responses, particularly failing to provide responses that take user emotions into consideration. Furthermore, the lack of mechanisms for quickly notifying external organizations when users experience anxiety or stress makes immediate response difficult, especially in urgent situations.

[0413] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0414] In this invention, the server includes a device means for receiving voice input, a conversion means for converting voice to text, an analysis means for analyzing intent from text, an emotion recognition means for determining the user's emotions, a response generation means for generating a response using a generative AI model, and a voice output means for outputting the generated response as voice. This enables accurate understanding of the user's intent, the provision of natural responses that are in line with their emotions, and the ability to quickly notify external organizations if anxiety or stress is detected, thereby realizing appropriate support for the user.

[0415] A "user" refers to an individual or group that uses the system, and is the entity that provides or obtains information through voice input.

[0416] "Voice input" refers to the format in which the words spoken by the user are incorporated into an information processing system.

[0417] "Device means" refers to equipment or devices for inputting and outputting sound and information, and primarily functions as an interface with the user.

[0418] "Conversion means" refers to technologies and devices that convert audio data into text data, and involves the process of changing speech into text through speech recognition.

[0419] "Analysis methods" refer to technologies and algorithms used to understand user intent and content from text data, and play a role in extracting user requests and objectives.

[0420] "Emotion recognition means" refers to technology that determines the user's emotional state from input voice or text and uses that determination to provide an appropriate response.

[0421] A "generative AI model" refers to a generative model that utilizes artificial intelligence technology to automatically generate natural-sounding text based on user input data.

[0422] "Response generation means" refers to the process or technology of generating a message to respond to the user based on analysis results and emotion recognition data.

[0423] "Audio output means" refers to the technology or device that converts text generated by the system into speech and outputs it in a format that can be heard by the user.

[0424] An "information processing system" refers to a computer system that integrates the above means to enable interaction with the user, and is responsible for processing and outputting input data.

[0425] This invention is an interactive information processing system that utilizes user voice input. The user dials a specific telephone number using a communication terminal and inputs voice. The terminal records the user's voice in digital voice format and sends the data to a server. The server converts the received voice data into text data using speech recognition software. A general-purpose speech recognition engine is suitable for use as the software for this purpose.

[0426] Next, the server uses a generative AI model to analyze the text data, understand the user's intent, and perform sentiment recognition. This involves applying software that utilizes natural language processing (NLP) technology. This analysis allows the system to identify what requests or emotions are underlying the user's statements.

[0427] For example, if a user says, "I feel very anxious today," this text is converted by speech recognition, and the emotion of anxiety is extracted through sentiment analysis. Based on this, the server inputs a prompt into the generation AI model to generate words of comfort. An example of a prompt might be an instruction such as, "Generate an appropriate response for when the user feels anxious."

[0428] The generated response is converted back into voice data using a speech synthesis engine and returned to the user via the terminal. This allows the user to naturally receive feedback from the system. Furthermore, emotion recognition means that if the user shows particularly strong stress or anxiety, the server automatically sends a notification to a pre-registered external organization, enabling a rapid response. This entire process provides the user with personalized support and enhances their sense of security.

[0429] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0430] Step 1:

[0431] The user makes a call to a specific phone number and enters a message by voice. This voice is recorded digitally by the device. The input is an audio signal, and the output is digital audio data. The device performs the necessary encoding to capture the audio signal in high quality.

[0432] Step 2:

[0433] The terminal sends recorded digital audio data to the server. The server receives this data and converts it into text data using speech recognition software. The input is digital audio data, and the output is a text string. The server analyzes the phonemes from the audio data and generates the corresponding text.

[0434] Step 3:

[0435] The server analyzes the generated text data and performs natural language processing to understand the user's intent. In this process, intent analysis is used to determine what the user is requesting in their utterance. The input is text data, and the output is intent information. Intent analysis uses algorithms to extract specific keywords and phrases from the text.

[0436] Step 4:

[0437] The server further detects the user's emotional state by performing sentiment recognition through text analysis. Using sentiment analysis algorithms, it outputs sentiment labels (e.g., positive, anxious, negative) from the input text data. Sentiment recognition utilizes machine learning models to identify language patterns and tones.

[0438] Step 5:

[0439] The server uses a generative AI model to generate an appropriate response based on intent information and emotion labels. A prompt (e.g., "Generate an appropriate response when the user feels anxious.") is input to the AI ​​model, causing it to output a text response. In this process, the AI ​​model performs a complex language generation task.

[0440] Step 6:

[0441] The generated text response is converted into speech data using a speech synthesis engine. The input is the response text, and the output is the synthesized speech data. Speech synthesis involves the process of converting text into sounds with natural pronunciation.

[0442] Step 7:

[0443] The server sends the converted audio data to the terminal, which then plays it back to the user. The user can then hear the system's response. The input is synthesized speech data, and the output is the audible sound. The terminal provides high-quality sound output.

[0444] Step 8:

[0445] The server performs a function that sends a notification to a pre-registered external organization when the emotion recognition system detects an emotion exceeding a certain threshold (e.g., severe anxiety). The input is the emotion recognition result, and the output is the notification message. This function enables security and rapid response.

[0446] (Application Example 2)

[0447] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0448] A challenge exists in that elderly users and those requiring emotional support often struggle to receive immediate and appropriate assistance when they experience feelings of loneliness or anxiety. Furthermore, these emotional states can persist, potentially leading to a decline in their quality of life. While conventional technologies have succeeded in analyzing user intentions, they have been insufficient in rapidly detecting emotional changes and providing the necessary support.

[0449] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0450] In this invention, the server includes an input device for receiving voice input from the user, a recognition device for converting the received voice into text information, an analysis device for analyzing the user's intent, an emotion analysis device for identifying the user's emotions, a communication device for communicating with a designated recipient based on the analyzed intent and emotions, and an output device for outputting the generated response as voice. This makes it possible to quickly detect the emotional state of the user when they feel anxious or lonely and provide appropriate support. Furthermore, if excessive stress or anxiety is detected, an alert can be quickly sent to a supporter using the notification function.

[0451] An "input device" is a device that has the function of receiving voice data from the user and incorporating it into the system.

[0452] A "recognition device" is a device that has the function of analyzing received audio data and converting it into text information.

[0453] An "analysis device" is a device that analyzes user intent from textual information to understand their requests and objectives.

[0454] An "emotion analysis device" is a device that identifies emotions from a user's voice data and determines their emotional state, such as positive, negative, or neutral.

[0455] A "communication device" is a device that has the function of transmitting information to a designated recipient based on analyzed intentions and emotions.

[0456] An "output device" is a device that has the function of providing the generated response to the user as audio.

[0457] The "notification function" is a feature that detects emotional states during emergencies and quickly sends alerts to registered supporters and organizations.

[0458] This invention is primarily intended to be integrated into smartwatches and digital assistant devices as a system to support elderly users and those requiring emotional support. When a user speaks into the device, the smartwatch's microphone captures the audio data. The captured audio data is sent to a server using the PyAudio library. The audio is then converted into text data using the Google Cloud Speech-to-Text API.

[0459] The server analyzes the converted character data using the NLTK library to determine the user's intent and emotions. Sentiment analysis identifies emotional states such as positive, negative, and neutral. Based on the analysis results, the server generates an appropriate response and provides voice feedback to the user using a speech synthesis engine. This response is designed to alleviate the user's feelings of loneliness and anxiety.

[0460] Furthermore, if the emotion analysis detects excessive stress or anxiety, Firebase Cloud Messaging is used to send an alert to the user's pre-registered supporters or family members. This notification feature is an important means of receiving prompt support.

[0461] For example, if a user says in an everyday situation, "I feel like I've been getting tired more easily lately," the system will offer advice such as, "You must be tired. Try having some tea and taking a short break." Furthermore, the system will contact family members as needed, increasing the user's sense of security.

[0462] A concrete example of a prompt might be the instruction, "If the user says, 'I've been feeling down lately,' send a message to soothe their emotions." Based on such prompts, the generative AI model creates a flexible response.

[0463] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0464] Step 1:

[0465] When a user makes a voice input to their smartwatch, the device's microphone acquires the audio data. The input data is a raw audio signal and cannot be analyzed directly, so it is converted into a digital format for the next step.

[0466] Step 2:

[0467] The terminal uses the PyAudio library to convert the acquired audio data into a digital format and send it to the server. Here, the analog audio signal is processed as data packets and sent to the server.

[0468] Step 3:

[0469] The server uses the Google Cloud Speech-to-Text API to convert received digital audio data into text data. This process involves analyzing the audio waveform and converting the information contained in the audio into an alphabetical string. This results in a string that can be read by humans.

[0470] Step 4:

[0471] The server analyzes text data using the NLTK library to determine the user's intent. It extracts keywords from the input text and performs data analysis to identify the information and actions the user is seeking.

[0472] Step 5:

[0473] The server uses the NLTK library to recognize emotions from text data. It analyzes the frequency and context of specific words and performs data processing to classify emotional states as positive, negative, or neutral. This emotional information is then used in the next step.

[0474] Step 6:

[0475] The server generates appropriate responses using a prompt message generated by an AI model based on the analyzed intent and emotions. These responses include content designed to soothe the user's emotions. The responses are output in text format.

[0476] Step 7:

[0477] The server converts the generated text-based response into audio data using a speech synthesis engine and plays it back to the user through the terminal. Speech synthesis is the phase in which information is provided to the user in an auditory-friendly format using algorithms that mimic a more realistic human voice.

[0478] Step 8:

[0479] If excessive stress or anxiety is detected, the server uses Firebase Cloud Messaging to send an alert to pre-registered caregivers or family members. This step involves retrieving contact information from the database and sending the alert message.

[0480] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0481] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0482] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0483] [Third Embodiment]

[0484] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0485] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0486] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0487] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0488] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0489] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0490] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0491] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0492] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0493] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0494] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0495] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0496] In order to implement this invention, it is necessary to construct a system that includes the following components.

[0497] The system begins when a user makes a phone call. First, the terminal receives the user's voice and sends it to the server. The server uses speech recognition to convert the voice data into text data, and this text data is analyzed by intent analysis. Intent analysis identifies the user's intent, and an appropriate action is determined accordingly.

[0498] As an example, consider a case where a user says, "Call my son." The server analyzes the intent of "Call my son" from the text data and uses a memory device to identify the son's contact information, which is registered in advance. Then, the communication device uses that phone number to actually make the call.

[0499] On the other hand, if a user encounters an emergency and says "Help," the server analyzes this as an emergency keyword. The notification system is activated, automatically sending notifications to the user's pre-configured emergency contacts and external emergency services. In this case, the user's current location information and voice message may be included in the notification.

[0500] Furthermore, the results of communications and notifications, or the system's response, are fed back to the user using speech synthesis. The speech synthesis converts the generated response into natural-sounding speech and sends it to the terminal, which then plays it back to the user.

[0501] In this system, each function is designed to be intuitively operable by the user, taking into consideration that elderly people can use it despite limitations in vision and memory. As a result, users can communicate with peace of mind, easily obtain necessary information, and respond quickly in emergencies. This embodiment effectively achieves the objective of the invention.

[0502] The following describes the processing flow.

[0503] Step 1:

[0504] The user makes a call to a landline phone number. The device receives this incoming call and records the audio as digital audio data.

[0505] Step 2:

[0506] The terminal sends the recorded audio data to the server. The server receives the transmitted audio data and uses a speech recognition engine to convert it into text data.

[0507] Step 3:

[0508] The server uses the converted text data to analyze the user's intent. Intent analysis tools are used to interpret the content of the call, such as "who to call" or "what to do."

[0509] Step 4:

[0510] Once the user's intent is analyzed, the server uses a communication method to make a phone call to the specified contact. For example, if the instruction is "Call my son," the server will search for the son's phone number from the pre-registered contact information and make the call.

[0511] Step 5:

[0512] The server generates processing results and responses, and uses a speech synthesis engine to convert text responses into speech data.

[0513] Step 6:

[0514] The server sends the generated audio data to the terminal. The terminal plays this audio data and provides feedback to the user. For example, it might play a response like, "I'm calling your son."

[0515] Step 7:

[0516] If a user says "emergency" or "help," the server will recognize this as an emergency. The server will then use notification methods to automatically send notifications to registered emergency contacts and emergency services. These notifications may include the user's location information and a voice message.

[0517] (Example 1)

[0518] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0519] There is a need to provide a system that allows elderly and visually impaired users to communicate intuitively without complex operations and to respond quickly in emergencies. Therefore, technology is required that facilitates a smooth process from voice input to decision-making and provides user-friendly feedback.

[0520] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0521] In this invention, the server includes voice receiving means for receiving voice input from a user, voice recognition means for converting the received voice into text data, and intent analysis means for analyzing the user's intent from the text data. This makes it possible for the user to easily communicate their intent to the system by voice and to quickly perform appropriate actions based on that intent.

[0522] "Voice receiving means" refers to a device or technology that has the function of detecting voice input from a user, converting its content into digital data, and transmitting it to subsequent processing.

[0523] "Speech recognition means" refers to a technology that analyzes speech data acquired from a speech receiving means and converts its content into text data.

[0524] "Intention analysis means" refers to a technology that analyzes text data obtained by speech recognition means and identifies the user's intended actions or requests based on that data.

[0525] "Communication means" refers to technology or devices that provide the function of establishing a connection with a specific communication partner based on an analyzed intent and sending and receiving information.

[0526] "Speech synthesis means" refers to a technology that converts responses generated by a system into a speech format and provides them to the user in an easily understandable way.

[0527] A "notification mechanism" is a technology that has the function of automatically notifying relevant external organizations or pre-configured recipients under specific conditions, particularly in emergency situations.

[0528] "Storage means" refers to a device or technology that has the function of storing a user's contact information and other necessary data, and making it quickly accessible and available as needed.

[0529] To implement this invention, it is necessary to construct an information processing system having a voice input interface and multiple technical components for processing voice. The user initiates an operation by voice, the terminal receives the voice, and the server processes it.

[0530] The server primarily functions as a means of receiving audio. When a user gives voice commands via a phone or smart device, the device captures this audio. This audio is converted into digital audio data and sent to the server. In this process, a typical mobile device or headset equipped with a microphone is often used as the hardware.

[0531] Next, the server operates as a "speech recognition tool," converting digital speech data into text data. This process can utilize, for example, a "speech recognition API." This conversion allows the user's speech to be obtained in text format, enabling subsequent intent analysis.

[0532] Next, the server analyzes the text data using "intent analysis tools" to understand the user's intent. This process utilizes "generative AI models" and "language understanding APIs" that perform natural language processing. This allows the server to identify specific action requests and inquiries from the user's utterances.

[0533] As a practical example, if a user says, "Call my son," the server recognizes this instruction from the text data, retrieves the appropriate phone number by referring to pre-registered contacts, and then activates the server's "communication means" to place a call to the identified contact. This process is achieved through interfaces with "communication APIs" and "telephone service providers."

[0534] If a user utters an emergency phrase such as "Help," the server utilizes its "notification system" to automatically send location information and a voice message to designated recipients in an emergency. Notifications are also sent to pre-configured emergency contacts.

[0535] Finally, the server uses a "speech synthesis method" to convert the system response into a natural-sounding voice format. This allows the user to receive feedback from the system in voice through their terminal. It is expected that a "speech synthesis API" or similar will be used for this speech synthesis.

[0536] An example of a prompt might be, "Find my son's number in my phone book and make a call." This system allows users to intuitively operate the device with their voice, enabling smooth information retrieval and emergency response.

[0537] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0538] Step 1:

[0539] The user initiates the operation using voice commands.

[0540] Input: User voice command (e.g., "Call my son")

[0541] Output: Audio data

[0542] Specific operation: The user speaks a specified instruction into the microphone of their smartphone or other voice-enabled device. This voice is captured by the device's microphone and recorded as digital audio data.

[0543] Step 2:

[0544] The device sends audio to the server.

[0545] Input: Digital audio data

[0546] Output: Transfer of audio data

[0547] Specific operation: The terminal sends the captured digital audio data to the server via the network. This enables data processing on the server side.

[0548] Step 3:

[0549] The server converts the audio to text.

[0550] Input: Digital audio data

[0551] Output: Character data

[0552] Specific operation: The server uses speech recognition technology (e.g., "Speech Recognition API") to convert the received audio data into text data. This conversion process involves analyzing sound waves to convert spoken words into text format.

[0553] Step 4:

[0554] The server analyzes the intent.

[0555] Input: Text data

[0556] Output: User intent (e.g., "Make a phone call")

[0557] Specific operation: The server uses generative AI models and language understanding APIs to analyze text data and identify the actions and information the user is seeking. This allows for a clear understanding of what the user wants.

[0558] Step 5:

[0559] The server initiates communication.

[0560] Input: User intent and contact information

[0561] Output: Communication execution

[0562] Specific operation: Based on the user's intent, the server refers to pre-stored contact information and performs the necessary communication. This process involves using communication methods to perform actions such as making phone calls or sending messages.

[0563] Step 6:

[0564] The server will send notifications (if necessary).

[0565] Input: Emergency phrases or conditions

[0566] Output: Notification to external organizations and contacts

[0567] Specific operation: When a user utters an emergency phrase such as "Help," the server activates an emergency notification system and sends a notification containing location information and the situation to pre-configured recipients. This enables a rapid response.

[0568] Step 7:

[0569] The server generates an audio response.

[0570] Input: System response information

[0571] Output: Audio data

[0572] Specific operation: The server uses speech synthesis technology to convert the generated response into natural-sounding speech. This speech data is then used as feedback to the user.

[0573] Step 8:

[0574] The terminal plays the response.

[0575] Input: Audio data

[0576] Output: Voice feedback to the user

[0577] Specific operation: The terminal receives audio data sent from the server and plays it back to the user through the speaker. This allows the user to confirm the system's response.

[0578] (Application Example 1)

[0579] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0580] There is a problem in that users who are unfamiliar with technology, such as the elderly, have difficulty using voice control to quickly and easily perform important communication and respond to emergencies. For example, they may forget when to take their medication or be unable to immediately access the appropriate contacts in an emergency. Therefore, there is a need to provide technology that supports users with tasks necessary for daily life intuitively and without burden, and that allows them to respond with confidence in emergencies.

[0581] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0582] In this invention, the server includes voice acquisition means for acquiring voice information from the user, voice conversion means for converting the acquired voice information into text data, and purpose analysis means for analyzing the user's purpose from the text data. This allows the user to operate intuitively through voice, prompt specific actions at pre-set times, and provide rapid notifications in emergencies.

[0583] A "voice acquisition means" is a device that has the function of receiving voice information from a user and transferring it to a server for analysis.

[0584] A "speech conversion means" is a device that converts acquired speech information into text data.

[0585] A "purpose analysis tool" is a device that analyzes the purpose of a user's utterance from text data and identifies their intent.

[0586] An "information transmission means" is a device that transmits appropriate information to selected contacts based on the results of an analysis conducted according to the user's objectives.

[0587] A "speech generation means" is a device that has the function of converting the generated response into speech and presenting it to the user audibly.

[0588] A "command generation means" is a device that has the function of generating commands to prompt the user to take a specific action based on the results of objective analysis.

[0589] An "information storage device" is a device that stores user contact information and related data and has the function of referencing it as needed.

[0590] To implement this invention, it is necessary to construct a series of systems including voice acquisition means, voice conversion means, purpose analysis means, information transmission means, voice generation means, command generation means, and information storage means. The server first receives voice data sent from the terminal. Then, it converts this voice data into text data using the voice conversion means. The Google Cloud Speech-to-Text API can be used for voice conversion. Next, the purpose analysis means analyzes the text data using a BERT model to identify the user's intent. Once the intent is identified, the information transmission means sends the information to the user's designated contact via the Twilio API. In emergencies, the command generation means responds to the user with voice generated using Amazon Polly and instructs them on the necessary actions.

[0591] This system is designed to be intuitive and easy to use, especially for users unfamiliar with technology, such as the elderly. It provides not only daily support but also rapid response in emergencies. For example, if a user says, "Tell me when to take my medication," the system will refer to a pre-set schedule and notify them via voice to take their medication at the appropriate time. Similarly, if a user says, "Help me," an emergency contact will be automatically notified.

[0592] A concrete example of a prompt might be, "How to respond when the user says, 'Tell me when to take my medication'." Using this prompt, it becomes possible to generate more natural responses using a generative AI model.

[0593] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0594] Step 1:

[0595] The terminal receives voice input from the user. This voice input includes spoken information from the user. The terminal then prepares to send this voice data to the server.

[0596] Input: User's voice information

[0597] Output: Audio data ready to be sent to the server.

[0598] Step 2:

[0599] The server converts the audio data sent from the terminal into text data using a speech-to-text conversion method. Specifically, it uses the Google Cloud Speech-to-Text API to convert speech to text.

[0600] Input: Audio data

[0601] Output: Converted character data

[0602] Step 3:

[0603] The server analyzes the generated text data using a target analysis tool. This analysis utilizes the BERT model to perform data processing that identifies the user's intent from their utterance.

[0604] Input: Text data

[0605] Output: Analyzed user intent

[0606] Step 4:

[0607] The server decides its actions based on the results of the purpose analysis. For example, if the identified intention is "tell me when to take my medicine," it will access the information storage system to obtain the necessary schedule information.

[0608] Input: Analyzed user intent

[0609] Output: Decision made or information

[0610] Step 5:

[0611] Based on the decided action, the server provides necessary information to the user via an information transmission means. This information is output as voice using a voice generation means and Amazon Polly.

[0612] Input: Decision made or information

[0613] Output: Voice feedback to the user

[0614] Step 6:

[0615] The user takes necessary actions based on the audio feedback received from the server. For example, they might take medication according to the time notified.

[0616] Input: Voice feedback

[0617] Output: User behavior

[0618] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0619] This invention provides a dialogue system that incorporates an emotion engine in addition to the conventional process of receiving user voice input and converting voice data into text. The system is implemented according to the following procedure.

[0620] First, the user makes a call to a fixed phone number to input their voice. The terminal records the user's voice in digital format and sends it to the server. The server uses speech recognition to convert the received voice data into text data.

[0621] Next, the server analyzes the text data using intent analysis tools to understand the user's requests and objectives. This includes things like "who to call" and "how to handle emergencies." Furthermore, it analyzes the user's emotions using newly incorporated emotion recognition tools. This analysis allows the server to determine whether the emotion is positive, negative, or neutral.

[0622] For example, if a user says, "I feel very anxious today," the server will determine through intent analysis that a conversation is necessary and detect anxiety through sentiment analysis. Based on these results, the server will generate words of comfort and encouragement.

[0623] The generated response is converted into audio data using a speech synthesis engine and played back to the user through the device. This provides the user with a sense of security.

[0624] Furthermore, if the emotion recognition system identifies the user's emotions and detects excessively strong anxiety or stress, the server uses a notification system to send an alert to pre-registered family members or caregivers. This feature enables a quick response and support.

[0625] The overall processing of this system aims to provide more personalized care to elderly users and those who require emotional support. This can improve users' quality of life and reduce feelings of loneliness.

[0626] The following describes the processing flow.

[0627] Step 1:

[0628] The user makes a call to a landline phone number. The device receives this incoming call and records the user's voice as digital audio data.

[0629] Step 2:

[0630] The terminal sends the recorded audio data to the server. The server converts the received audio data into text data using speech recognition technology.

[0631] Step 3:

[0632] The server analyzes text data using intent analysis tools. It understands the user's requests and objectives and determines what the appropriate action is. In this process, it identifies voice content such as "who to call" or "to talk about anxieties."

[0633] Step 4:

[0634] The server uses emotion recognition technology to analyze the user's voice to determine their emotions. This analysis determines whether the user is anxious, happy, or in any other emotional state.

[0635] Step 5:

[0636] Based on the analyzed emotional state, the server determines the response. For example, if anxiety is detected, it generates words of comfort or encouragement and takes corresponding action.

[0637] Step 6:

[0638] The server converts the generated response into audio data using a speech synthesis system. The terminal receives this audio data and plays it back to the user to provide feedback. The user receives an appropriate response based on the situation.

[0639] Step 7:

[0640] If the emotion recognition system detects strong anxiety or stress, the server sends an emergency notification to family members or caregivers via a notification system. The notification will include information about the user's emotional state and the support needed.

[0641] (Example 2)

[0642] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0643] Conventional interactive information processing systems have struggled to accurately understand user intent and generate appropriate responses, particularly failing to provide responses that take user emotions into consideration. Furthermore, the lack of mechanisms for quickly notifying external organizations when users experience anxiety or stress makes immediate response difficult, especially in urgent situations.

[0644] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0645] In this invention, the server includes a device means for receiving voice input, a conversion means for converting voice to text, an analysis means for analyzing intent from text, an emotion recognition means for determining the user's emotions, a response generation means for generating a response using a generative AI model, and a voice output means for outputting the generated response as voice. This enables accurate understanding of the user's intent, the provision of natural responses that are in line with their emotions, and the ability to quickly notify external organizations if anxiety or stress is detected, thereby realizing appropriate support for the user.

[0646] A "user" refers to an individual or group that uses the system, and is the entity that provides or obtains information through voice input.

[0647] "Voice input" refers to the format in which the words spoken by the user are incorporated into an information processing system.

[0648] "Device means" refers to equipment or devices for inputting and outputting sound and information, and primarily functions as an interface with the user.

[0649] "Conversion means" refers to technologies and devices that convert audio data into text data, and involves the process of changing speech into text through speech recognition.

[0650] "Analysis methods" refer to technologies and algorithms used to understand user intent and content from text data, and play a role in extracting user requests and objectives.

[0651] "Emotion recognition means" refers to technology that determines the user's emotional state from input voice or text and uses that determination to provide an appropriate response.

[0652] A "generative AI model" refers to a generative model that utilizes artificial intelligence technology to automatically generate natural-sounding text based on user input data.

[0653] "Response generation means" refers to the process or technology of generating a message to respond to the user based on analysis results and emotion recognition data.

[0654] "Audio output means" refers to the technology or device that converts text generated by the system into speech and outputs it in a format that can be heard by the user.

[0655] An "information processing system" refers to a computer system that integrates the above means to enable interaction with the user, and is responsible for processing and outputting input data.

[0656] This invention is an interactive information processing system that utilizes user voice input. The user dials a specific telephone number using a communication terminal and inputs voice. The terminal records the user's voice in digital voice format and sends the data to a server. The server converts the received voice data into text data using speech recognition software. A general-purpose speech recognition engine is suitable for use as the software for this purpose.

[0657] Next, the server uses a generative AI model to analyze the text data, understand the user's intent, and perform sentiment recognition. This involves applying software that utilizes natural language processing (NLP) technology. This analysis allows the system to identify what requests or emotions are underlying the user's statements.

[0658] For example, if a user says, "I feel very anxious today," this text is converted by speech recognition, and the emotion of anxiety is extracted through sentiment analysis. Based on this, the server inputs a prompt into the generation AI model to generate words of comfort. An example of a prompt might be an instruction such as, "Generate an appropriate response for when the user feels anxious."

[0659] The generated response is converted back into voice data using a speech synthesis engine and returned to the user via the terminal. This allows the user to naturally receive feedback from the system. Furthermore, emotion recognition means that if the user shows particularly strong stress or anxiety, the server automatically sends a notification to a pre-registered external organization, enabling a rapid response. This entire process provides the user with personalized support and enhances their sense of security.

[0660] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0661] Step 1:

[0662] The user makes a call to a specific phone number and enters a message by voice. This voice is recorded digitally by the device. The input is an audio signal, and the output is digital audio data. The device performs the necessary encoding to capture the audio signal in high quality.

[0663] Step 2:

[0664] The terminal sends recorded digital audio data to the server. The server receives this data and converts it into text data using speech recognition software. The input is digital audio data, and the output is a text string. The server analyzes the phonemes from the audio data and generates the corresponding text.

[0665] Step 3:

[0666] The server analyzes the generated text data and performs natural language processing to understand the user's intent. In this process, intent analysis is used to determine what the user is requesting in their utterance. The input is text data, and the output is intent information. Intent analysis uses algorithms to extract specific keywords and phrases from the text.

[0667] Step 4:

[0668] The server further detects the user's emotional state by performing sentiment recognition through text analysis. Using sentiment analysis algorithms, it outputs sentiment labels (e.g., positive, anxious, negative) from the input text data. Sentiment recognition utilizes machine learning models to identify language patterns and tones.

[0669] Step 5:

[0670] The server uses a generative AI model to generate an appropriate response based on intent information and emotion labels. A prompt (e.g., "Generate an appropriate response when the user feels anxious.") is input to the AI ​​model, causing it to output a text response. In this process, the AI ​​model performs a complex language generation task.

[0671] Step 6:

[0672] The generated text response is converted into speech data using a speech synthesis engine. The input is the response text, and the output is the synthesized speech data. Speech synthesis involves the process of converting text into sounds with natural pronunciation.

[0673] Step 7:

[0674] The server sends the converted audio data to the terminal, which then plays it back to the user. The user can then hear the system's response. The input is synthesized speech data, and the output is the audible sound. The terminal provides high-quality sound output.

[0675] Step 8:

[0676] The server performs a function that sends a notification to a pre-registered external organization when the emotion recognition system detects an emotion exceeding a certain threshold (e.g., severe anxiety). The input is the emotion recognition result, and the output is the notification message. This function enables security and rapid response.

[0677] (Application Example 2)

[0678] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0679] A challenge exists in that elderly users and those requiring emotional support often struggle to receive immediate and appropriate assistance when they experience feelings of loneliness or anxiety. Furthermore, these emotional states can persist, potentially leading to a decline in their quality of life. While conventional technologies have succeeded in analyzing user intentions, they have been insufficient in rapidly detecting emotional changes and providing the necessary support.

[0680] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0681] In this invention, the server includes an input device for receiving voice input from the user, a recognition device for converting the received voice into text information, an analysis device for analyzing the user's intent, an emotion analysis device for identifying the user's emotions, a communication device for communicating with a designated recipient based on the analyzed intent and emotions, and an output device for outputting the generated response as voice. This makes it possible to quickly detect the emotional state of the user when they feel anxious or lonely and provide appropriate support. Furthermore, if excessive stress or anxiety is detected, an alert can be quickly sent to a supporter using the notification function.

[0682] An "input device" is a device that has the function of receiving voice data from the user and incorporating it into the system.

[0683] A "recognition device" is a device that has the function of analyzing received audio data and converting it into text information.

[0684] An "analysis device" is a device that analyzes user intent from textual information to understand their requests and objectives.

[0685] An "emotion analysis device" is a device that identifies emotions from a user's voice data and determines their emotional state, such as positive, negative, or neutral.

[0686] A "communication device" is a device that has the function of transmitting information to a designated recipient based on analyzed intentions and emotions.

[0687] An "output device" is a device that has the function of providing the generated response to the user as audio.

[0688] The "notification function" is a feature that detects emotional states during emergencies and quickly sends alerts to registered supporters and organizations.

[0689] This invention is primarily intended to be integrated into smartwatches and digital assistant devices as a system to support elderly users and those requiring emotional support. When a user speaks into the device, the smartwatch's microphone captures the audio data. The captured audio data is sent to a server using the PyAudio library. The audio is then converted into text data using the Google Cloud Speech-to-Text API.

[0690] The server analyzes the converted character data using the NLTK library to determine the user's intent and emotions. Sentiment analysis identifies emotional states such as positive, negative, and neutral. Based on the analysis results, the server generates an appropriate response and provides voice feedback to the user using a speech synthesis engine. This response is designed to alleviate the user's feelings of loneliness and anxiety.

[0691] Furthermore, if the emotion analysis detects excessive stress or anxiety, Firebase Cloud Messaging is used to send an alert to the user's pre-registered supporters or family members. This notification feature is an important means of receiving prompt support.

[0692] For example, if a user says in an everyday situation, "I feel like I've been getting tired more easily lately," the system will offer advice such as, "You must be tired. Try having some tea and taking a short break." Furthermore, the system will contact family members as needed, increasing the user's sense of security.

[0693] A concrete example of a prompt might be the instruction, "If the user says, 'I've been feeling down lately,' send a message to soothe their emotions." Based on such prompts, the generative AI model creates a flexible response.

[0694] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0695] Step 1:

[0696] When a user makes a voice input to their smartwatch, the device's microphone acquires the audio data. The input data is a raw audio signal and cannot be analyzed directly, so it is converted into a digital format for the next step.

[0697] Step 2:

[0698] The terminal uses the PyAudio library to convert the acquired audio data into a digital format and send it to the server. Here, the analog audio signal is processed as data packets and sent to the server.

[0699] Step 3:

[0700] The server uses the Google Cloud Speech-to-Text API to convert received digital audio data into text data. This process involves analyzing the audio waveform and converting the information contained in the audio into an alphabetical string. This results in a string that can be read by humans.

[0701] Step 4:

[0702] The server analyzes text data using the NLTK library to determine the user's intent. It extracts keywords from the input text and performs data analysis to identify the information and actions the user is seeking.

[0703] Step 5:

[0704] The server uses the NLTK library to recognize emotions from text data. It analyzes the frequency and context of specific words and performs data processing to classify emotional states as positive, negative, or neutral. This emotional information is then used in the next step.

[0705] Step 6:

[0706] The server generates appropriate responses using a prompt message generated by an AI model based on the analyzed intent and emotions. These responses include content designed to soothe the user's emotions. The responses are output in text format.

[0707] Step 7:

[0708] The server converts the generated text-based response into audio data using a speech synthesis engine and plays it back to the user through the terminal. Speech synthesis is the phase in which information is provided to the user in an auditory-friendly format using algorithms that mimic a more realistic human voice.

[0709] Step 8:

[0710] If excessive stress or anxiety is detected, the server uses Firebase Cloud Messaging to send an alert to pre-registered caregivers or family members. This step involves retrieving contact information from the database and sending the alert message.

[0711] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0712] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0713] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0714] [Fourth Embodiment]

[0715] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0716] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0717] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0718] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0719] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0720] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0721] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0722] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0723] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0724] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0725] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0726] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0727] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0728] In order to implement this invention, it is necessary to construct a system that includes the following components.

[0729] The system begins when a user makes a phone call. First, the terminal receives the user's voice and sends it to the server. The server uses speech recognition to convert the voice data into text data, and this text data is analyzed by intent analysis. Intent analysis identifies the user's intent, and an appropriate action is determined accordingly.

[0730] As an example, consider a case where a user says, "Call my son." The server analyzes the intent of "Call my son" from the text data and uses a memory device to identify the son's contact information, which is registered in advance. Then, the communication device uses that phone number to actually make the call.

[0731] On the other hand, if a user encounters an emergency and says "Help," the server analyzes this as an emergency keyword. The notification system is activated, automatically sending notifications to the user's pre-configured emergency contacts and external emergency services. In this case, the user's current location information and voice message may be included in the notification.

[0732] Furthermore, the results of communications and notifications, or the system's response, are fed back to the user using speech synthesis. The speech synthesis converts the generated response into natural-sounding speech and sends it to the terminal, which then plays it back to the user.

[0733] In this system, each function is designed to be intuitively operable by the user, taking into consideration that elderly people can use it despite limitations in vision and memory. As a result, users can communicate with peace of mind, easily obtain necessary information, and respond quickly in emergencies. This embodiment effectively achieves the objective of the invention.

[0734] The following describes the processing flow.

[0735] Step 1:

[0736] The user makes a call to a landline phone number. The device receives this incoming call and records the audio as digital audio data.

[0737] Step 2:

[0738] The terminal sends the recorded audio data to the server. The server receives the transmitted audio data and uses a speech recognition engine to convert it into text data.

[0739] Step 3:

[0740] The server uses the converted text data to analyze the user's intent. Intent analysis tools are used to interpret the content of the call, such as "who to call" or "what to do."

[0741] Step 4:

[0742] Once the user's intent is analyzed, the server uses a communication method to make a phone call to the specified contact. For example, if the instruction is "Call my son," the server will search for the son's phone number from the pre-registered contact information and make the call.

[0743] Step 5:

[0744] The server generates processing results and responses, and uses a speech synthesis engine to convert text responses into speech data.

[0745] Step 6:

[0746] The server sends the generated audio data to the terminal. The terminal plays this audio data and provides feedback to the user. For example, it might play a response like, "I'm calling your son."

[0747] Step 7:

[0748] If a user says "emergency" or "help," the server will recognize this as an emergency. The server will then use notification methods to automatically send notifications to registered emergency contacts and emergency services. These notifications may include the user's location information and a voice message.

[0749] (Example 1)

[0750] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0751] There is a need to provide a system that allows elderly and visually impaired users to communicate intuitively without complex operations and to respond quickly in emergencies. Therefore, technology is required that facilitates a smooth process from voice input to decision-making and provides user-friendly feedback.

[0752] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0753] In this invention, the server includes voice receiving means for receiving voice input from a user, voice recognition means for converting the received voice into text data, and intent analysis means for analyzing the user's intent from the text data. This makes it possible for the user to easily communicate their intent to the system by voice and to quickly perform appropriate actions based on that intent.

[0754] "Voice receiving means" refers to a device or technology that has the function of detecting voice input from a user, converting its content into digital data, and transmitting it to subsequent processing.

[0755] "Speech recognition means" refers to a technology that analyzes speech data acquired from a speech receiving means and converts its content into text data.

[0756] "Intention analysis means" refers to a technology that analyzes text data obtained by speech recognition means and identifies the user's intended actions or requests based on that data.

[0757] "Communication means" refers to technology or devices that provide the function of establishing a connection with a specific communication partner based on an analyzed intent and sending and receiving information.

[0758] "Speech synthesis means" refers to a technology that converts responses generated by a system into a speech format and provides them to the user in an easily understandable way.

[0759] A "notification mechanism" is a technology that has the function of automatically notifying relevant external organizations or pre-configured recipients under specific conditions, particularly in emergency situations.

[0760] "Storage means" refers to a device or technology that has the function of storing a user's contact information and other necessary data, and making it quickly accessible and available as needed.

[0761] To implement this invention, it is necessary to construct an information processing system having a voice input interface and multiple technical components for processing voice. The user initiates an operation by voice, the terminal receives the voice, and the server processes it.

[0762] The server primarily functions as a means of receiving audio. When a user gives voice commands via a phone or smart device, the device captures this audio. This audio is converted into digital audio data and sent to the server. In this process, a typical mobile device or headset equipped with a microphone is often used as the hardware.

[0763] Next, the server operates as a "speech recognition tool," converting digital speech data into text data. This process can utilize, for example, a "speech recognition API." This conversion allows the user's speech to be obtained in text format, enabling subsequent intent analysis.

[0764] Next, the server analyzes the text data using "intent analysis tools" to understand the user's intent. This process utilizes "generative AI models" and "language understanding APIs" that perform natural language processing. This allows the server to identify specific action requests and inquiries from the user's utterances.

[0765] As a practical example, if a user says, "Call my son," the server recognizes this instruction from the text data, retrieves the appropriate phone number by referring to pre-registered contacts, and then activates the server's "communication means" to place a call to the identified contact. This process is achieved through interfaces with "communication APIs" and "telephone service providers."

[0766] If a user utters an emergency phrase such as "Help," the server utilizes its "notification system" to automatically send location information and a voice message to designated recipients in an emergency. Notifications are also sent to pre-configured emergency contacts.

[0767] Finally, the server uses a "speech synthesis method" to convert the system response into a natural-sounding voice format. This allows the user to receive feedback from the system in voice through their terminal. It is expected that a "speech synthesis API" or similar will be used for this speech synthesis.

[0768] An example of a prompt might be, "Find my son's number in my phone book and make a call." This system allows users to intuitively operate the device with their voice, enabling smooth information retrieval and emergency response.

[0769] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0770] Step 1:

[0771] The user initiates the operation using voice commands.

[0772] Input: User voice command (e.g., "Call my son")

[0773] Output: Audio data

[0774] Specific operation: The user speaks a specified instruction into the microphone of their smartphone or other voice-enabled device. This voice is captured by the device's microphone and recorded as digital audio data.

[0775] Step 2:

[0776] The device sends audio to the server.

[0777] Input: Digital audio data

[0778] Output: Transfer of audio data

[0779] Specific operation: The terminal sends the captured digital audio data to the server via the network. This enables data processing on the server side.

[0780] Step 3:

[0781] The server converts the audio to text.

[0782] Input: Digital audio data

[0783] Output: Character data

[0784] Specific operation: The server uses speech recognition technology (e.g., "Speech Recognition API") to convert the received audio data into text data. This conversion process involves analyzing sound waves to convert spoken words into text format.

[0785] Step 4:

[0786] The server analyzes the intent.

[0787] Input: Text data

[0788] Output: User intent (e.g., "Make a phone call")

[0789] Specific operation: The server uses generative AI models and language understanding APIs to analyze text data and identify the actions and information the user is seeking. This allows for a clear understanding of what the user wants.

[0790] Step 5:

[0791] The server initiates communication.

[0792] Input: User intent and contact information

[0793] Output: Communication execution

[0794] Specific operation: Based on the user's intent, the server refers to pre-stored contact information and performs the necessary communication. This process involves using communication methods to perform actions such as making phone calls or sending messages.

[0795] Step 6:

[0796] The server will send notifications (if necessary).

[0797] Input: Emergency phrases or conditions

[0798] Output: Notification to external organizations and contacts

[0799] Specific operation: When a user utters an emergency phrase such as "Help," the server activates an emergency notification system and sends a notification containing location information and the situation to pre-configured recipients. This enables a rapid response.

[0800] Step 7:

[0801] The server generates an audio response.

[0802] Input: System response information

[0803] Output: Audio data

[0804] Specific operation: The server uses speech synthesis technology to convert the generated response into natural-sounding speech. This speech data is then used as feedback to the user.

[0805] Step 8:

[0806] The terminal plays the response.

[0807] Input: Audio data

[0808] Output: Voice feedback to the user

[0809] Specific operation: The terminal receives audio data sent from the server and plays it back to the user through the speaker. This allows the user to confirm the system's response.

[0810] (Application Example 1)

[0811] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0812] There is a problem in that users who are unfamiliar with technology, such as the elderly, have difficulty using voice control to quickly and easily perform important communication and respond to emergencies. For example, they may forget when to take their medication or be unable to immediately access the appropriate contacts in an emergency. Therefore, there is a need to provide technology that supports users with tasks necessary for daily life intuitively and without burden, and that allows them to respond with confidence in emergencies.

[0813] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0814] In this invention, the server includes voice acquisition means for acquiring voice information from the user, voice conversion means for converting the acquired voice information into text data, and purpose analysis means for analyzing the user's purpose from the text data. This allows the user to operate intuitively through voice, prompt specific actions at pre-set times, and provide rapid notifications in emergencies.

[0815] A "voice acquisition means" is a device that has the function of receiving voice information from a user and transferring it to a server for analysis.

[0816] A "speech conversion means" is a device that converts acquired speech information into text data.

[0817] A "purpose analysis tool" is a device that analyzes the purpose of a user's utterance from text data and identifies their intent.

[0818] An "information transmission means" is a device that transmits appropriate information to selected contacts based on the results of an analysis conducted according to the user's objectives.

[0819] A "speech generation means" is a device that has the function of converting the generated response into speech and presenting it to the user audibly.

[0820] A "command generation means" is a device that has the function of generating commands to prompt the user to take a specific action based on the results of objective analysis.

[0821] An "information storage device" is a device that stores user contact information and related data and has the function of referencing it as needed.

[0822] To implement this invention, it is necessary to construct a series of systems including voice acquisition means, voice conversion means, purpose analysis means, information transmission means, voice generation means, command generation means, and information storage means. The server first receives voice data sent from the terminal. Then, it converts this voice data into text data using the voice conversion means. The Google Cloud Speech-to-Text API can be used for voice conversion. Next, the purpose analysis means analyzes the text data using a BERT model to identify the user's intent. Once the intent is identified, the information transmission means sends the information to the user's designated contact via the Twilio API. In emergencies, the command generation means responds to the user with voice generated using Amazon Polly and instructs them on the necessary actions.

[0823] This system is designed to be intuitive and easy to use, especially for users unfamiliar with technology, such as the elderly. It provides not only daily support but also rapid response in emergencies. For example, if a user says, "Tell me when to take my medication," the system will refer to a pre-set schedule and notify them via voice to take their medication at the appropriate time. Similarly, if a user says, "Help me," an emergency contact will be automatically notified.

[0824] A concrete example of a prompt might be, "How to respond when the user says, 'Tell me when to take my medication'." Using this prompt, it becomes possible to generate more natural responses using a generative AI model.

[0825] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0826] Step 1:

[0827] The terminal receives voice input from the user. This voice input includes spoken information from the user. The terminal then prepares to send this voice data to the server.

[0828] Input: User's voice information

[0829] Output: Audio data ready to be sent to the server.

[0830] Step 2:

[0831] The server converts the audio data sent from the terminal into text data using a speech-to-text conversion method. Specifically, it uses the Google Cloud Speech-to-Text API to convert speech to text.

[0832] Input: Audio data

[0833] Output: Converted character data

[0834] Step 3:

[0835] The server analyzes the generated text data using a target analysis tool. This analysis utilizes the BERT model to perform data processing that identifies the user's intent from their utterance.

[0836] Input: Text data

[0837] Output: Analyzed user intent

[0838] Step 4:

[0839] The server decides its actions based on the results of the purpose analysis. For example, if the identified intention is "tell me when to take my medicine," it will access the information storage system to obtain the necessary schedule information.

[0840] Input: Analyzed user intent

[0841] Output: Decision made or information

[0842] Step 5:

[0843] Based on the decided action, the server provides necessary information to the user via an information transmission means. This information is output as voice using a voice generation means and Amazon Polly.

[0844] Input: Decision made or information

[0845] Output: Voice feedback to the user

[0846] Step 6:

[0847] The user takes necessary actions based on the audio feedback received from the server. For example, they might take medication according to the time notified.

[0848] Input: Voice feedback

[0849] Output: User behavior

[0850] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0851] This invention provides a dialogue system that incorporates an emotion engine in addition to the conventional process of receiving user voice input and converting voice data into text. The system is implemented according to the following procedure.

[0852] First, the user makes a call to a fixed phone number to input their voice. The terminal records the user's voice in digital format and sends it to the server. The server uses speech recognition to convert the received voice data into text data.

[0853] Next, the server analyzes the text data using intent analysis tools to understand the user's requests and objectives. This includes things like "who to call" and "how to handle emergencies." Furthermore, it analyzes the user's emotions using newly incorporated emotion recognition tools. This analysis allows the server to determine whether the emotion is positive, negative, or neutral.

[0854] For example, if a user says, "I feel very anxious today," the server will determine through intent analysis that a conversation is necessary and detect anxiety through sentiment analysis. Based on these results, the server will generate words of comfort and encouragement.

[0855] The generated response is converted into audio data using a speech synthesis engine and played back to the user through the device. This provides the user with a sense of security.

[0856] Furthermore, if the emotion recognition system identifies the user's emotions and detects excessively strong anxiety or stress, the server uses a notification system to send an alert to pre-registered family members or caregivers. This feature enables a quick response and support.

[0857] The overall processing of this system aims to provide more personalized care to elderly users and those who require emotional support. This can improve users' quality of life and reduce feelings of loneliness.

[0858] The following describes the processing flow.

[0859] Step 1:

[0860] The user makes a call to a landline phone number. The device receives this incoming call and records the user's voice as digital audio data.

[0861] Step 2:

[0862] The terminal sends the recorded audio data to the server. The server converts the received audio data into text data using speech recognition technology.

[0863] Step 3:

[0864] The server analyzes text data using intent analysis tools. It understands the user's requests and objectives and determines what the appropriate action is. In this process, it identifies voice content such as "who to call" or "to talk about anxieties."

[0865] Step 4:

[0866] The server uses emotion recognition technology to analyze the user's voice to determine their emotions. This analysis determines whether the user is anxious, happy, or in any other emotional state.

[0867] Step 5:

[0868] Based on the analyzed emotional state, the server determines the response. For example, if anxiety is detected, it generates words of comfort or encouragement and takes corresponding action.

[0869] Step 6:

[0870] The server converts the generated response into audio data using a speech synthesis system. The terminal receives this audio data and plays it back to the user to provide feedback. The user receives an appropriate response based on the situation.

[0871] Step 7:

[0872] If the emotion recognition system detects strong anxiety or stress, the server sends an emergency notification to family members or caregivers via a notification system. The notification will include information about the user's emotional state and the support needed.

[0873] (Example 2)

[0874] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0875] Conventional interactive information processing systems have struggled to accurately understand user intent and generate appropriate responses, particularly failing to provide responses that take user emotions into consideration. Furthermore, the lack of mechanisms for quickly notifying external organizations when users experience anxiety or stress makes immediate response difficult, especially in urgent situations.

[0876] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0877] In this invention, the server includes a device means for receiving voice input, a conversion means for converting voice to text, an analysis means for analyzing intent from text, an emotion recognition means for determining the user's emotions, a response generation means for generating a response using a generative AI model, and a voice output means for outputting the generated response as voice. This enables accurate understanding of the user's intent, the provision of natural responses that are in line with their emotions, and the ability to quickly notify external organizations if anxiety or stress is detected, thereby realizing appropriate support for the user.

[0878] A "user" refers to an individual or group that uses the system, and is the entity that provides or obtains information through voice input.

[0879] "Voice input" refers to the format in which the words spoken by the user are incorporated into an information processing system.

[0880] "Device means" refers to equipment or devices for inputting and outputting sound and information, and primarily functions as an interface with the user.

[0881] "Conversion means" refers to technologies and devices that convert audio data into text data, and involves the process of changing speech into text through speech recognition.

[0882] "Analysis methods" refer to technologies and algorithms used to understand user intent and content from text data, and play a role in extracting user requests and objectives.

[0883] "Emotion recognition means" refers to technology that determines the user's emotional state from input voice or text and uses that determination to provide an appropriate response.

[0884] A "generative AI model" refers to a generative model that utilizes artificial intelligence technology to automatically generate natural-sounding text based on user input data.

[0885] "Response generation means" refers to the process or technology of generating a message to respond to the user based on analysis results and emotion recognition data.

[0886] "Audio output means" refers to the technology or device that converts text generated by the system into speech and outputs it in a format that can be heard by the user.

[0887] An "information processing system" refers to a computer system that integrates the above means to enable interaction with the user, and is responsible for processing and outputting input data.

[0888] This invention is an interactive information processing system that utilizes user voice input. The user dials a specific telephone number using a communication terminal and inputs voice. The terminal records the user's voice in digital voice format and sends the data to a server. The server converts the received voice data into text data using speech recognition software. A general-purpose speech recognition engine is suitable for use as the software for this purpose.

[0889] Next, the server uses a generative AI model to analyze the text data, understand the user's intent, and perform sentiment recognition. This involves applying software that utilizes natural language processing (NLP) technology. This analysis allows the system to identify what requests or emotions are underlying the user's statements.

[0890] For example, if a user says, "I feel very anxious today," this text is converted by speech recognition, and the emotion of anxiety is extracted through sentiment analysis. Based on this, the server inputs a prompt into the generation AI model to generate words of comfort. An example of a prompt might be an instruction such as, "Generate an appropriate response for when the user feels anxious."

[0891] The generated response is converted back into voice data using a speech synthesis engine and returned to the user via the terminal. This allows the user to naturally receive feedback from the system. Furthermore, emotion recognition means that if the user shows particularly strong stress or anxiety, the server automatically sends a notification to a pre-registered external organization, enabling a rapid response. This entire process provides the user with personalized support and enhances their sense of security.

[0892] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0893] Step 1:

[0894] The user makes a call to a specific phone number and enters a message by voice. This voice is recorded digitally by the device. The input is an audio signal, and the output is digital audio data. The device performs the necessary encoding to capture the audio signal in high quality.

[0895] Step 2:

[0896] The terminal sends recorded digital audio data to the server. The server receives this data and converts it into text data using speech recognition software. The input is digital audio data, and the output is a text string. The server analyzes the phonemes from the audio data and generates the corresponding text.

[0897] Step 3:

[0898] The server analyzes the generated text data and performs natural language processing to understand the user's intent. In this process, intent analysis is used to determine what the user is requesting in their utterance. The input is text data, and the output is intent information. Intent analysis uses algorithms to extract specific keywords and phrases from the text.

[0899] Step 4:

[0900] The server further detects the user's emotional state by performing sentiment recognition through text analysis. Using sentiment analysis algorithms, it outputs sentiment labels (e.g., positive, anxious, negative) from the input text data. Sentiment recognition utilizes machine learning models to identify language patterns and tones.

[0901] Step 5:

[0902] The server uses a generative AI model to generate an appropriate response based on intent information and emotion labels. A prompt (e.g., "Generate an appropriate response when the user feels anxious.") is input to the AI ​​model, causing it to output a text response. In this process, the AI ​​model performs a complex language generation task.

[0903] Step 6:

[0904] The generated text response is converted into speech data using a speech synthesis engine. The input is the response text, and the output is the synthesized speech data. Speech synthesis involves the process of converting text into sounds with natural pronunciation.

[0905] Step 7:

[0906] The server sends the converted audio data to the terminal, which then plays it back to the user. The user can then hear the system's response. The input is synthesized speech data, and the output is the audible sound. The terminal provides high-quality sound output.

[0907] Step 8:

[0908] The server performs a function that sends a notification to a pre-registered external organization when the emotion recognition system detects an emotion exceeding a certain threshold (e.g., severe anxiety). The input is the emotion recognition result, and the output is the notification message. This function enables security and rapid response.

[0909] (Application Example 2)

[0910] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0911] A challenge exists in that elderly users and those requiring emotional support often struggle to receive immediate and appropriate assistance when they experience feelings of loneliness or anxiety. Furthermore, these emotional states can persist, potentially leading to a decline in their quality of life. While conventional technologies have succeeded in analyzing user intentions, they have been insufficient in rapidly detecting emotional changes and providing the necessary support.

[0912] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0913] In this invention, the server includes an input device for receiving voice input from the user, a recognition device for converting the received voice into text information, an analysis device for analyzing the user's intent, an emotion analysis device for identifying the user's emotions, a communication device for communicating with a designated recipient based on the analyzed intent and emotions, and an output device for outputting the generated response as voice. This makes it possible to quickly detect the emotional state of the user when they feel anxious or lonely and provide appropriate support. Furthermore, if excessive stress or anxiety is detected, an alert can be quickly sent to a supporter using the notification function.

[0914] An "input device" is a device that has the function of receiving voice data from the user and incorporating it into the system.

[0915] A "recognition device" is a device that has the function of analyzing received audio data and converting it into text information.

[0916] An "analysis device" is a device that analyzes user intent from textual information to understand their requests and objectives.

[0917] An "emotion analysis device" is a device that identifies emotions from a user's voice data and determines their emotional state, such as positive, negative, or neutral.

[0918] A "communication device" is a device that has the function of transmitting information to a designated recipient based on analyzed intentions and emotions.

[0919] An "output device" is a device that has the function of providing the generated response to the user as audio.

[0920] The "notification function" is a feature that detects emotional states during emergencies and quickly sends alerts to registered supporters and organizations.

[0921] This invention is primarily intended to be integrated into smartwatches and digital assistant devices as a system to support elderly users and those requiring emotional support. When a user speaks into the device, the smartwatch's microphone captures the audio data. The captured audio data is sent to a server using the PyAudio library. The audio is then converted into text data using the Google Cloud Speech-to-Text API.

[0922] The server analyzes the converted character data using the NLTK library to determine the user's intent and emotions. Sentiment analysis identifies emotional states such as positive, negative, and neutral. Based on the analysis results, the server generates an appropriate response and provides voice feedback to the user using a speech synthesis engine. This response is designed to alleviate the user's feelings of loneliness and anxiety.

[0923] Furthermore, if the emotion analysis detects excessive stress or anxiety, Firebase Cloud Messaging is used to send an alert to the user's pre-registered supporters or family members. This notification feature is an important means of receiving prompt support.

[0924] For example, if a user says in an everyday situation, "I feel like I've been getting tired more easily lately," the system will offer advice such as, "You must be tired. Try having some tea and taking a short break." Furthermore, the system will contact family members as needed, increasing the user's sense of security.

[0925] A concrete example of a prompt might be the instruction, "If the user says, 'I've been feeling down lately,' send a message to soothe their emotions." Based on such prompts, the generative AI model creates a flexible response.

[0926] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0927] Step 1:

[0928] When a user makes a voice input to their smartwatch, the device's microphone acquires the audio data. The input data is a raw audio signal and cannot be analyzed directly, so it is converted into a digital format for the next step.

[0929] Step 2:

[0930] The terminal uses the PyAudio library to convert the acquired audio data into a digital format and send it to the server. Here, the analog audio signal is processed as data packets and sent to the server.

[0931] Step 3:

[0932] The server uses the Google Cloud Speech-to-Text API to convert received digital audio data into text data. This process involves analyzing the audio waveform and converting the information contained in the audio into an alphabetical string. This results in a string that can be read by humans.

[0933] Step 4:

[0934] The server analyzes text data using the NLTK library to determine the user's intent. It extracts keywords from the input text and performs data analysis to identify the information and actions the user is seeking.

[0935] Step 5:

[0936] The server uses the NLTK library to recognize emotions from text data. It analyzes the frequency and context of specific words and performs data processing to classify emotional states as positive, negative, or neutral. This emotional information is then used in the next step.

[0937] Step 6:

[0938] The server generates appropriate responses using a prompt message generated by an AI model based on the analyzed intent and emotions. These responses include content designed to soothe the user's emotions. The responses are output in text format.

[0939] Step 7:

[0940] The server converts the generated text-based response into audio data using a speech synthesis engine and plays it back to the user through the terminal. Speech synthesis is the phase in which information is provided to the user in an auditory-friendly format using algorithms that mimic a more realistic human voice.

[0941] Step 8:

[0942] If excessive stress or anxiety is detected, the server uses Firebase Cloud Messaging to send an alert to pre-registered caregivers or family members. This step involves retrieving contact information from the database and sending the alert message.

[0943] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0944] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0945] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0946] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0947] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0948] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0949] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0950] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0951] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0952] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0953] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0954] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0955] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0956] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0957] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0958] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0959] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0960] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0961] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0962] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0963] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0964] The following is further disclosed regarding the embodiments described above.

[0965] (Claim 1)

[0966] A voice receiving means for receiving voice input from the user,

[0967] A speech recognition means that converts received audio into text data,

[0968] An intent analysis means for analyzing the user's intent from the aforementioned text data,

[0969] A communication means that communicates with a designated contact based on the analyzed intent,

[0970] A speech synthesis means that outputs the generated response as audio,

[0971] A system that includes this.

[0972] (Claim 2)

[0973] The system according to claim 1, wherein the intent analysis means includes a notification means that identifies a keyword indicating an emergency and automatically sends a notification to a relevant external organization.

[0974] (Claim 3)

[0975] The system according to claim 1, wherein the communication means includes a storage means for storing user contact information and has a function for accessing a predetermined contact and making a phone call.

[0976] "Example 1"

[0977] (Claim 1)

[0978] A voice receiving means for receiving voice input from the user,

[0979] A speech recognition means that converts received audio into text data,

[0980] An intent analysis means for analyzing the user's intent from the aforementioned text data,

[0981] A communication means that communicates with a designated communication partner based on the analyzed intent,

[0982] A speech synthesis means that outputs the generated response as audio,

[0983] A system that includes this.

[0984] (Claim 2)

[0985] The system according to claim 1, wherein the intent analysis means includes a notification means that identifies a word indicating an emergency and automatically sends a notification to a relevant external organization.

[0986] (Claim 3)

[0987] The system according to claim 1, wherein the communication means includes a storage means for storing user contact information and has a function for accessing a predetermined communication partner and making a phone call.

[0988] "Application Example 1"

[0989] (Claim 1)

[0990] A voice acquisition means for acquiring voice information from users,

[0991] A speech conversion means that converts acquired speech information into text data,

[0992] A means for analyzing the user's purpose from the aforementioned text data,

[0993] Information transmission means for sending information to contacts selected based on the analyzed purpose,

[0994] A speech generation means that converts the generated response into speech and outputs it,

[0995] The aforementioned objective analysis means includes a command generation means that issues a command to prompt a specific action at a specified time,

[0996] A system that includes this.

[0997] (Claim 2)

[0998] The system according to claim 1, wherein the command generation means is used to generate a command prompting the user to take action at a predetermined time.

[0999] (Claim 3)

[1000] The system according to claim 1, wherein the information transmission means includes an information storage means for storing user information and has a function to perform a predetermined action based on appropriate contact information.

[1001] "Example 2 of combining an emotion engine"

[1002] (Claim 1)

[1003] A device means for receiving voice input from a user,

[1004] A conversion means for converting received audio into text data,

[1005] An analysis means for analyzing the user's intent from the aforementioned text data,

[1006] A means of recognizing an emotion to determine the user's emotions,

[1007] A response generation means that generates a response using a generative AI model,

[1008] A voice output means that outputs the generated response as audio,

[1009] An information processing system that includes this.

[1010] (Claim 2)

[1011] The information processing system according to claim 1, wherein the emotion recognition means has a function to identify the user's emotional state and, if necessary, send a notification to an external party.

[1012] (Claim 3)

[1013] The information processing system according to claim 1, wherein the analysis means identifies keywords indicating anxiety or stress and sends an alert to a pre-registered external organization.

[1014] "Application example 2 when combining with an emotional engine"

[1015] (Claim 1)

[1016] An input device that receives voice input from the user,

[1017] A recognition device that converts received audio into text information,

[1018] An analysis device that analyzes the user's intent from the aforementioned textual information,

[1019] An emotion analysis device that identifies the user's emotions,

[1020] A communication device that communicates with a designated recipient based on analyzed intentions and emotions,

[1021] An output device that outputs the generated response as audio,

[1022] A system that includes this.

[1023] (Claim 2)

[1024] The system according to claim 1, wherein the emotion analysis device has a notification function that identifies the user's emotional state and automatically sends a notification to a relevant external organization or registered contact when an emergency emotion is detected.

[1025] (Claim 3)

[1026] The system according to claim 1, wherein the communication device includes a storage device for storing user communication information and has a function to access a predetermined party and perform voice communication. [Explanation of symbols]

[1027] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A voice acquisition means for acquiring voice information from users, A speech conversion means that converts acquired speech information into text data, A means for analyzing the user's purpose from the aforementioned text data, Information transmission means for sending information to contacts selected based on the analyzed purpose, A speech generation means that converts the generated response into speech and outputs it, The aforementioned objective analysis means includes a command generation means that issues a command to prompt a specific action at a specified time, A system that includes this.

2. The system according to claim 1, wherein the command generation means is used to generate a command prompting the user to take action at a predetermined time.

3. The system according to claim 1, wherein the information transmission means includes an information storage means for storing user information and has a function to perform a predetermined action based on appropriate contact information.

Images