System

Abstract

A system is provided.SOLUTION: A system including means for collecting voice data of a user, means for converting the collected voice data into text data, means for transmitting the converted text data to a server, means for analyzing the text data in the server and extracting a feature of the user, means for learning a generative artificial intelligence model based on the extracted feature, means for transmitting the generated model to a terminal, means for installing the generative artificial intelligence model in the terminal, means for generating a response from the generative artificial intelligence model based on an input of the user, and means for outputting the generated response as a voice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Elderly people living in nursing homes are often restricted from visiting family and friends, especially during infectious disease outbreaks. This can lead to feelings of loneliness and mental stress, potentially adversely affecting their physical and mental health. Conventional telephone and video calls often lack the sense of realism of face-to-face conversations, making intimate conversations difficult. The present invention aims to address these issues and enable elderly people to experience simulated conversations with family and friends, thereby reducing mental stress and improving their quality of life. [Means for solving the problem]

[0005] The present invention provides the following means. First, a means for collecting user voice data is provided. Then, a means for converting this voice data into text data is provided, and a means for transmitting the converted text data to a server is used. In the server, a means for analyzing the text data and extracting user features is provided. Then, a means for training a generative artificial intelligence model based on the extracted features is provided. Then, a means for transmitting the generated model to a terminal is used. In the terminal, a means for installing the generative artificial intelligence model is provided, and a means for generating a response from the generative artificial intelligence model based on user input is provided. Finally, a means for outputting the generated response as voice is provided, thereby building a system that provides a user with an experience that feels like they are actually having a conversation with family or friends.

[0006] "User" refers to a person who uses the system, and in this case, the target is mainly elderly people.

[0007] "Voice data" refers to data that is a digital recording of a user's speech or voice.

[0008] "Text data" refers to digital data that has been converted from voice data into text using voice recognition technology.

[0009] "Server" refers to a computer system on a network that centrally manages and processes data.

[0010] "Characteristics" refers to individual characteristics such as a user's speaking style or thought patterns.

[0011] A "generative AI model" is an AI model that is generated based on the characteristics of a user and is capable of imitating specific speaking styles and thought patterns.

[0012] "Terminal" refers to a device (such as a smartphone or tablet) that is directly operated by a user.

[0013] "Response" refers to the reply that a generative artificial intelligence model outputs in response to a user's input.

[0014] "Simulated" refers to something that appears to be happening or that is experienced as if it were actually happening. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] System Overview

[0037] This system allows elderly people who are unable to meet with family and friends due to an infectious disease pandemic to enjoy simulated conversations. The system collects user voice data, analyzes it to generate a generative artificial intelligence model, and installs the model on the elderly person's device to realize simulated conversations.

[0038] Program Overview

[0039] The program performs the following specific processing.

[0040] User data collection phase

[0041] 1. The user launches the app on their smartphone and speaks into the microphone.

[0042] For example: "Hi, Mom. How was your day?"

[0043] 2. The device collects the user's voice and saves it as voice data.

[0044] 3. The device uses a voice recognition system to convert the collected voice data into text data.

[0045] 4. The terminal sends the converted text data to the server.

[0046] Data analysis phase

[0047] 1. The server receives the text data sent from the terminal.

[0048] 2. The server analyzes the text data using a natural language processing (NLP) engine.

[0049] Specifically, it extracts sentence structure, speaking style, frequently used phrases, emotional expressions, etc.

[0050] 3. The server extracts and saves the user's characteristics from the analysis results.

[0051] Model generation phase

[0052] 1. The server trains a generative artificial intelligence model based on the extracted features.

[0053] This generative artificial intelligence model mimics specific speech and thought patterns.

[0054] 2. The server sends the trained and generated model to the device.

[0055] Model installation phase

[0056] 1. The terminal receives the generative artificial intelligence model sent from the server.

[0057] 2. The device installs the received model into the app.

[0058] 3. The terminal checks the operation of the installed model.

[0059] Conversation Simulation Phase

[0060] 1. The user (elderly person) launches the app and speaks into the microphone.

[0061] For example: "The weather is lovely today, did you go for a walk?"

[0062] 2. The device collects the user's voice and converts it into text data using a voice recognition system.

[0063] 3. The terminal inputs the converted text data into a generative artificial intelligence model.

[0064] 4. The device receives the response generated by the generative artificial intelligence model and outputs this response as speech.

[0065] For example: "Yes, Mom. The weather was nice today, so I went for a little walk."

[0066] Specific examples

[0067] For example, consider the case where a son uses this system for his mother who lives far away.

[0068] The user (son) talks to the app

[0069] The son says, "Hello, Mom. How was your day?" The app collects this speech and converts it into text using a speech recognition system. This text data is then sent to the server.

[0070] Server analysis and model generation

[0071] The server analyzes the text data and extracts the son's speaking style and thought patterns. Based on this, a generative AI model is trained to generate a model that reproduces the son's unique conversational style. This model is then sent to the device.

[0072] Model installation and conversation simulation

[0073] When a mother launches the app and says, "The weather is nice today. Did you go for a walk?", the device analyzes the speech and generates an appropriate response based on a generative artificial intelligence model. This response is output as a voice message saying, "Yes, Mom. The weather is nice today, so I went for a short walk."

[0074] This system allows elderly people to enjoy simulated conversations with family and friends, reducing feelings of loneliness and mental stress.

[0075] The processing flow will be explained below.

[0076] Step 1:

[0077] The user launches the app on their smartphone, and the app interface appears, switching to audio recording mode.

[0078] Step 2:

[0079] The user speaks into the microphone, for example, "Hi, Mom. How was your day?"

[0080] Step 3:

[0081] The device collects the user's voice and stores it as digital audio data.

[0082] Step 4:

[0083] The device sends the collected voice data to a voice recognition system, which converts it into text data. The voice recognition system analyzes the voice and generates the text, "Hello, Mom. How was your day?"

[0084] Step 5:

[0085] The terminal transmits the converted text data to the server, where the data is transferred using a secure communication protocol.

[0086] Step 6:

[0087] The server receives the text data sent from the device and stores it in a data store for analysis.

[0088] Step 7:

[0089] The server runs a natural language processing (NLP) engine to analyze the text data it receives, analyzing specific phrases, grammatical structures, emotional expressions, etc., and extracting the user's speaking style and thought patterns.

[0090] Step 8:

[0091] The server trains a generative artificial intelligence model based on the extracted features. The model uses machine learning algorithms to reflect the user's characteristics.

[0092] Step 9:

[0093] The server generates a trained generative AI model and sends it to the device. The generated model includes functions that can handle a variety of conversation scenarios.

[0094] Step 10:

[0095] The device receives the generative AI model sent from the server, and then installs the model into the app.

[0096] Step 11:

[0097] The operation of the generative AI model installed on the device is checked using a simple conversation scenario to verify whether the model generates accurate responses.

[0098] Step 12:

[0099] The user (elderly person) restarts the app and starts a conversation with the generative AI model, for example, saying, "The weather is nice today. Have you gone for a walk?"

[0100] Step 13:

[0101] The device collects the user's (elderly person's) voice and converts it into text data using a voice recognition system.

[0102] Step 14:

[0103] The device inputs the converted text data into a generative artificial intelligence model, which generates an appropriate response based on the input.

[0104] Step 15:

[0105] The device then plays back the generated response as speech, for example, "Yes, Mom. The weather was nice today, so I went for a short walk."

[0106] Through this series of steps, the system enables elderly people to enjoy simulated conversations with family and friends.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] Elderly people face the problem of feeling lonely and experiencing increased mental stress due to fewer opportunities to talk with family and friends due to infectious disease outbreaks and other factors. To solve this problem, a system that allows elderly people to enjoy simulated conversations is needed. Furthermore, an advanced, non-contact conversation system that can reproduce the user's speaking style and thought patterns is also needed.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, means for transmitting the converted text data to the server, means for analyzing the text data and extracting user characteristics in the server, means for training a generative artificial intelligence model based on the extracted characteristics, means for transmitting the generated model to a terminal, means for installing the generative artificial intelligence model in the terminal, means for generating a response from the generative artificial intelligence model based on the user's voice input, means for outputting the generated response as voice, means for learning and saving the user's characteristic speaking style and thought patterns using the generative artificial intelligence model, and means for confirming the operation of the generative artificial intelligence model in the terminal. This allows elderly people to enjoy simulated conversations with family and friends, thereby reducing feelings of loneliness and mental burden.

[0112] "Voice data" refers to data that is a digital recording of a user's voice.

[0113] "Text data" refers to voice data converted into a string of characters using a voice recognition system.

[0114] A "server" is a computer system that processes and stores data and communicates with other computers over a network.

[0115] A "terminal" is a device operated by a user, such as a smartphone or tablet.

[0116] "Converting means" means the system or algorithm used to convert audio data into text data.

[0117] "Means of analysis" refers to software or algorithms that analyze text data and extract specific information, such as natural language processing engines.

[0118] "Means of feature extraction" refers to the technology that performs the process of recognizing and extracting certain patterns or features from data.

[0119] A "generative artificial intelligence model" is a machine learning model trained to mimic a user's characteristic speaking and thinking patterns.

[0120] "Means of learning" refers to techniques for using data to train generative artificial intelligence models.

[0121] "Means for installing" refers to the process of incorporating the generated artificial intelligence model into the software of the terminal.

[0122] A "response generator" is a system or algorithm that generates an appropriate reply based on user input.

[0123] The "means for outputting as voice" is a system that converts the generated text response into voice and outputs it.

[0124] "Means for operational verification" is a process for testing whether the installed artificial intelligence model is functioning correctly.

[0125] "Simulated conversation" refers to a simulated dialogue that is conducted in a format similar to a real conversation.

[0126] This invention is a system that collects user voice data, analyzes it to generate a generative artificial intelligence model that imitates specific speaking styles and thought patterns, and installs the model on a terminal to realize simulated conversations.

[0127] This system mainly consists of devices such as smartphones and tablets, a server responsible for data analysis, a voice recognition system, a natural language processing engine, and a generative artificial intelligence model.

[0128] User data collection

[0129] First, the user opens the app on their smartphone and speaks into the microphone. For example, you can use the following prompt:

[0130] "Hi, Mom. How was your day?"

[0131] The device collects this voice and stores it as voice data in local storage. The collected voice data is converted into text data using a voice recognition system such as Google® Cloud Speech-to-Text or IBM Watson® Speech to Text. The converted text data is then sent to the server as an HTTP POST request.

[0132] Data analysis and model generation on the server

[0133] The server receives the text data sent from the device and analyzes it using Python and a natural language processing engine such as NLTK or spaCy. The analysis includes sentence structure analysis, speech feature extraction, frequently used phrase analysis, and emotional expression analysis. The server extracts user features from the analysis results and uses this feature data to train a generative artificial intelligence model (e.g., GPT-3 (registered trademark)). The generated model is then sent to the device.

[0134] Installing the model on the device and checking its operation

[0135] The device receives the generative AI model sent from the server and installs it in the app. The operation of the installed model is confirmed by generating test inputs on the device and verifying that the model's responses are accurate.

[0136] Conversation simulation for elderly people

[0137] Elderly people can launch the app and say things like:

[0138] "The weather is nice today, did you go for a walk?"

[0139] The device collects this speech and converts it into text data using a speech recognition system. The converted text data is input into a generative artificial intelligence model, and the response generated by the model is output as speech. For example, the following response may be returned:

[0140] "Yes, Mom. The weather was nice today, so I went for a little walk."

[0141] Using this system, elderly people can enjoy simulated conversations with family and friends, reducing feelings of loneliness and mental stress.

[0142] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0143] Step 1:

[0144] Data Collection Phase

[0145] 1. Start voice input

[0146] The user launches the app on their smartphone and speaks into the microphone.

[0147] Input: What the user says (e.g., "Hi, Mom. How was your day?")

[0148] Specific actions: Tap the voice input button in the app to start recording.

[0149] Output: Recorded audio data

[0150] 2. Saving audio data

[0151] The terminal collects the user's voice and stores it as voice data in local storage.

[0152] Input: Recorded audio data

[0153] Specific operation: When recording is finished, the audio data will be saved in WAV format to the device's local storage.

[0154] Output: Saved audio file

[0155] Step 2:

[0156] Data Transformation Phase

[0157] 3. Convert voice to text data

[0158] The terminal converts the collected voice data into text data using a voice recognition system.

[0159] Input: Saved audio file

[0160] What happens: The audio file is sent to a speech recognition API, such as Google Cloud Speech-to-Text, and converted to text.

[0161] Output: Converted text data

[0162] Step 3:

[0163] Data transmission phase

[0164] 4. Sending text data to the server

[0165] The terminal transmits the converted text data to the server.

[0166] Input: Text data

[0167] Specific behavior: Sends text data to the server as an HTTP POST request.

[0168] Output: Text data sent to the server

[0169] Step 4:

[0170] Data analysis phase

[0171] 5. Receiving text data

[0172] The server receives the text data sent from the terminal.

[0173] Input: Text data sent from the terminal

[0174] Specific behavior: The server accepts an HTTP POST request at an API endpoint and retrieves text data.

[0175] Output: Text data received on the server

[0176] 6. Analysis of Text Data

[0177] The server analyzes the text data using a natural language processing (NLP) engine.

[0178] Input: Text data received on the server

[0179] Specific operation: Text data is analyzed using Python, NLTK, spaCy, etc. to extract sentence structure, speaking style, frequently used phrases, emotional expressions, etc.

[0180] Output: Parsed data and feature information

[0181] Step 5:

[0182] Model generation phase

[0183] 7. Training generative AI models

[0184] The server trains a generative artificial intelligence model based on the extracted features.

[0185] Input: Analyzed data and feature information

[0186] Specific operation: Train a generative artificial intelligence model (e.g., GPT-3) using feature data.

[0187] Output: A trained generative artificial intelligence model

[0188] 8. Submitting the generative AI model

[0189] The server sends the generated model to the terminal.

[0190] Input: A trained generative artificial intelligence model

[0191] Specific operation: The model file is sent to the terminal as the payload of the HTTP response.

[0192] Output: Generative AI model sent to device

[0193] Step 6:

[0194] Model Install Phase

[0195] 9. Receiving the generative AI model

[0196] The terminal receives the generative artificial intelligence model sent from the server.

[0197] Input: Generative AI model received from the server

[0198] Specific behavior: Saves the model file to local storage.

[0199] Output: Generative AI model stored in internal storage

[0200] 10. Installing the generative AI model

[0201] The device installs the received generative artificial intelligence model into the app.

[0202] Input: Generative AI model stored in local storage

[0203] What happens: The model file is integrated into the app's internal database.

[0204] Output: Installed generative artificial intelligence model

[0205] 11. Checking the operation of the generative AI model

[0206] The terminal checks the operation of the installed model.

[0207] Input: Installed generative artificial intelligence model

[0208] What it does: Generate test inputs and verify that the model responds appropriately.

[0209] Output: A generative AI model that has been verified to work

[0210] Step 7:

[0211] Conversation Simulation Phase

[0212] 12. App launch and voice input

[0213] The user (elderly person) launches the app and speaks into the microphone.

[0214] Input: User utterance (e.g., "The weather is nice today, did you go for a walk?")

[0215] Specific actions: Tap the voice input button in the app to start recording.

[0216] Output: Recorded audio data

[0217] 13. Converting voice to text data

[0218] The terminal collects the user's voice and converts it into text data using a voice recognition system.

[0219] Input: Recorded audio data

[0220] What happens: The audio file is sent to a speech recognition API and converted to text.

[0221] Output: Converted text data

[0222] 14. Inputting text data into a generative AI model

[0223] The terminal inputs the converted text data into a generative artificial intelligence model.

[0224] Input: Converted text data

[0225] Specific behavior: Text data is passed to a model in the app.

[0226] Output: The response generated by the model

[0227] 15. Response generation and speech output

[0228] The terminal outputs the response generated by the generative artificial intelligence model as voice.

[0229] Input: The response generated by the model

[0230] Specific behavior: The response is converted into audio through a speech synthesis engine (e.g., Google Text-to-Speech) and played back to the user.

[0231] Output: A voice response the user hears (e.g., "Hi, Mom. The weather was nice today, so I went for a little walk.")

[0232] (Application example 1)

[0233] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0234] When elderly people live far from family and friends, they may feel lonely and have fewer opportunities to talk, which can increase their mental burden. Furthermore, even when using food delivery services, the service often ends up being limited to simply handing over the goods, with no real human interaction. The present invention aims to solve these problems and provide a system that allows elderly people to enjoy more fulfilling communication.

[0235] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0236] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, and means for transmitting the converted text data to the server, thereby enabling the elderly person and the virtual conversation host to enjoy natural conversation during food delivery.

[0237] "User voice data" means voice signals collected when a user speaks.

[0238] "Means of collection" refers to the method by which devices such as smartphones and tablets capture audio via microphones and then convert it into data.

[0239] "Means for converting into text data" refers to technology that uses a voice recognition system to convert voice data into text information.

[0240] "Means for sending to a server" refers to a method for sending text data to a cloud server or a remote computer system via the Internet.

[0241] "Means for analyzing text data" refers to technology that uses a natural language processing engine to analyze text data and extract conversation characteristics and emotions.

[0242] "Means for extracting user characteristics" refers to a method of analyzing speaking style, frequently used phrases, thought patterns, etc., to extract information specific to the user.

[0243] "Means for learning generative artificial intelligence models" refers to a method of using machine learning algorithms to create complex generative AI models based on user feature data.

[0244] "Means for transmitting the generated model to the terminal" refers to a method for transferring the generated AI model from the cloud to the user's device.

[0245] "Means for installing a generative artificial intelligence model on a terminal" refers to a method for installing a received AI model on a device and making it usable.

[0246] "Means for generating responses from a generative artificial intelligence model" refers to a technique that uses an AI model to generate responses to user input.

[0247] "Means for outputting a response as voice" refers to a technology that converts the generated text response into voice using a voice synthesis system and plays it aloud to the user.

[0248] "A means for a conversational agent to converse with a user on behalf of a delivery person when delivering a package" is a way for a conversational AI agent to interact with a user when delivering food or other deliveries.

[0249] System Overview

[0250] This system allows elderly people to enjoy conversations with food delivery services by having a conversation agent instead of the delivery person, thereby reducing feelings of loneliness. The system mainly consists of a server, a terminal, and a user.

[0251] Program Overview

[0252] User data collection phase

[0253] A user launches an app on their smartphone and speaks into the microphone, for example, saying something like: "Hello, Mom. How was your day?" The device collects the spoken voice and converts it into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text). The text data is then sent to the server.

[0254] Data analysis phase

[0255] The server receives the text data and analyzes it using a natural language processing (NLP) engine (e.g., OpenAI (registered trademark) GPT-3). This analysis includes speaking style, frequently used phrases, emotional expressions, etc. The analysis results are extracted and saved as user characteristics.

[0256] Model generation phase

[0257] The server trains a generative artificial intelligence model based on the extracted features, which mimics specific speech and thought patterns, and sends the trained model to the device.

[0258] Model installation phase

[0259] The device receives the generative AI model sent from the server and installs it into the app. After installation, operation is checked.

[0260] Conversation Simulation Phase

[0261] When food is delivered, the user (elderly person) can launch the app and enjoy a conversation with the conversational agent. For example, if the user asks, "What kind of food has arrived today?", the device collects the speech and converts it into text data using a speech recognition system. The converted text data is input into a generative artificial intelligence model, which generates an appropriate response. The generated response is output as a voice such as, "Today's menu is an Italian set! It includes delicious pasta."

[0262] Hardware and software used

[0263] Hardware: smartphone, tablet, microphone

[0264] Software: Google Cloud Speech-to-Text API, OpenAI GPT-3, PyTorch, Hugging Face Transformers

[0265] Prompt Sentence Examples

[0266] The following is an example of a prompt statement to pass user input to the model:

[0267] User: Hello, what food arrived today?

[0268] Prompt the model:

[0269] "The user asks what kind of food has arrived. Example response: 'It's the Italian set today! It includes delicious pasta.'"

[0270] This system allows elderly people to enjoy conversation during food delivery, reducing the mental burden.

[0271] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0272] Step 1:

[0273] A user launches the app on their smartphone and speaks into the microphone, for example, "Hi, Mom. How was your day?"

[0274] Input: User's voice

[0275] Output: Audio data

[0276] How it works: Your smartphone's microphone picks up your voice, which is then collected by the app as digital audio data.

[0277] Step 2:

[0278] The device converts the collected voice data into text data using a voice recognition system (e.g., Google Cloud Speech-to-Text).

[0279] Input: Audio data

[0280] Output: Text data

[0281] How it works: Voice data is sent to a cloud-based voice recognition API, where it is processed and converted into text data in string format.

[0282] Step 3:

[0283] The terminal transmits the converted text data to the server.

[0284] Input: Text data

[0285] Output: Sends text data to the server

[0286] Specific operation: Text data is uploaded to a server via the Internet.

[0287] Step 4:

[0288] The server receives the text data and analyzes it using a natural language processing engine (e.g., OpenAI GPT-3).

[0289] Input: Text data

[0290] Output: Parsed user features

[0291] Specific operation: The server analyzes the text data and extracts characteristics such as speaking style, frequently used phrases, and emotional expressions.

[0292] Step 5:

[0293] The server trains a generative artificial intelligence model based on the extracted features.

[0294] Input: User characteristics

[0295] Output: A trained generative artificial intelligence model

[0296] How it works: A machine learning algorithm is run using the user's feature data to generate a customized generative AI model.

[0297] Step 6:

[0298] The server transmits the trained model to the terminal.

[0299] Input: A trained generative artificial intelligence model

[0300] Output: Sending the AI ​​model to the device

[0301] Specific operation: The generated AI model is sent to the device via the network.

[0302] Step 7:

[0303] The device receives the generative artificial intelligence model sent from the server and installs it within the app.

[0304] Input: Generative AI model

[0305] Output: Installed AI model

[0306] Specific operation: The received AI model is saved in the app and its operation is confirmed.

[0307] Step 8:

[0308] When a user (elderly person) receives a food delivery, they launch the app and speak into the microphone, for example, "What kind of food has arrived today?"

[0309] Input: User's voice

[0310] Output: Audio data

[0311] How it works: Your smartphone's microphone picks up your voice, which is then collected by the app as digital audio data.

[0312] Step 9:

[0313] The terminal converts the collected voice data into text data using a voice recognition system.

[0314] Input: Audio data

[0315] Output: Text data

[0316] What it does: The speech recognition system analyzes the audio data and converts it into text.

[0317] Step 10:

[0318] The terminal inputs the converted text data into a generative artificial intelligence model to generate an appropriate response.

[0319] Input: Text data

[0320] Output: Response text

[0321] Specific operation: The generative artificial intelligence model generates a response based on the input text data.

[0322] Step 11:

[0323] The device outputs the generated response as voice using a speech synthesis system (e.g., Google Cloud Text-to-Speech API).

[0324] Input: Response text

[0325] Output: Response voice

[0326] Specific operation: Text data is converted into audio data and output to the user through the speaker.

[0327] This will enable elderly people to enjoy fun conversations with a conversational agent during food delivery.

[0328] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0329] System Overview

[0330] The purpose of this invention is to enable elderly people in nursing homes to enjoy simulated conversations with family and friends when they are unable to meet in person. In particular, this invention analyzes the user's voice data and simulates conversations using a generative artificial intelligence model that learns the user's speaking style and thought patterns. In addition, by combining it with an emotion recognition engine, it is possible to recognize the user's emotions and reflect them in the content of the responses.

[0331] Program Overview

[0332] The program performs the following processing:

[0333] User data collection phase

[0334] 1. The user launches the app and speaks into the microphone.

[0335] For example: "Hi, Mom. How was your day?"

[0336] 2. The device collects the user's voice and saves it as voice data.

[0337] 3. The voice data collected by the device is converted into text data using a voice recognition system.

[0338] 4. The device sends the converted text data to the server.

[0339] Data analysis phase

[0340] 1. The server receives the text data sent from the terminal.

[0341] 2. The server uses a natural language processing (NLP) engine to analyze the text data and extract speech and thought patterns.

[0342] 3. The server uses an emotion engine to recognize the user's emotion from the voice data and adds the emotion information to the text data.

[0343] Model generation phase

[0344] 1. The server trains a generative AI model based on the features and emotional information extracted by the server.

[0345] 2. The server generates a trained generative artificial intelligence model and sends it to the terminal.

[0346] Model installation phase

[0347] 1. The device receives the generative AI model and installs it in the app.

[0348] 2. The device will check that the new model works.

[0349] Conversation Simulation Phase

[0350] 1. The user (elderly person) restarts the app and starts talking.

[0351] For example: "The weather is lovely today. Did you go for a walk?"

[0352] 2. The device collects the voice and converts it into text data using a voice recognition system.

[0353] 3. The device inputs this text data and emotional information into a generative artificial intelligence model.

[0354] 4. The device outputs the generated response to the user as speech.

[0355] For example: "Yes, Mom. The weather was nice today, so I went for a little walk."

[0356] Specific examples

[0357] For example, consider the case where a son uses this system for his mother who lives far away.

[0358] The user (son) talks to the app

[0359] The son says, "Hi, Mom. How was your day?" The app collects this speech and converts it into text using a speech recognition system. The converted text and voice data are then sent to the server.

[0360] Server analysis and model generation

[0361] The server analyzes the text data and extracts the son's speech patterns and thought patterns. The emotion engine also analyzes the voice data and recognizes the son's emotions. A generative AI model is trained based on the analysis results, and a model reflecting the son's speech patterns and emotions is generated. This model is then sent to the device.

[0362] Model installation and conversation simulation

[0363] Mom launches the app and says, "The weather is nice today. Did you go for a walk?" The device collects this voice and converts it into text data using a speech recognition system. The emotion engine then recognizes emotions from the input voice data. By inputting the text data and emotional information into a generative artificial intelligence model, an appropriate response is generated. This response is output as voice and played back as, "Yes, Mom. The weather is nice today, so I went for a short walk."

[0364] This system provides simulated conversations that take emotions into account, helping to alleviate feelings of loneliness and mental stress among the elderly and improving their quality of life.

[0365] The processing flow will be explained below.

[0366] Step 1:

[0367] The user launches the app on their smartphone, and the app interface appears, switching to audio recording mode.

[0368] Step 2:

[0369] The user speaks into the microphone, saying, "Hi, Mom. How was your day?"

[0370] Step 3:

[0371] The device collects the user's voice and stores it as digital audio data.

[0372] Step 4:

[0373] The device sends the collected voice data to the emotion engine, which recognizes the user's emotions from the voice. The emotion engine analyzes the emotions and saves the results as data.

[0374] Step 5:

[0375] The device sends the voice data, with the output of the emotion engine added, to the voice recognition system, which converts it into text data. The voice recognition system analyzes the voice and generates the text, "Hello, Mom. How was your day?"

[0376] Step 6:

[0377] The device then transmits the converted text data and emotion information to the server, where the data is transferred using a secure communication protocol.

[0378] Step 7:

[0379] The server receives the text data and emotion information sent from the device, and stores the received data in a data store for analysis.

[0380] Step 8:

[0381] The text data received by the server is analyzed using a natural language processing (NLP) engine to extract speaking style, frequently used phrases, emotional expressions, etc.

[0382] Step 9:

[0383] The server extracts the user's characteristics and emotional patterns from the analysis results and emotional information, and trains a generative AI model that uses machine learning algorithms to reflect the user's characteristics and emotions.

[0384] Step 10:

[0385] The server generates a trained generative AI model and sends it to the device. The generated model includes functions that can handle a variety of conversation scenarios.

[0386] Step 11:

[0387] The device receives the generative AI model sent from the server, and then installs the model into the app.

[0388] Step 12:

[0389] The operation of the generative AI model installed on the device is checked using a simple conversation scenario to verify whether the model generates accurate responses.

[0390] Step 13:

[0391] The user (an elderly person) restarts the app and starts a conversation with the generative AI model, saying, "The weather is nice today. Have you gone for a walk?"

[0392] Step 14:

[0393] The terminal collects the user's (elderly person's) voice, sends it to an emotion engine, adds emotional information, and then converts it into text data using a voice recognition system.

[0394] Step 15:

[0395] The device inputs the converted text data and emotion information into a generative artificial intelligence model, which generates an appropriate response based on this input.

[0396] Step 16:

[0397] The device will then output the generated response as speech, which will be "Yes, Mom. The weather was nice today, so I went for a short walk."

[0398] As a result, the system allows elderly people to enjoy simulated conversations with family and friends, and by adding emotion recognition functionality, it can provide more realistic and emotionally appropriate responses.

[0399] Example 2

[0400] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0401] In situations where it is difficult for elderly people to communicate directly with family and friends in nursing homes, there is a need to reduce their feelings of loneliness and mental burden and improve their quality of life through simulated conversations. However, conventional systems have difficulty in creating conversations that fully reflect the user's speaking style, thought patterns, and emotions, and have not been able to achieve simulated interactions that are close to real conversations.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0403] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, means for transmitting the converted text data to the server, means for analyzing the text data and extracting user characteristics in the server, means for recognizing the user's emotions from the extracted characteristics and voice data and adding emotional information, means for training a generative AI model based on the extracted characteristics and emotional information, means for transmitting the trained generative AI model to a terminal, means for installing the generative AI model in the terminal, means for generating a response from the generative AI model based on a user's input, and means for outputting the generated response as voice. This makes it possible to provide a more natural and emotional simulated conversation that reflects the user's speaking style, thought patterns, and emotions.

[0404] "Users" refer to individuals who use this system to provide voice data, including seniors, their families, and friends.

[0405] "Voice data" refers to data that is a digital recording of the user's speech.

[0406] "Text data" refers to data obtained by converting voice data into character information.

[0407] "Server" refers to a remotely located computer system that receives, analyzes, generates models, and performs other processing.

[0408] "Features" refers to specific information about an utterance, such as the user's speaking style, thought patterns, and emotions.

[0409] "Emotion information" refers to information indicating emotions recognized from the user's voice data.

[0410] A "generative artificial intelligence model" refers to an artificial intelligence system that learns based on the user's characteristics and emotional information.

[0411] "Device" refers to a device directly operated by a user that stores collected voice data, converts it to text data, installs generative AI models, etc. Examples include smartphones and tablets.

[0412] "Response" refers to the content of the utterance generated by the generative artificial intelligence model based on the user's input.

[0413] A "voice recognition system" refers to a technology that converts voice data into text data.

[0414] A "voice synthesis system" refers to a technology that outputs generated text data as voice.

[0415] MODE FOR CARRYING OUT THE INVENTION

[0416] This invention is a system that provides simulated conversations and psychological support to elderly people, particularly those living in nursing homes where it is difficult to converse directly with family and friends. This system uses technology that collects user voice data, trains a generative artificial intelligence model based on the user's speaking style and thought patterns, and generates responses that reflect emotional information.

[0417] Specifically, the system operates as follows.

[0418] User data collection phase

[0419] A user launches an application on a device such as a smartphone or tablet and speaks into the microphone. For example, they might say, "Hello, Mom. How was your day?" The device collects and stores this speech. The hardware used could be a smartphone or tablet, and the software could be a voice recording application.

[0420] Next, the device converts the collected voice data into text data using a voice recognition system (such as Google Speech-to-Text API or IBM Watson Speech-to-Text), and then transmits the converted text data to a server via HTTP / HTTPS protocol.

[0421] Data analysis phase

[0422] The server receives the text data sent from the device and analyzes it using a natural language processing (NLP) engine (such as NLTK or spaCy) to extract the user's speaking style and thought patterns.

[0423] Next, the server uses an emotion engine (for example, IBM Watson Tone Analyzer or Azure (registered trademark) Cognitive Services) to recognize the user's emotion from the voice data and adds the emotion information to the text data.

[0424] Model generation phase

[0425] Based on the results of the analysis, the server trains a generative AI model using the extracted features and emotion information. This process uses a machine learning framework (e.g., TENSORFLOW (registered trademark) or PyTorch). The trained generative AI model is output as a file and sent back to the terminal via HTTP / HTTPS protocol.

[0426] Model installation phase

[0427] The device installs the generative AI model received from the server into the application. This procedure includes application update functionality and an automatic installation process. The device also runs test scripts to verify the model's correct operation.

[0428] Conversation Simulation Phase

[0429] The user (e.g., an elderly person) launches the app again and says something like, "The weather is nice today. Did you go for a walk?" The device collects the audio and converts it into text again using a speech recognition system. The emotion engine also recognizes emotions from the input voice data. Finally, the device inputs the text data and emotion information into a generative artificial intelligence model to generate an appropriate response. This response is output as audio through a speech synthesis system (e.g., Google Text-to-Speech or Amazon Polly), and is played back as, "Yes, Mom. The weather is nice today, so I went for a short walk."

[0430] This system provides simulated conversations that take emotions into consideration, making it possible to reduce feelings of loneliness and mental stress, particularly among the elderly, and improve their quality of life.

[0431] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0432] Step 1:

[0433] The user launches the app and speaks into the microphone.

[0434] Input: Tap the app from the home screen of your smartphone or tablet to launch it. What you say (e.g., "Hi, Mom. How was your day?")

[0435] Specific behavior: The user taps the application to launch it and speaks into the microphone.

[0436] Output: Collected audio data

[0437] Step 2:

[0438] The device collects the user's voice and stores it as voice data.

[0439] Input: User speech, audio collected through a microphone

[0440] Specific operation: An application on the device activates the microphone, records and saves the user's voice.

[0441] Output: Saved audio data

[0442] Step 3:

[0443] The voice data collected by the terminal is converted into text data using a voice recognition system.

[0444] Input: Saved audio data

[0445] Specific operation: Calls the Google Speech-to-Text API or IBM Watson Speech-to-Text to convert voice data into text data.

[0446] Output: Converted text data

[0447] Step 4:

[0448] The terminal transmits the converted text data to the server.

[0449] Input: Converted text data

[0450] What it does: Sends data over an internet connection using HTTP / HTTPS protocols.

[0451] Output: Text data sent to the server

[0452] Step 5:

[0453] The server receives the text data sent from the terminal.

[0454] Input: Text data sent via HTTP / HTTPS protocol

[0455] What happens: The server receives the HTTP request and stores the data for analysis.

[0456] Output: Saved text data

[0457] Step 6:

[0458] The server uses a natural language processing (NLP) engine to analyze the text data and extract user characteristics.

[0459] Input: Saved text data

[0460] Specific operation: Analyzes text data using NLTK and spaCy to extract the user's speaking style and thought patterns.

[0461] Output: Extracted feature data

[0462] Step 7:

[0463] The server uses an emotion engine to recognize the user's emotion from the voice data and adds the emotion information to the text data.

[0464] Input: Audio data and text data

[0465] Specific operation: Emotions are analyzed using IBM Watson Tone Analyzer and Azure Cognitive Services, and emotional information is added to text data.

[0466] Output: Text data with emotional information added

[0467] Step 8:

[0468] The generative artificial intelligence model is trained based on the features and emotional information extracted by the server.

[0469] Input: Text data with extracted feature data and emotional information

[0470] Specific operations: Train generative artificial intelligence models using TensorFlow and PyTorch.

[0471] Output: A trained generative artificial intelligence model

[0472] Step 9:

[0473] The server generates a trained generative artificial intelligence model and transmits it to the terminal.

[0474] Input: A trained generative artificial intelligence model

[0475] Specific operation: The generated model is output in file format and sent to the terminal via HTTP / HTTPS protocol.

[0476] Output: The generative artificial intelligence model sent

[0477] Step 10:

[0478] The device receives the generative artificial intelligence model and installs it in the app.

[0479] Input: Generative AI model sent from the server

[0480] Specific operation: Install the model using the application's automatic update function.

[0481] Output: Installed generative artificial intelligence model

[0482] Step 11:

[0483] The device will check the operation of the new model.

[0484] Input: Installed generative artificial intelligence model

[0485] Specific behavior: Run test scripts to verify correct behavior of the model.

[0486] Output: Operation check result

[0487] Step 12:

[0488] The user (elderly person) restarts the app and speaks.

[0489] Input: Restart the app and what you say (e.g., "The weather is nice today. Have you gone for a walk?")

[0490] Specific behavior: The user launches the app and speaks into the microphone.

[0491] Output: Collected audio data

[0492] Step 13:

[0493] The device collects the voice and converts it into text data using a voice recognition system.

[0494] Input: Collected audio data

[0495] Specific operation: Converts into text data using Google Speech-to-Text API or IBM Watson Speech-to-Text.

[0496] Output: Converted text data

[0497] Step 14:

[0498] The device inputs text data and emotional information into a generative artificial intelligence model.

[0499] Input: Text data and emotion information

[0500] Specific operation: Text data and emotional information are input into a generative artificial intelligence model to generate a response.

[0501] Output: The generated response

[0502] Step 15:

[0503] The terminal outputs the generated response as speech to the user.

[0504] Input: Generated response text

[0505] Specific operation: Speech is synthesized using Google Text-to-Speech or Amazon Polly and output to the speaker.

[0506] Output: The response played as speech (e.g., "Yes, Mom. The weather was nice today, so I went for a short walk.")

[0507] These processing steps provide a simulated conversation that takes emotions into consideration, helping to alleviate feelings of loneliness and mental stress among the elderly and improving their quality of life.

[0508] (Application example 2)

[0509] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0510] One of the problems facing the elderly in modern society is loneliness and mental stress. In particular, there are many situations in which elderly people need support in physical stores, but they may not receive appropriate assistance. Furthermore, as opportunities for elderly people to enjoy face-to-face conversations with family and friends decrease, their quality of life may decline. To address these issues, a system is needed that can reduce feelings of loneliness for elderly people through simulated conversations and provide necessary guidance and support in physical stores.

[0511] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's voice data and extracting the user's characteristics and emotional information, means for training a generative AI model based on the extracted characteristics and emotional information, and means for transmitting the generated model to the interface device. This makes it possible to provide the elderly with simulated conversations with family or friends using a generative AI model that reproduces the user's speaking style, thought patterns, and emotions, and to further guide and support the elderly in physical stores.

[0512] "User's voice data" refers to information that is orally uttered by a user, and the content of that information is recorded as an acoustic signal.

[0513] "Text data" refers to data expressed as a string of characters based on voice data, and is the subject of natural language processing.

[0514] A "remote server" refers to a computer device installed in a remote location that is connected via a network such as the Internet, and that processes and stores data.

[0515] "Characteristic and emotional information" refers to information that indicates speaking style, thought patterns, and emotions extracted from voice data.

[0516] A "generative artificial intelligence model" refers to an artificial intelligence that generates responses to input data based on a trained algorithm.

[0517] An "interface device" refers to a device that allows a user and a system to exchange data with each other, such as a robot or a smart device.

[0518] "Response" refers to the reply or reaction that a generative artificial intelligence model generates based on user input.

[0519] A "physical store" refers to a commercial establishment located in a physical location where goods and services are offered.

[0520] System Overview

[0521] The system of the present invention allows elderly people to reduce feelings of loneliness and receive guidance and support through simulated conversations with family and friends in brick-and-mortar stores. The system collects and analyzes user voice data, and uses the results to train a generative artificial intelligence model to provide appropriate responses.

[0522] Hardware and software used

[0523] Hardware:

[0524] Remote Server: A powerful computing device that processes and stores data.

[0525] Interface devices: These include robots and smart devices used by the elderly. For example, robots are equipped with microphones, speakers, and voice recognition systems.

[0526] software:

[0527] Speech recognition systems (e.g., Google Speech-to-Text API)

[0528] Natural language processing engines (e.g., Google NLP, Microsoft® Azure NLP)

[0529] Emotion recognition engine (e.g. IBM Watson Tone Analyzer)

[0530] Data processing and calculation

[0531] The server uses a speech recognition system to collect the user's voice data and convert it into text data. The text data is analyzed by the server to extract the user's characteristics and emotional information. In particular, a natural language processing engine is used to analyze the grammar and syntax of the text data, and an emotion recognition engine is used to add emotional information.

[0532] A generative AI model is trained based on the analyzed data. The trained model is sent to the interface device, where it is installed and tested. Finally, when the user speaks to the interface device, their speech is again collected and converted into text data, and the generative AI model generates an appropriate response.

[0533] Specific examples

[0534] For example, an elderly person in a physical store might speak to an interface device (robot) and say, "I'm not sure which product to buy today. What do you recommend?" This voice data is collected in real time and converted into text data using a voice recognition system. The text data is sent to a remote server and analyzed by a natural language processing engine and an emotion recognition engine. Based on the analysis results, a generative artificial intelligence model generates a response, and the robot responds aloud with, "That's a good choice. It's especially popular with the elderly."

[0535] Prompt Sentence Examples

[0536] We are building a simulated conversation system for elderly people. We need to analyze the user's voice data and provide conversations that reflect their speaking style and emotions. Please talk to them about the following topics:

[0537] 1. I'm not sure which product to buy today, what do you recommend?

[0538] 2. What do you think of this product?

[0539] Consider emotional information (e.g., happy, sad, confused).

[0540] In this way, seniors can receive the guidance and support they need even when they are in a physical store, which is expected to reduce feelings of loneliness and improve their quality of life.

[0541] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0542] Step 1:

[0543] The user speaks to the robot, which is an interface device. For example, they input a voice message such as, "I'm not sure which product to buy today. What do you recommend?" This voice message is sent to the robot via a microphone.

[0544] Input: User's voice data

[0545] Output: Audio data collected by the robot

[0546] Specific operation: The robot's microphone picks up the user's speech and saves it as audio data.

[0547] Step 2:

[0548] The robot converts the collected voice data into text data using a voice recognition system, utilizing voice recognition software such as the Google Speech-to-Text API.

[0549] Input: Collected audio data

[0550] Output: Text data converted by the speech recognition system

[0551] Specific operation: Audio data is sent to the server and converted into text data using the Google Speech-to-Text API.

[0552] Step 3:

[0553] The robot sends the converted text data to a remote server, which receives the data.

[0554] Input: Text data

[0555] Output: Text data sent to the remote server

[0556] Specific operation: The robot uploads text data to a remote server via the network.

[0557] Step 4:

[0558] The server analyzes the received text data and extracts user characteristics using a natural language processing (NLP) engine. It also uses an emotion recognition engine to obtain emotional information and add it to the text data.

[0559] Input: Text data, emotion information

[0560] Output: Analysis data with user characteristics and emotional information added

[0561] Specific operation: The grammar and syntax of text data is analyzed using Google NLP and Microsoft Azure NLP, and emotions are recognized using IBM Watson Tone Analyzer.

[0562] Step 5:

[0563] The server trains a generative artificial intelligence model based on the analyzed features and emotion information. After the model is trained, the remote server transmits the model to the interface device.

[0564] Input: Analysis data

[0565] Output: A trained generative artificial intelligence model

[0566] Specific operation: Learning is performed using the analysis data, a model is generated, and it is deployed to the interface device.

[0567] Step 6:

[0568] The robot, which acts as an interface device, receives the learned generative AI model and installs it into the application, while simultaneously verifying its operation.

[0569] Input: A trained generative artificial intelligence model

[0570] Output: Installed generative artificial intelligence model

[0571] Specific operation: The robot downloads the model from the remote server, installs it in the app, and checks its operation.

[0572] Step 7:

[0573] When the user speaks to the interface device again, the voice data is collected again and converted into text data by the voice recognition system. The text data and emotion information are input into the generative artificial intelligence model.

[0574] Input: User's voice data, converted text data

[0575] Output: Data input into the generative artificial intelligence model

[0576] How it works: The robot collects the audio again, converts it into text data using the Google Speech-to-Text API, and inputs it into the model.

[0577] Step 8:

[0578] The robot generates an appropriate response from a generative artificial intelligence model and outputs the response to the user as voice.

[0579] Input: Text data input to the generative artificial intelligence model

[0580] Output: The response output as audio

[0581] Specific operation: The generative artificial intelligence model generates an appropriate response and replies to the user audibly through the robot's speaker.

[0582] This step will provide seniors with guidance and support in physical stores, reducing feelings of loneliness and improving their quality of life.

[0583] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0584] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0585] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0586] [Second embodiment]

[0587] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0588] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0589] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0590] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0591] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0592] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0593] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0594] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0595] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0596] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0597] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0598] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0599] System Overview

[0600] This system allows elderly people who are unable to meet with family and friends due to an infectious disease pandemic to enjoy simulated conversations. The system collects user voice data, analyzes it to generate a generative artificial intelligence model, and installs the model on the elderly person's device to realize simulated conversations.

[0601] Program Overview

[0602] The program performs the following specific processing.

[0603] User data collection phase

[0604] 1. The user launches the app on their smartphone and speaks into the microphone.

[0605] For example: "Hi, Mom. How was your day?"

[0606] 2. The device collects the user's voice and saves it as voice data.

[0607] 3. The device uses a voice recognition system to convert the collected voice data into text data.

[0608] 4. The terminal sends the converted text data to the server.

[0609] Data analysis phase

[0610] 1. The server receives the text data sent from the terminal.

[0611] 2. The server analyzes the text data using a natural language processing (NLP) engine.

[0612] Specifically, it extracts sentence structure, speaking style, frequently used phrases, emotional expressions, etc.

[0613] 3. The server extracts and saves the user's characteristics from the analysis results.

[0614] Model generation phase

[0615] 1. The server trains a generative artificial intelligence model based on the extracted features.

[0616] This generative artificial intelligence model mimics specific speech and thought patterns.

[0617] 2. The server sends the trained and generated model to the device.

[0618] Model installation phase

[0619] 1. The terminal receives the generative artificial intelligence model sent from the server.

[0620] 2. The device installs the received model into the app.

[0621] 3. The terminal checks the operation of the installed model.

[0622] Conversation Simulation Phase

[0623] 1. The user (elderly person) launches the app and speaks into the microphone.

[0624] For example: "The weather is lovely today, did you go for a walk?"

[0625] 2. The device collects the user's voice and converts it into text data using a voice recognition system.

[0626] 3. The terminal inputs the converted text data into a generative artificial intelligence model.

[0627] 4. The device receives the response generated by the generative artificial intelligence model and outputs this response as speech.

[0628] For example: "Yes, Mom. The weather was nice today, so I went for a little walk."

[0629] Specific examples

[0630] For example, consider the case where a son uses this system for his mother who lives far away.

[0631] The user (son) talks to the app

[0632] The son says, "Hello, Mom. How was your day?" The app collects this speech and converts it into text using a speech recognition system. This text data is then sent to the server.

[0633] Server analysis and model generation

[0634] The server analyzes the text data and extracts the son's speaking style and thought patterns. Based on this, a generative AI model is trained to generate a model that reproduces the son's unique conversational style. This model is then sent to the device.

[0635] Model installation and conversation simulation

[0636] When a mother launches the app and says, "The weather is nice today. Did you go for a walk?", the device analyzes the speech and generates an appropriate response based on a generative artificial intelligence model. This response is output as a voice message saying, "Yes, Mom. The weather is nice today, so I went for a short walk."

[0637] This system allows elderly people to enjoy simulated conversations with family and friends, reducing feelings of loneliness and mental stress.

[0638] The processing flow will be explained below.

[0639] Step 1:

[0640] The user launches the app on their smartphone, and the app interface appears, switching to audio recording mode.

[0641] Step 2:

[0642] The user speaks into the microphone, for example, "Hi, Mom. How was your day?"

[0643] Step 3:

[0644] The device collects the user's voice and stores it as digital audio data.

[0645] Step 4:

[0646] The device sends the collected voice data to a voice recognition system, which converts it into text data. The voice recognition system analyzes the voice and generates the text, "Hello, Mom. How was your day?"

[0647] Step 5:

[0648] The terminal transmits the converted text data to the server, where the data is transferred using a secure communication protocol.

[0649] Step 6:

[0650] The server receives the text data sent from the device and stores it in a data store for analysis.

[0651] Step 7:

[0652] The server runs a natural language processing (NLP) engine to analyze the text data it receives, analyzing specific phrases, grammatical structures, emotional expressions, etc., and extracting the user's speaking style and thought patterns.

[0653] Step 8:

[0654] The server trains a generative artificial intelligence model based on the extracted features. The model uses machine learning algorithms to reflect the user's characteristics.

[0655] Step 9:

[0656] The server generates a trained generative AI model and sends it to the device. The generated model includes functions that can handle a variety of conversation scenarios.

[0657] Step 10:

[0658] The device receives the generative AI model sent from the server, and then installs the model into the app.

[0659] Step 11:

[0660] The operation of the generative AI model installed on the device is checked using a simple conversation scenario to verify whether the model generates accurate responses.

[0661] Step 12:

[0662] The user (elderly person) restarts the app and starts a conversation with the generative AI model, for example, saying, "The weather is nice today. Have you gone for a walk?"

[0663] Step 13:

[0664] The device collects the user's (elderly person's) voice and converts it into text data using a voice recognition system.

[0665] Step 14:

[0666] The device inputs the converted text data into a generative artificial intelligence model, which generates an appropriate response based on the input.

[0667] Step 15:

[0668] The device then plays back the generated response as speech, for example, "Yes, Mom. The weather was nice today, so I went for a short walk."

[0669] Through this series of steps, the system enables elderly people to enjoy simulated conversations with family and friends.

[0670] Example 1

[0671] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0672] Elderly people face the problem of feeling lonely and experiencing increased mental stress due to fewer opportunities to talk with family and friends due to infectious disease outbreaks and other factors. To solve this problem, a system that allows elderly people to enjoy simulated conversations is needed. Furthermore, an advanced, non-contact conversation system that can reproduce the user's speaking style and thought patterns is also needed.

[0673] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0674] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, means for transmitting the converted text data to the server, means for analyzing the text data and extracting user characteristics in the server, means for training a generative artificial intelligence model based on the extracted characteristics, means for transmitting the generated model to a terminal, means for installing the generative artificial intelligence model in the terminal, means for generating a response from the generative artificial intelligence model based on the user's voice input, means for outputting the generated response as voice, means for learning and saving the user's characteristic speaking style and thought patterns using the generative artificial intelligence model, and means for confirming the operation of the generative artificial intelligence model in the terminal. This allows elderly people to enjoy simulated conversations with family and friends, thereby reducing feelings of loneliness and mental burden.

[0675] "Voice data" refers to data that is a digital recording of a user's voice.

[0676] "Text data" refers to voice data converted into a string of characters using a voice recognition system.

[0677] A "server" is a computer system that processes and stores data and communicates with other computers over a network.

[0678] A "terminal" is a device operated by a user, such as a smartphone or tablet.

[0679] "Converting means" means the system or algorithm used to convert audio data into text data.

[0680] "Means of analysis" refers to software or algorithms that analyze text data and extract specific information, such as natural language processing engines.

[0681] "Means of feature extraction" refers to the technology that performs the process of recognizing and extracting certain patterns or features from data.

[0682] A "generative artificial intelligence model" is a machine learning model trained to mimic a user's characteristic speaking and thinking patterns.

[0683] "Means of learning" refers to techniques for using data to train generative artificial intelligence models.

[0684] "Means for installing" refers to the process of incorporating the generated artificial intelligence model into the software of the terminal.

[0685] A "response generator" is a system or algorithm that generates an appropriate reply based on user input.

[0686] The "means for outputting as voice" is a system that converts the generated text response into voice and outputs it.

[0687] "Means for operational verification" is a process for testing whether the installed artificial intelligence model is functioning correctly.

[0688] "Simulated conversation" refers to a simulated dialogue that is conducted in a format similar to a real conversation.

[0689] This invention is a system that collects user voice data, analyzes it to generate a generative artificial intelligence model that imitates specific speaking styles and thought patterns, and installs the model on a terminal to realize simulated conversations.

[0690] This system mainly consists of devices such as smartphones and tablets, a server responsible for data analysis, a voice recognition system, a natural language processing engine, and a generative artificial intelligence model.

[0691] User data collection

[0692] First, the user opens the app on their smartphone and speaks into the microphone. For example, you can use the following prompt:

[0693] "Hi, Mom. How was your day?"

[0694] The device collects this voice and stores it in local storage as audio data. The collected voice data is converted into text data using a speech recognition system such as Google Cloud Speech-to-Text or IBM Watson Speech to Text. The converted text data is then sent to the server as an HTTP POST request.

[0695] Data analysis and model generation on the server

[0696] The server receives the text data sent from the device and analyzes it using Python and a natural language processing engine such as NLTK or spaCy. The analysis includes analyzing sentence structure, extracting speaking characteristics, analyzing frequently used phrases, and analyzing emotional expressions. The server extracts user features from the analysis results and uses this feature data to train a generative artificial intelligence model (e.g., GPT-3). The generated model is then sent to the device.

[0697] Installing the model on the device and checking its operation

[0698] The device receives the generative AI model sent from the server and installs it in the app. The operation of the installed model is confirmed by generating test inputs on the device and verifying that the model's responses are accurate.

[0699] Conversation simulation for elderly people

[0700] Elderly people can launch the app and say things like:

[0701] "The weather is nice today, did you go for a walk?"

[0702] The device collects this speech and converts it into text data using a speech recognition system. The converted text data is input into a generative artificial intelligence model, and the response generated by the model is output as speech. For example, the following response may be returned:

[0703] "Yes, Mom. The weather was nice today, so I went for a little walk."

[0704] Using this system, elderly people can enjoy simulated conversations with family and friends, reducing feelings of loneliness and mental stress.

[0705] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0706] Step 1:

[0707] Data Collection Phase

[0708] 1. Start voice input

[0709] The user launches the app on their smartphone and speaks into the microphone.

[0710] Input: What the user says (e.g., "Hi, Mom. How was your day?")

[0711] Specific actions: Tap the voice input button in the app to start recording.

[0712] Output: Recorded audio data

[0713] 2. Saving audio data

[0714] The terminal collects the user's voice and stores it as voice data in local storage.

[0715] Input: Recorded audio data

[0716] Specific operation: When recording is finished, the audio data will be saved in WAV format to the device's local storage.

[0717] Output: Saved audio file

[0718] Step 2:

[0719] Data Transformation Phase

[0720] 3. Convert voice to text data

[0721] The terminal converts the collected voice data into text data using a voice recognition system.

[0722] Input: Saved audio file

[0723] What happens: The audio file is sent to a speech recognition API, such as Google Cloud Speech-to-Text, and converted to text.

[0724] Output: Converted text data

[0725] Step 3:

[0726] Data transmission phase

[0727] 4. Sending text data to the server

[0728] The terminal transmits the converted text data to the server.

[0729] Input: Text data

[0730] Specific behavior: Sends text data to the server as an HTTP POST request.

[0731] Output: Text data sent to the server

[0732] Step 4:

[0733] Data analysis phase

[0734] 5. Receiving text data

[0735] The server receives the text data sent from the terminal.

[0736] Input: Text data sent from the terminal

[0737] Specific behavior: The server accepts an HTTP POST request at an API endpoint and retrieves text data.

[0738] Output: Text data received on the server

[0739] 6. Analysis of Text Data

[0740] The server analyzes the text data using a natural language processing (NLP) engine.

[0741] Input: Text data received on the server

[0742] Specific operation: Text data is analyzed using Python, NLTK, spaCy, etc. to extract sentence structure, speaking style, frequently used phrases, emotional expressions, etc.

[0743] Output: Parsed data and feature information

[0744] Step 5:

[0745] Model generation phase

[0746] 7. Training generative AI models

[0747] The server trains a generative artificial intelligence model based on the extracted features.

[0748] Input: Analyzed data and feature information

[0749] Specific operation: Train a generative artificial intelligence model (e.g., GPT-3) using feature data.

[0750] Output: A trained generative artificial intelligence model

[0751] 8. Submitting the generative AI model

[0752] The server sends the generated model to the terminal.

[0753] Input: A trained generative artificial intelligence model

[0754] Specific operation: The model file is sent to the terminal as the payload of the HTTP response.

[0755] Output: Generative AI model sent to device

[0756] Step 6:

[0757] Model Install Phase

[0758] 9. Receiving the generative AI model

[0759] The terminal receives the generative artificial intelligence model sent from the server.

[0760] Input: Generative AI model received from the server

[0761] Specific behavior: Saves the model file to local storage.

[0762] Output: Generative AI model stored in internal storage

[0763] 10. Installing the generative AI model

[0764] The device installs the received generative artificial intelligence model into the app.

[0765] Input: Generative AI model stored in local storage

[0766] What happens: The model file is integrated into the app's internal database.

[0767] Output: Installed generative artificial intelligence model

[0768] 11. Checking the operation of the generative AI model

[0769] The terminal checks the operation of the installed model.

[0770] Input: Installed generative artificial intelligence model

[0771] What it does: Generate test inputs and verify that the model responds appropriately.

[0772] Output: A generative AI model that has been verified to work

[0773] Step 7:

[0774] Conversation Simulation Phase

[0775] 12. App launch and voice input

[0776] The user (elderly person) launches the app and speaks into the microphone.

[0777] Input: User utterance (e.g., "The weather is nice today, did you go for a walk?")

[0778] Specific actions: Tap the voice input button in the app to start recording.

[0779] Output: Recorded audio data

[0780] 13. Converting voice to text data

[0781] The terminal collects the user's voice and converts it into text data using a voice recognition system.

[0782] Input: Recorded audio data

[0783] What happens: The audio file is sent to a speech recognition API and converted to text.

[0784] Output: Converted text data

[0785] 14. Inputting text data into a generative AI model

[0786] The terminal inputs the converted text data into a generative artificial intelligence model.

[0787] Input: Converted text data

[0788] Specific behavior: Text data is passed to a model in the app.

[0789] Output: The response generated by the model

[0790] 15. Response generation and speech output

[0791] The terminal outputs the response generated by the generative artificial intelligence model as voice.

[0792] Input: The response generated by the model

[0793] Specific behavior: The response is converted into audio through a speech synthesis engine (e.g., Google Text-to-Speech) and played back to the user.

[0794] Output: A voice response the user hears (e.g., "Hi, Mom. The weather was nice today, so I went for a little walk.")

[0795] (Application example 1)

[0796] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0797] When elderly people live far from family and friends, they may feel lonely and have fewer opportunities to talk, which can increase their mental burden. Furthermore, even when using food delivery services, the service often ends up being limited to simply handing over the goods, with no real human interaction. The present invention aims to solve these problems and provide a system that allows elderly people to enjoy more fulfilling communication.

[0798] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0799] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, and means for transmitting the converted text data to the server, thereby enabling the elderly person and the virtual conversation host to enjoy natural conversation during food delivery.

[0800] "User voice data" means voice signals collected when a user speaks.

[0801] "Means of collection" refers to the method by which devices such as smartphones and tablets capture audio via microphones and then convert it into data.

[0802] "Means for converting into text data" refers to technology that uses a voice recognition system to convert voice data into text information.

[0803] "Means for sending to a server" refers to a method for sending text data to a cloud server or a remote computer system via the Internet.

[0804] "Means for analyzing text data" refers to technology that uses a natural language processing engine to analyze text data and extract conversation characteristics and emotions.

[0805] "Means for extracting user characteristics" refers to a method of analyzing speaking style, frequently used phrases, thought patterns, etc., to extract information specific to the user.

[0806] "Means for learning generative artificial intelligence models" refers to a method of using machine learning algorithms to create complex generative AI models based on user feature data.

[0807] "Means for transmitting the generated model to the terminal" refers to a method for transferring the generated AI model from the cloud to the user's device.

[0808] "Means for installing a generative artificial intelligence model on a terminal" refers to a method for installing a received AI model on a device and making it usable.

[0809] "Means for generating responses from a generative artificial intelligence model" refers to a technique that uses an AI model to generate responses to user input.

[0810] "Means for outputting a response as voice" refers to a technology that converts the generated text response into voice using a voice synthesis system and plays it aloud to the user.

[0811] "A means for a conversational agent to converse with a user on behalf of a delivery person when delivering a package" is a way for a conversational AI agent to interact with a user when delivering food or other deliveries.

[0812] System Overview

[0813] This system allows elderly people to enjoy conversations with food delivery services by having a conversation agent instead of the delivery person, thereby reducing feelings of loneliness. The system mainly consists of a server, a terminal, and a user.

[0814] Program Overview

[0815] User data collection phase

[0816] A user launches an app on their smartphone and speaks into the microphone, for example, saying something like: "Hello, Mom. How was your day?" The device collects the spoken voice and converts it into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text). The text data is then sent to the server.

[0817] Data analysis phase

[0818] The server receives the text data and analyzes it using a natural language processing (NLP) engine (e.g., OpenAI GPT-3). This analysis includes speaking style, frequently used phrases, emotional expressions, etc. The analysis results are extracted and saved as user characteristics.

[0819] Model generation phase

[0820] The server trains a generative artificial intelligence model based on the extracted features, which mimics specific speech and thought patterns, and sends the trained model to the device.

[0821] Model installation phase

[0822] The device receives the generative AI model sent from the server and installs it into the app. After installation, operation is checked.

[0823] Conversation Simulation Phase

[0824] When food is delivered, the user (elderly person) can launch the app and enjoy a conversation with the conversational agent. For example, if the user asks, "What kind of food has arrived today?", the device collects the speech and converts it into text data using a speech recognition system. The converted text data is input into a generative artificial intelligence model, which generates an appropriate response. The generated response is output as a voice such as, "Today's menu is an Italian set! It includes delicious pasta."

[0825] Hardware and software used

[0826] Hardware: smartphone, tablet, microphone

[0827] Software: Google Cloud Speech-to-Text API, OpenAI GPT-3, PyTorch, Hugging Face Transformers

[0828] Prompt Sentence Examples

[0829] The following is an example of a prompt statement to pass user input to the model:

[0830] User: Hello, what food arrived today?

[0831] Prompt the model:

[0832] "The user asks what kind of food has arrived. Example response: 'It's the Italian set today! It includes delicious pasta.'"

[0833] This system allows elderly people to enjoy conversation during food delivery, reducing the mental burden.

[0834] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0835] Step 1:

[0836] A user launches the app on their smartphone and speaks into the microphone, for example, "Hi, Mom. How was your day?"

[0837] Input: User's voice

[0838] Output: Audio data

[0839] How it works: Your smartphone's microphone picks up your voice, which is then collected by the app as digital audio data.

[0840] Step 2:

[0841] The device converts the collected voice data into text data using a voice recognition system (e.g., Google Cloud Speech-to-Text).

[0842] Input: Audio data

[0843] Output: Text data

[0844] How it works: Voice data is sent to a cloud-based voice recognition API, where it is processed and converted into text data in string format.

[0845] Step 3:

[0846] The terminal transmits the converted text data to the server.

[0847] Input: Text data

[0848] Output: Sends text data to the server

[0849] Specific operation: Text data is uploaded to a server via the Internet.

[0850] Step 4:

[0851] The server receives the text data and analyzes it using a natural language processing engine (e.g., OpenAI GPT-3).

[0852] Input: Text data

[0853] Output: Parsed user features

[0854] Specific operation: The server analyzes the text data and extracts characteristics such as speaking style, frequently used phrases, and emotional expressions.

[0855] Step 5:

[0856] The server trains a generative artificial intelligence model based on the extracted features.

[0857] Input: User characteristics

[0858] Output: A trained generative artificial intelligence model

[0859] How it works: A machine learning algorithm is run using the user's feature data to generate a customized generative AI model.

[0860] Step 6:

[0861] The server transmits the trained model to the terminal.

[0862] Input: A trained generative artificial intelligence model

[0863] Output: Sending the AI ​​model to the device

[0864] Specific operation: The generated AI model is sent to the device via the network.

[0865] Step 7:

[0866] The device receives the generative artificial intelligence model sent from the server and installs it within the app.

[0867] Input: Generative AI model

[0868] Output: Installed AI model

[0869] Specific operation: The received AI model is saved in the app and its operation is confirmed.

[0870] Step 8:

[0871] When a user (elderly person) receives a food delivery, they launch the app and speak into the microphone, for example, "What kind of food has arrived today?"

[0872] Input: User's voice

[0873] Output: Audio data

[0874] How it works: Your smartphone's microphone picks up your voice, which is then collected by the app as digital audio data.

[0875] Step 9:

[0876] The terminal converts the collected voice data into text data using a voice recognition system.

[0877] Input: Audio data

[0878] Output: Text data

[0879] What it does: The speech recognition system analyzes the audio data and converts it into text.

[0880] Step 10:

[0881] The terminal inputs the converted text data into a generative artificial intelligence model to generate an appropriate response.

[0882] Input: Text data

[0883] Output: Response text

[0884] Specific operation: The generative artificial intelligence model generates a response based on the input text data.

[0885] Step 11:

[0886] The device outputs the generated response as voice using a speech synthesis system (e.g., Google Cloud Text-to-Speech API).

[0887] Input: Response text

[0888] Output: Response voice

[0889] Specific operation: Text data is converted into audio data and output to the user through the speaker.

[0890] This will enable elderly people to enjoy fun conversations with a conversational agent during food delivery.

[0891] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0892] System Overview

[0893] The purpose of this invention is to enable elderly people in nursing homes to enjoy simulated conversations with family and friends when they are unable to meet in person. In particular, this invention analyzes the user's voice data and simulates conversations using a generative artificial intelligence model that learns the user's speaking style and thought patterns. In addition, by combining it with an emotion recognition engine, it is possible to recognize the user's emotions and reflect them in the content of the responses.

[0894] Program Overview

[0895] The program performs the following processing:

[0896] User data collection phase

[0897] 1. The user launches the app and speaks into the microphone.

[0898] For example: "Hi, Mom. How was your day?"

[0899] 2. The device collects the user's voice and saves it as voice data.

[0900] 3. The voice data collected by the device is converted into text data using a voice recognition system.

[0901] 4. The device sends the converted text data to the server.

[0902] Data analysis phase

[0903] 1. The server receives the text data sent from the terminal.

[0904] 2. The server uses a natural language processing (NLP) engine to analyze the text data and extract speech and thought patterns.

[0905] 3. The server uses an emotion engine to recognize the user's emotion from the voice data and adds the emotion information to the text data.

[0906] Model generation phase

[0907] 1. The server trains a generative AI model based on the features and emotional information extracted by the server.

[0908] 2. The server generates a trained generative artificial intelligence model and sends it to the terminal.

[0909] Model installation phase

[0910] 1. The device receives the generative AI model and installs it in the app.

[0911] 2. The device will check that the new model works.

[0912] Conversation Simulation Phase

[0913] 1. The user (elderly person) restarts the app and starts talking.

[0914] For example: "The weather is lovely today. Did you go for a walk?"

[0915] 2. The device collects the voice and converts it into text data using a voice recognition system.

[0916] 3. The device inputs this text data and emotional information into a generative artificial intelligence model.

[0917] 4. The device outputs the generated response to the user as speech.

[0918] For example: "Yes, Mom. The weather was nice today, so I went for a little walk."

[0919] Specific examples

[0920] For example, consider the case where a son uses this system for his mother who lives far away.

[0921] The user (son) talks to the app

[0922] The son says, "Hi, Mom. How was your day?" The app collects this speech and converts it into text using a speech recognition system. The converted text and voice data are then sent to the server.

[0923] Server analysis and model generation

[0924] The server analyzes the text data and extracts the son's speech patterns and thought patterns. The emotion engine also analyzes the voice data and recognizes the son's emotions. A generative AI model is trained based on the analysis results, and a model reflecting the son's speech patterns and emotions is generated. This model is then sent to the device.

[0925] Model installation and conversation simulation

[0926] Mom launches the app and says, "The weather is nice today. Did you go for a walk?" The device collects this voice and converts it into text data using a speech recognition system. The emotion engine then recognizes emotions from the input voice data. By inputting the text data and emotional information into a generative artificial intelligence model, an appropriate response is generated. This response is output as voice and played back as, "Yes, Mom. The weather is nice today, so I went for a short walk."

[0927] This system provides simulated conversations that take emotions into account, helping to alleviate feelings of loneliness and mental stress among the elderly and improving their quality of life.

[0928] The processing flow will be explained below.

[0929] Step 1:

[0930] The user launches the app on their smartphone, and the app interface appears, switching to audio recording mode.

[0931] Step 2:

[0932] The user speaks into the microphone, saying, "Hi, Mom. How was your day?"

[0933] Step 3:

[0934] The device collects the user's voice and stores it as digital audio data.

[0935] Step 4:

[0936] The device sends the collected voice data to the emotion engine, which recognizes the user's emotions from the voice. The emotion engine analyzes the emotions and saves the results as data.

[0937] Step 5:

[0938] The device sends the voice data, with the output of the emotion engine added, to the voice recognition system, which converts it into text data. The voice recognition system analyzes the voice and generates the text, "Hello, Mom. How was your day?"

[0939] Step 6:

[0940] The device then transmits the converted text data and emotion information to the server, where the data is transferred using a secure communication protocol.

[0941] Step 7:

[0942] The server receives the text data and emotion information sent from the device, and stores the received data in a data store for analysis.

[0943] Step 8:

[0944] The text data received by the server is analyzed using a natural language processing (NLP) engine to extract speaking style, frequently used phrases, emotional expressions, etc.

[0945] Step 9:

[0946] The server extracts the user's characteristics and emotional patterns from the analysis results and emotional information, and trains a generative AI model that uses machine learning algorithms to reflect the user's characteristics and emotions.

[0947] Step 10:

[0948] The server generates a trained generative AI model and sends it to the device. The generated model includes functions that can handle a variety of conversation scenarios.

[0949] Step 11:

[0950] The device receives the generative AI model sent from the server, and then installs the model into the app.

[0951] Step 12:

[0952] The operation of the generative AI model installed on the device is checked using a simple conversation scenario to verify whether the model generates accurate responses.

[0953] Step 13:

[0954] The user (an elderly person) restarts the app and starts a conversation with the generative AI model, saying, "The weather is nice today. Have you gone for a walk?"

[0955] Step 14:

[0956] The terminal collects the user's (elderly person's) voice, sends it to an emotion engine, adds emotional information, and then converts it into text data using a voice recognition system.

[0957] Step 15:

[0958] The device inputs the converted text data and emotion information into a generative artificial intelligence model, which generates an appropriate response based on this input.

[0959] Step 16:

[0960] The device will then output the generated response as speech, which will be "Yes, Mom. The weather was nice today, so I went for a short walk."

[0961] As a result, the system allows elderly people to enjoy simulated conversations with family and friends, and by adding emotion recognition functionality, it can provide more realistic and emotionally appropriate responses.

[0962] Example 2

[0963] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0964] In situations where it is difficult for elderly people to communicate directly with family and friends in nursing homes, there is a need to reduce their feelings of loneliness and mental burden and improve their quality of life through simulated conversations. However, conventional systems have difficulty in creating conversations that fully reflect the user's speaking style, thought patterns, and emotions, and have not been able to achieve simulated interactions that are close to real conversations.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0966] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, means for transmitting the converted text data to the server, means for analyzing the text data and extracting user characteristics in the server, means for recognizing the user's emotions from the extracted characteristics and voice data and adding emotional information, means for training a generative AI model based on the extracted characteristics and emotional information, means for transmitting the trained generative AI model to a terminal, means for installing the generative AI model in the terminal, means for generating a response from the generative AI model based on a user's input, and means for outputting the generated response as voice. This makes it possible to provide a more natural and emotional simulated conversation that reflects the user's speaking style, thought patterns, and emotions.

[0967] "Users" refer to individuals who use this system to provide voice data, including seniors, their families, and friends.

[0968] "Voice data" refers to data that is a digital recording of the user's speech.

[0969] "Text data" refers to data obtained by converting voice data into character information.

[0970] "Server" refers to a remotely located computer system that receives, analyzes, generates models, and performs other processing.

[0971] "Features" refers to specific information about an utterance, such as the user's speaking style, thought patterns, and emotions.

[0972] "Emotion information" refers to information indicating emotions recognized from the user's voice data.

[0973] A "generative artificial intelligence model" refers to an artificial intelligence system that learns based on the user's characteristics and emotional information.

[0974] "Device" refers to a device directly operated by a user that stores collected voice data, converts it to text data, installs generative AI models, etc. Examples include smartphones and tablets.

[0975] "Response" refers to the content of the utterance generated by the generative artificial intelligence model based on the user's input.

[0976] A "voice recognition system" refers to a technology that converts voice data into text data.

[0977] A "voice synthesis system" refers to a technology that outputs generated text data as voice.

[0978] MODE FOR CARRYING OUT THE INVENTION

[0979] This invention is a system that provides simulated conversations and psychological support to elderly people, particularly those living in nursing homes where it is difficult to converse directly with family and friends. This system uses technology that collects user voice data, trains a generative artificial intelligence model based on the user's speaking style and thought patterns, and generates responses that reflect emotional information.

[0980] Specifically, the system operates as follows.

[0981] User data collection phase

[0982] A user launches an application on a device such as a smartphone or tablet and speaks into the microphone. For example, they might say, "Hello, Mom. How was your day?" The device collects and stores this speech. The hardware used could be a smartphone or tablet, and the software could be a voice recording application.

[0983] Next, the device converts the collected voice data into text data using a voice recognition system (such as Google Speech-to-Text API or IBM Watson Speech-to-Text), and then transmits the converted text data to a server via HTTP / HTTPS protocol.

[0984] Data analysis phase

[0985] The server receives the text data sent from the device and analyzes it using a natural language processing (NLP) engine (such as NLTK or spaCy) to extract the user's speaking style and thought patterns.

[0986] Next, the server uses an emotion engine (e.g., IBM Watson Tone Analyzer or Azure Cognitive Services) to recognize the user's emotion from the voice data and add the emotion information to the text data.

[0987] Model generation phase

[0988] Based on the results of the analysis, the server trains a generative AI model using the extracted features and emotion information. This process uses a machine learning framework (e.g., TensorFlow or PyTorch). The trained generative AI model is output as a file and sent back to the device via HTTP / HTTPS.

[0989] Model installation phase

[0990] The device installs the generative AI model received from the server into the application. This procedure includes application update functionality and an automatic installation process. The device also runs test scripts to verify the model's correct operation.

[0991] Conversation Simulation Phase

[0992] The user (e.g., an elderly person) launches the app again and says something like, "The weather is nice today. Did you go for a walk?" The device collects the audio and converts it into text again using a speech recognition system. The emotion engine also recognizes emotions from the input voice data. Finally, the device inputs the text data and emotion information into a generative artificial intelligence model to generate an appropriate response. This response is output as audio through a speech synthesis system (e.g., Google Text-to-Speech or Amazon Polly), and is played back as, "Yes, Mom. The weather is nice today, so I went for a short walk."

[0993] This system provides simulated conversations that take emotions into consideration, making it possible to reduce feelings of loneliness and mental stress, particularly among the elderly, and improve their quality of life.

[0994] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0995] Step 1:

[0996] The user launches the app and speaks into the microphone.

[0997] Input: Tap the app from the home screen of your smartphone or tablet to launch it. What you say (e.g., "Hi, Mom. How was your day?")

[0998] Specific behavior: The user taps the application to launch it and speaks into the microphone.

[0999] Output: Collected audio data

[1000] Step 2:

[1001] The device collects the user's voice and stores it as voice data.

[1002] Input: User speech, audio collected through a microphone

[1003] Specific operation: An application on the device activates the microphone, records and saves the user's voice.

[1004] Output: Saved audio data

[1005] Step 3:

[1006] The voice data collected by the terminal is converted into text data using a voice recognition system.

[1007] Input: Saved audio data

[1008] Specific operation: Calls the Google Speech-to-Text API or IBM Watson Speech-to-Text to convert voice data into text data.

[1009] Output: Converted text data

[1010] Step 4:

[1011] The terminal transmits the converted text data to the server.

[1012] Input: Converted text data

[1013] What it does: Sends data over an internet connection using HTTP / HTTPS protocols.

[1014] Output: Text data sent to the server

[1015] Step 5:

[1016] The server receives the text data sent from the terminal.

[1017] Input: Text data sent via HTTP / HTTPS protocol

[1018] What happens: The server receives the HTTP request and stores the data for analysis.

[1019] Output: Saved text data

[1020] Step 6:

[1021] The server uses a natural language processing (NLP) engine to analyze the text data and extract user characteristics.

[1022] Input: Saved text data

[1023] Specific operation: Analyzes text data using NLTK and spaCy to extract the user's speaking style and thought patterns.

[1024] Output: Extracted feature data

[1025] Step 7:

[1026] The server uses an emotion engine to recognize the user's emotion from the voice data and adds the emotion information to the text data.

[1027] Input: Audio data and text data

[1028] Specific operation: Emotions are analyzed using IBM Watson Tone Analyzer and Azure Cognitive Services, and emotional information is added to text data.

[1029] Output: Text data with emotional information added

[1030] Step 8:

[1031] The generative artificial intelligence model is trained based on the features and emotional information extracted by the server.

[1032] Input: Text data with extracted feature data and emotional information

[1033] Specific operations: Train generative artificial intelligence models using TensorFlow and PyTorch.

[1034] Output: A trained generative artificial intelligence model

[1035] Step 9:

[1036] The server generates a trained generative artificial intelligence model and transmits it to the terminal.

[1037] Input: A trained generative artificial intelligence model

[1038] Specific operation: The generated model is output in file format and sent to the terminal via HTTP / HTTPS protocol.

[1039] Output: The generative artificial intelligence model sent

[1040] Step 10:

[1041] The device receives the generative artificial intelligence model and installs it in the app.

[1042] Input: Generative AI model sent from the server

[1043] Specific operation: Install the model using the application's automatic update function.

[1044] Output: Installed generative artificial intelligence model

[1045] Step 11:

[1046] The device will check the operation of the new model.

[1047] Input: Installed generative artificial intelligence model

[1048] Specific behavior: Run test scripts to verify correct behavior of the model.

[1049] Output: Operation check result

[1050] Step 12:

[1051] The user (elderly person) restarts the app and speaks.

[1052] Input: Restart the app and what you say (e.g., "The weather is nice today. Have you gone for a walk?")

[1053] Specific behavior: The user launches the app and speaks into the microphone.

[1054] Output: Collected audio data

[1055] Step 13:

[1056] The device collects the voice and converts it into text data using a voice recognition system.

[1057] Input: Collected audio data

[1058] Specific operation: Converts into text data using Google Speech-to-Text API or IBM Watson Speech-to-Text.

[1059] Output: Converted text data

[1060] Step 14:

[1061] The device inputs text data and emotional information into a generative artificial intelligence model.

[1062] Input: Text data and emotion information

[1063] Specific operation: Text data and emotional information are input into a generative artificial intelligence model to generate a response.

[1064] Output: The generated response

[1065] Step 15:

[1066] The terminal outputs the generated response as speech to the user.

[1067] Input: Generated response text

[1068] Specific operation: Speech is synthesized using Google Text-to-Speech or Amazon Polly and output to the speaker.

[1069] Output: The response played as speech (e.g., "Yes, Mom. The weather was nice today, so I went for a short walk.")

[1070] These processing steps provide a simulated conversation that takes emotions into consideration, helping to alleviate feelings of loneliness and mental stress among the elderly and improving their quality of life.

[1071] (Application example 2)

[1072] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1073] One of the problems facing the elderly in modern society is loneliness and mental stress. In particular, there are many situations in which elderly people need support in physical stores, but they may not receive appropriate assistance. Furthermore, as opportunities for elderly people to enjoy face-to-face conversations with family and friends decrease, their quality of life may decline. To address these issues, a system is needed that can reduce feelings of loneliness for elderly people through simulated conversations and provide necessary guidance and support in physical stores.

[1074] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's voice data and extracting the user's characteristics and emotional information, means for training a generative AI model based on the extracted characteristics and emotional information, and means for transmitting the generated model to the interface device. This makes it possible to provide the elderly with simulated conversations with family or friends using a generative AI model that reproduces the user's speaking style, thought patterns, and emotions, and to further guide and support the elderly in physical stores.

[1075] "User's voice data" refers to information that is orally uttered by a user, and the content of that information is recorded as an acoustic signal.

[1076] "Text data" refers to data expressed as a string of characters based on voice data, and is the subject of natural language processing.

[1077] A "remote server" refers to a computer device installed in a remote location that is connected via a network such as the Internet, and that processes and stores data.

[1078] "Characteristic and emotional information" refers to information that indicates speaking style, thought patterns, and emotions extracted from voice data.

[1079] A "generative artificial intelligence model" refers to an artificial intelligence that generates responses to input data based on a trained algorithm.

[1080] An "interface device" refers to a device that allows a user and a system to exchange data with each other, such as a robot or a smart device.

[1081] "Response" refers to the reply or reaction that a generative artificial intelligence model generates based on user input.

[1082] A "physical store" refers to a commercial establishment located in a physical location where goods and services are offered.

[1083] System Overview

[1084] The system of the present invention allows elderly people to reduce feelings of loneliness and receive guidance and support through simulated conversations with family and friends in brick-and-mortar stores. The system collects and analyzes user voice data, and uses the results to train a generative artificial intelligence model to provide appropriate responses.

[1085] Hardware and software used

[1086] Hardware:

[1087] Remote Server: A powerful computing device that processes and stores data.

[1088] Interface devices: These include robots and smart devices used by the elderly. For example, robots are equipped with microphones, speakers, and voice recognition systems.

[1089] software:

[1090] Speech recognition systems (e.g., Google Speech-to-Text API)

[1091] Natural language processing engines (e.g., Google NLP, Microsoft Azure NLP)

[1092] Emotion recognition engine (e.g. IBM Watson Tone Analyzer)

[1093] Data processing and calculation

[1094] The server uses a speech recognition system to collect the user's voice data and convert it into text data. The text data is analyzed by the server to extract the user's characteristics and emotional information. In particular, a natural language processing engine is used to analyze the grammar and syntax of the text data, and an emotion recognition engine is used to add emotional information.

[1095] A generative AI model is trained based on the analyzed data. The trained model is sent to the interface device, where it is installed and tested. Finally, when the user speaks to the interface device, their speech is again collected and converted into text data, and the generative AI model generates an appropriate response.

[1096] Specific examples

[1097] For example, an elderly person in a physical store might speak to an interface device (robot) and say, "I'm not sure which product to buy today. What do you recommend?" This voice data is collected in real time and converted into text data using a voice recognition system. The text data is sent to a remote server and analyzed by a natural language processing engine and an emotion recognition engine. Based on the analysis results, a generative artificial intelligence model generates a response, and the robot responds aloud with, "That's a good choice. It's especially popular with the elderly."

[1098] Prompt Sentence Examples

[1099] We are building a simulated conversation system for elderly people. We need to analyze the user's voice data and provide conversations that reflect their speaking style and emotions. Please talk to them about the following topics:

[1100] 1. I'm not sure which product to buy today, what do you recommend?

[1101] 2. What do you think of this product?

[1102] Consider emotional information (e.g., happy, sad, confused).

[1103] In this way, seniors can receive the guidance and support they need even when they are in a physical store, which is expected to reduce feelings of loneliness and improve their quality of life.

[1104] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1105] Step 1:

[1106] The user speaks to the robot, which is an interface device. For example, they input a voice message such as, "I'm not sure which product to buy today. What do you recommend?" This voice message is sent to the robot via a microphone.

[1107] Input: User's voice data

[1108] Output: Audio data collected by the robot

[1109] Specific operation: The robot's microphone picks up the user's speech and saves it as audio data.

[1110] Step 2:

[1111] The robot converts the collected voice data into text data using a voice recognition system, utilizing voice recognition software such as the Google Speech-to-Text API.

[1112] Input: Collected audio data

[1113] Output: Text data converted by the speech recognition system

[1114] Specific operation: Audio data is sent to the server and converted into text data using the Google Speech-to-Text API.

[1115] Step 3:

[1116] The robot sends the converted text data to a remote server, which receives the data.

[1117] Input: Text data

[1118] Output: Text data sent to the remote server

[1119] Specific operation: The robot uploads text data to a remote server via the network.

[1120] Step 4:

[1121] The server analyzes the received text data and extracts user characteristics using a natural language processing (NLP) engine. It also uses an emotion recognition engine to obtain emotional information and add it to the text data.

[1122] Input: Text data, emotion information

[1123] Output: Analysis data with user characteristics and emotional information added

[1124] Specific operation: The grammar and syntax of text data is analyzed using Google NLP and Microsoft Azure NLP, and emotions are recognized using IBM Watson Tone Analyzer.

[1125] Step 5:

[1126] The server trains a generative artificial intelligence model based on the analyzed features and emotion information. After the model is trained, the remote server transmits the model to the interface device.

[1127] Input: Analysis data

[1128] Output: A trained generative artificial intelligence model

[1129] Specific operation: Learning is performed using the analysis data, a model is generated, and it is deployed to the interface device.

[1130] Step 6:

[1131] The robot, which acts as an interface device, receives the learned generative AI model and installs it into the application, while simultaneously verifying its operation.

[1132] Input: A trained generative artificial intelligence model

[1133] Output: Installed generative artificial intelligence model

[1134] Specific operation: The robot downloads the model from the remote server, installs it in the app, and checks its operation.

[1135] Step 7:

[1136] When the user speaks to the interface device again, the voice data is collected again and converted into text data by the voice recognition system. The text data and emotion information are input into the generative artificial intelligence model.

[1137] Input: User's voice data, converted text data

[1138] Output: Data input into the generative artificial intelligence model

[1139] How it works: The robot collects the audio again, converts it into text data using the Google Speech-to-Text API, and inputs it into the model.

[1140] Step 8:

[1141] The robot generates an appropriate response from a generative artificial intelligence model and outputs the response to the user as voice.

[1142] Input: Text data input to the generative artificial intelligence model

[1143] Output: The response output as audio

[1144] Specific operation: The generative artificial intelligence model generates an appropriate response and replies to the user audibly through the robot's speaker.

[1145] This step will provide seniors with guidance and support in physical stores, reducing feelings of loneliness and improving their quality of life.

[1146] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1147] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1148] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1149] [Third embodiment]

[1150] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1151] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1152] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1153] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1154] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1155] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1156] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1157] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1158] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1159] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1160] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1161] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1162] System Overview

[1163] This system allows elderly people who are unable to meet with family and friends due to an infectious disease pandemic to enjoy simulated conversations. The system collects user voice data, analyzes it to generate a generative artificial intelligence model, and installs the model on the elderly person's device to realize simulated conversations.

[1164] Program Overview

[1165] The program performs the following specific processing.

[1166] User data collection phase

[1167] 1. The user launches the app on their smartphone and speaks into the microphone.

[1168] For example: "Hi, Mom. How was your day?"

[1169] 2. The device collects the user's voice and saves it as voice data.

[1170] 3. The device uses a voice recognition system to convert the collected voice data into text data.

[1171] 4. The terminal sends the converted text data to the server.

[1172] Data analysis phase

[1173] 1. The server receives the text data sent from the terminal.

[1174] 2. The server analyzes the text data using a natural language processing (NLP) engine.

[1175] Specifically, it extracts sentence structure, speaking style, frequently used phrases, emotional expressions, etc.

[1176] 3. The server extracts and saves the user's characteristics from the analysis results.

[1177] Model generation phase

[1178] 1. The server trains a generative artificial intelligence model based on the extracted features.

[1179] This generative artificial intelligence model mimics specific speech and thought patterns.

[1180] 2. The server sends the trained and generated model to the device.

[1181] Model installation phase

[1182] 1. The terminal receives the generative artificial intelligence model sent from the server.

[1183] 2. The device installs the received model into the app.

[1184] 3. The terminal checks the operation of the installed model.

[1185] Conversation Simulation Phase

[1186] 1. The user (elderly person) launches the app and speaks into the microphone.

[1187] For example: "The weather is lovely today, did you go for a walk?"

[1188] 2. The device collects the user's voice and converts it into text data using a voice recognition system.

[1189] 3. The terminal inputs the converted text data into a generative artificial intelligence model.

[1190] 4. The device receives the response generated by the generative artificial intelligence model and outputs this response as speech.

[1191] For example: "Yes, Mom. The weather was nice today, so I went for a little walk."

[1192] Specific examples

[1193] For example, consider the case where a son uses this system for his mother who lives far away.

[1194] The user (son) talks to the app

[1195] The son says, "Hello, Mom. How was your day?" The app collects this speech and converts it into text using a speech recognition system. This text data is then sent to the server.

[1196] Server analysis and model generation

[1197] The server analyzes the text data and extracts the son's speaking style and thought patterns. Based on this, a generative AI model is trained to generate a model that reproduces the son's unique conversational style. This model is then sent to the device.

[1198] Model installation and conversation simulation

[1199] When a mother launches the app and says, "The weather is nice today. Did you go for a walk?", the device analyzes the speech and generates an appropriate response based on a generative artificial intelligence model. This response is output as a voice message saying, "Yes, Mom. The weather is nice today, so I went for a short walk."

[1200] This system allows elderly people to enjoy simulated conversations with family and friends, reducing feelings of loneliness and mental stress.

[1201] The processing flow will be explained below.

[1202] Step 1:

[1203] The user launches the app on their smartphone, and the app interface appears, switching to audio recording mode.

[1204] Step 2:

[1205] The user speaks into the microphone, for example, "Hi, Mom. How was your day?"

[1206] Step 3:

[1207] The device collects the user's voice and stores it as digital audio data.

[1208] Step 4:

[1209] The device sends the collected voice data to a voice recognition system, which converts it into text data. The voice recognition system analyzes the voice and generates the text, "Hello, Mom. How was your day?"

[1210] Step 5:

[1211] The terminal transmits the converted text data to the server, where the data is transferred using a secure communication protocol.

[1212] Step 6:

[1213] The server receives the text data sent from the device and stores it in a data store for analysis.

[1214] Step 7:

[1215] The server runs a natural language processing (NLP) engine to analyze the text data it receives, analyzing specific phrases, grammatical structures, emotional expressions, etc., and extracting the user's speaking style and thought patterns.

[1216] Step 8:

[1217] The server trains a generative artificial intelligence model based on the extracted features. The model uses machine learning algorithms to reflect the user's characteristics.

[1218] Step 9:

[1219] The server generates a trained generative AI model and sends it to the device. The generated model includes functions that can handle a variety of conversation scenarios.

[1220] Step 10:

[1221] The device receives the generative AI model sent from the server, and then installs the model into the app.

[1222] Step 11:

[1223] The operation of the generative AI model installed on the device is checked using a simple conversation scenario to verify whether the model generates accurate responses.

[1224] Step 12:

[1225] The user (elderly person) restarts the app and starts a conversation with the generative AI model, for example, saying, "The weather is nice today. Have you gone for a walk?"

[1226] Step 13:

[1227] The device collects the user's (elderly person's) voice and converts it into text data using a voice recognition system.

[1228] Step 14:

[1229] The device inputs the converted text data into a generative artificial intelligence model, which generates an appropriate response based on the input.

[1230] Step 15:

[1231] The device then plays back the generated response as speech, for example, "Yes, Mom. The weather was nice today, so I went for a short walk."

[1232] Through this series of steps, the system enables elderly people to enjoy simulated conversations with family and friends.

[1233] Example 1

[1234] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1235] Elderly people face the problem of feeling lonely and experiencing increased mental stress due to fewer opportunities to talk with family and friends due to infectious disease outbreaks and other factors. To solve this problem, a system that allows elderly people to enjoy simulated conversations is needed. Furthermore, an advanced, non-contact conversation system that can reproduce the user's speaking style and thought patterns is also needed.

[1236] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1237] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, means for transmitting the converted text data to the server, means for analyzing the text data and extracting user characteristics in the server, means for training a generative artificial intelligence model based on the extracted characteristics, means for transmitting the generated model to a terminal, means for installing the generative artificial intelligence model in the terminal, means for generating a response from the generative artificial intelligence model based on the user's voice input, means for outputting the generated response as voice, means for learning and saving the user's characteristic speaking style and thought patterns using the generative artificial intelligence model, and means for confirming the operation of the generative artificial intelligence model in the terminal. This allows elderly people to enjoy simulated conversations with family and friends, thereby reducing feelings of loneliness and mental burden.

[1238] "Voice data" refers to data that is a digital recording of a user's voice.

[1239] "Text data" refers to voice data converted into a string of characters using a voice recognition system.

[1240] A "server" is a computer system that processes and stores data and communicates with other computers over a network.

[1241] A "terminal" is a device operated by a user, such as a smartphone or tablet.

[1242] "Converting means" means the system or algorithm used to convert audio data into text data.

[1243] "Means of analysis" refers to software or algorithms that analyze text data and extract specific information, such as natural language processing engines.

[1244] "Means of feature extraction" refers to the technology that performs the process of recognizing and extracting certain patterns or features from data.

[1245] A "generative artificial intelligence model" is a machine learning model trained to mimic a user's characteristic speaking and thinking patterns.

[1246] "Means of learning" refers to techniques for using data to train generative artificial intelligence models.

[1247] "Means for installing" refers to the process of incorporating the generated artificial intelligence model into the software of the terminal.

[1248] A "response generator" is a system or algorithm that generates an appropriate reply based on user input.

[1249] The "means for outputting as voice" is a system that converts the generated text response into voice and outputs it.

[1250] "Means for operational verification" is a process for testing whether the installed artificial intelligence model is functioning correctly.

[1251] "Simulated conversation" refers to a simulated dialogue that is conducted in a format similar to a real conversation.

[1252] This invention is a system that collects user voice data, analyzes it to generate a generative artificial intelligence model that imitates specific speaking styles and thought patterns, and installs the model on a terminal to realize simulated conversations.

[1253] This system mainly consists of devices such as smartphones and tablets, a server responsible for data analysis, a voice recognition system, a natural language processing engine, and a generative artificial intelligence model.

[1254] User data collection

[1255] First, the user opens the app on their smartphone and speaks into the microphone. For example, you can use the following prompt:

[1256] "Hi, Mom. How was your day?"

[1257] The device collects this voice and stores it in local storage as audio data. The collected voice data is converted into text data using a speech recognition system such as Google Cloud Speech-to-Text or IBM Watson Speech to Text. The converted text data is then sent to the server as an HTTP POST request.

[1258] Data analysis and model generation on the server

[1259] The server receives the text data sent from the device and analyzes it using Python and a natural language processing engine such as NLTK or spaCy. The analysis includes analyzing sentence structure, extracting speaking characteristics, analyzing frequently used phrases, and analyzing emotional expressions. The server extracts user features from the analysis results and uses this feature data to train a generative artificial intelligence model (e.g., GPT-3). The generated model is then sent to the device.

[1260] Installing the model on the device and checking its operation

[1261] The device receives the generative AI model sent from the server and installs it in the app. The operation of the installed model is confirmed by generating test inputs on the device and verifying that the model's responses are accurate.

[1262] Conversation simulation for elderly people

[1263] Elderly people can launch the app and say things like:

[1264] "The weather is nice today, did you go for a walk?"

[1265] The device collects this speech and converts it into text data using a speech recognition system. The converted text data is input into a generative artificial intelligence model, and the response generated by the model is output as speech. For example, the following response may be returned:

[1266] "Yes, Mom. The weather was nice today, so I went for a little walk."

[1267] Using this system, elderly people can enjoy simulated conversations with family and friends, reducing feelings of loneliness and mental stress.

[1268] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1269] Step 1:

[1270] Data Collection Phase

[1271] 1. Start voice input

[1272] The user launches the app on their smartphone and speaks into the microphone.

[1273] Input: What the user says (e.g., "Hi, Mom. How was your day?")

[1274] Specific actions: Tap the voice input button in the app to start recording.

[1275] Output: Recorded audio data

[1276] 2. Saving audio data

[1277] The terminal collects the user's voice and stores it as voice data in local storage.

[1278] Input: Recorded audio data

[1279] Specific operation: When recording is finished, the audio data will be saved in WAV format to the device's local storage.

[1280] Output: Saved audio file

[1281] Step 2:

[1282] Data Transformation Phase

[1283] 3. Convert voice to text data

[1284] The terminal converts the collected voice data into text data using a voice recognition system.

[1285] Input: Saved audio file

[1286] What happens: The audio file is sent to a speech recognition API, such as Google Cloud Speech-to-Text, and converted to text.

[1287] Output: Converted text data

[1288] Step 3:

[1289] Data transmission phase

[1290] 4. Sending text data to the server

[1291] The terminal transmits the converted text data to the server.

[1292] Input: Text data

[1293] Specific behavior: Sends text data to the server as an HTTP POST request.

[1294] Output: Text data sent to the server

[1295] Step 4:

[1296] Data analysis phase

[1297] 5. Receiving text data

[1298] The server receives the text data sent from the terminal.

[1299] Input: Text data sent from the terminal

[1300] Specific behavior: The server accepts an HTTP POST request at an API endpoint and retrieves text data.

[1301] Output: Text data received on the server

[1302] 6. Analysis of Text Data

[1303] The server analyzes the text data using a natural language processing (NLP) engine.

[1304] Input: Text data received on the server

[1305] Specific operation: Text data is analyzed using Python, NLTK, spaCy, etc. to extract sentence structure, speaking style, frequently used phrases, emotional expressions, etc.

[1306] Output: Parsed data and feature information

[1307] Step 5:

[1308] Model generation phase

[1309] 7. Training generative AI models

[1310] The server trains a generative artificial intelligence model based on the extracted features.

[1311] Input: Analyzed data and feature information

[1312] Specific operation: Train a generative artificial intelligence model (e.g., GPT-3) using feature data.

[1313] Output: A trained generative artificial intelligence model

[1314] 8. Submitting the generative AI model

[1315] The server sends the generated model to the terminal.

[1316] Input: A trained generative artificial intelligence model

[1317] Specific operation: The model file is sent to the terminal as the payload of the HTTP response.

[1318] Output: Generative AI model sent to device

[1319] Step 6:

[1320] Model Install Phase

[1321] 9. Receiving the generative AI model

[1322] The terminal receives the generative artificial intelligence model sent from the server.

[1323] Input: Generative AI model received from the server

[1324] Specific behavior: Saves the model file to local storage.

[1325] Output: Generative AI model stored in internal storage

[1326] 10. Installing the generative AI model

[1327] The device installs the received generative artificial intelligence model into the app.

[1328] Input: Generative AI model stored in local storage

[1329] What happens: The model file is integrated into the app's internal database.

[1330] Output: Installed generative artificial intelligence model

[1331] 11. Checking the operation of the generative AI model

[1332] The terminal checks the operation of the installed model.

[1333] Input: Installed generative artificial intelligence model

[1334] What it does: Generate test inputs and verify that the model responds appropriately.

[1335] Output: A generative AI model that has been verified to work

[1336] Step 7:

[1337] Conversation Simulation Phase

[1338] 12. App launch and voice input

[1339] The user (elderly person) launches the app and speaks into the microphone.

[1340] Input: User utterance (e.g., "The weather is nice today, did you go for a walk?")

[1341] Specific actions: Tap the voice input button in the app to start recording.

[1342] Output: Recorded audio data

[1343] 13. Converting voice to text data

[1344] The terminal collects the user's voice and converts it into text data using a voice recognition system.

[1345] Input: Recorded audio data

[1346] What happens: The audio file is sent to a speech recognition API and converted to text.

[1347] Output: Converted text data

[1348] 14. Inputting text data into a generative AI model

[1349] The terminal inputs the converted text data into a generative artificial intelligence model.

[1350] Input: Converted text data

[1351] Specific behavior: Text data is passed to a model in the app.

[1352] Output: The response generated by the model

[1353] 15. Response generation and speech output

[1354] The terminal outputs the response generated by the generative artificial intelligence model as voice.

[1355] Input: The response generated by the model

[1356] Specific behavior: The response is converted into audio through a speech synthesis engine (e.g., Google Text-to-Speech) and played back to the user.

[1357] Output: A voice response the user hears (e.g., "Hi, Mom. The weather was nice today, so I went for a little walk.")

[1358] (Application example 1)

[1359] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1360] When elderly people live far from family and friends, they may feel lonely and have fewer opportunities to talk, which can increase their mental burden. Furthermore, even when using food delivery services, the service often ends up being limited to simply handing over the goods, with no real human interaction. The present invention aims to solve these problems and provide a system that allows elderly people to enjoy more fulfilling communication.

[1361] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1362] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, and means for transmitting the converted text data to the server, thereby enabling the elderly person and the virtual conversation host to enjoy natural conversation during food delivery.

[1363] "User voice data" means voice signals collected when a user speaks.

[1364] "Means of collection" refers to the method by which devices such as smartphones and tablets capture audio via microphones and then convert it into data.

[1365] "Means for converting into text data" refers to technology that uses a voice recognition system to convert voice data into text information.

[1366] "Means for sending to a server" refers to a method for sending text data to a cloud server or a remote computer system via the Internet.

[1367] "Means for analyzing text data" refers to technology that uses a natural language processing engine to analyze text data and extract conversation characteristics and emotions.

[1368] "Means for extracting user characteristics" refers to a method of analyzing speaking style, frequently used phrases, thought patterns, etc., to extract information specific to the user.

[1369] "Means for learning generative artificial intelligence models" refers to a method of using machine learning algorithms to create complex generative AI models based on user feature data.

[1370] "Means for transmitting the generated model to the terminal" refers to a method for transferring the generated AI model from the cloud to the user's device.

[1371] "Means for installing a generative artificial intelligence model on a terminal" refers to a method for installing a received AI model on a device and making it usable.

[1372] "Means for generating responses from a generative artificial intelligence model" refers to a technique that uses an AI model to generate responses to user input.

[1373] "Means for outputting a response as voice" refers to a technology that converts the generated text response into voice using a voice synthesis system and plays it aloud to the user.

[1374] "A means for a conversational agent to converse with a user on behalf of a delivery person when delivering a package" is a way for a conversational AI agent to interact with a user when delivering food or other deliveries.

[1375] System Overview

[1376] This system allows elderly people to enjoy conversations with food delivery services by having a conversation agent instead of the delivery person, thereby reducing feelings of loneliness. The system mainly consists of a server, a terminal, and a user.

[1377] Program Overview

[1378] User data collection phase

[1379] A user launches an app on their smartphone and speaks into the microphone, for example, saying something like: "Hello, Mom. How was your day?" The device collects the spoken voice and converts it into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text). The text data is then sent to the server.

[1380] Data analysis phase

[1381] The server receives the text data and analyzes it using a natural language processing (NLP) engine (e.g., OpenAI GPT-3). This analysis includes speaking style, frequently used phrases, emotional expressions, etc. The analysis results are extracted and saved as user characteristics.

[1382] Model generation phase

[1383] The server trains a generative artificial intelligence model based on the extracted features, which mimics specific speech and thought patterns, and sends the trained model to the device.

[1384] Model installation phase

[1385] The device receives the generative AI model sent from the server and installs it into the app. After installation, operation is checked.

[1386] Conversation Simulation Phase

[1387] When food is delivered, the user (elderly person) can launch the app and enjoy a conversation with the conversational agent. For example, if the user asks, "What kind of food has arrived today?", the device collects the speech and converts it into text data using a speech recognition system. The converted text data is input into a generative artificial intelligence model, which generates an appropriate response. The generated response is output as a voice such as, "Today's menu is an Italian set! It includes delicious pasta."

[1388] Hardware and software used

[1389] Hardware: smartphone, tablet, microphone

[1390] Software: Google Cloud Speech-to-Text API, OpenAI GPT-3, PyTorch, Hugging Face Transformers

[1391] Prompt Sentence Examples

[1392] The following is an example of a prompt statement to pass user input to the model:

[1393] User: Hello, what food arrived today?

[1394] Prompt the model:

[1395] "The user asks what kind of food has arrived. Example response: 'It's the Italian set today! It includes delicious pasta.'"

[1396] This system allows elderly people to enjoy conversation during food delivery, reducing the mental burden.

[1397] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1398] Step 1:

[1399] A user launches the app on their smartphone and speaks into the microphone, for example, "Hi, Mom. How was your day?"

[1400] Input: User's voice

[1401] Output: Audio data

[1402] How it works: Your smartphone's microphone picks up your voice, which is then collected by the app as digital audio data.

[1403] Step 2:

[1404] The device converts the collected voice data into text data using a voice recognition system (e.g., Google Cloud Speech-to-Text).

[1405] Input: Audio data

[1406] Output: Text data

[1407] How it works: Voice data is sent to a cloud-based voice recognition API, where it is processed and converted into text data in string format.

[1408] Step 3:

[1409] The terminal transmits the converted text data to the server.

[1410] Input: Text data

[1411] Output: Sends text data to the server

[1412] Specific operation: Text data is uploaded to a server via the Internet.

[1413] Step 4:

[1414] The server receives the text data and analyzes it using a natural language processing engine (e.g., OpenAI GPT-3).

[1415] Input: Text data

[1416] Output: Parsed user features

[1417] Specific operation: The server analyzes the text data and extracts characteristics such as speaking style, frequently used phrases, and emotional expressions.

[1418] Step 5:

[1419] The server trains a generative artificial intelligence model based on the extracted features.

[1420] Input: User characteristics

[1421] Output: A trained generative artificial intelligence model

[1422] How it works: A machine learning algorithm is run using the user's feature data to generate a customized generative AI model.

[1423] Step 6:

[1424] The server transmits the trained model to the terminal.

[1425] Input: A trained generative artificial intelligence model

[1426] Output: Sending the AI ​​model to the device

[1427] Specific operation: The generated AI model is sent to the device via the network.

[1428] Step 7:

[1429] The device receives the generative artificial intelligence model sent from the server and installs it within the app.

[1430] Input: Generative AI model

[1431] Output: Installed AI model

[1432] Specific operation: The received AI model is saved in the app and its operation is confirmed.

[1433] Step 8:

[1434] When a user (elderly person) receives a food delivery, they launch the app and speak into the microphone, for example, "What kind of food has arrived today?"

[1435] Input: User's voice

[1436] Output: Audio data

[1437] How it works: Your smartphone's microphone picks up your voice, which is then collected by the app as digital audio data.

[1438] Step 9:

[1439] The terminal converts the collected voice data into text data using a voice recognition system.

[1440] Input: Audio data

[1441] Output: Text data

[1442] What it does: The speech recognition system analyzes the audio data and converts it into text.

[1443] Step 10:

[1444] The terminal inputs the converted text data into a generative artificial intelligence model to generate an appropriate response.

[1445] Input: Text data

[1446] Output: Response text

[1447] Specific operation: The generative artificial intelligence model generates a response based on the input text data.

[1448] Step 11:

[1449] The device outputs the generated response as voice using a speech synthesis system (e.g., Google Cloud Text-to-Speech API).

[1450] Input: Response text

[1451] Output: Response voice

[1452] Specific operation: Text data is converted into audio data and output to the user through the speaker.

[1453] This will enable elderly people to enjoy fun conversations with a conversational agent during food delivery.

[1454] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1455] System Overview

[1456] The purpose of this invention is to enable elderly people in nursing homes to enjoy simulated conversations with family and friends when they are unable to meet in person. In particular, this invention analyzes the user's voice data and simulates conversations using a generative artificial intelligence model that learns the user's speaking style and thought patterns. In addition, by combining it with an emotion recognition engine, it is possible to recognize the user's emotions and reflect them in the content of the responses.

[1457] Program Overview

[1458] The program performs the following processing:

[1459] User data collection phase

[1460] 1. The user launches the app and speaks into the microphone.

[1461] For example: "Hi, Mom. How was your day?"

[1462] 2. The device collects the user's voice and saves it as voice data.

[1463] 3. The voice data collected by the device is converted into text data using a voice recognition system.

[1464] 4. The device sends the converted text data to the server.

[1465] Data analysis phase

[1466] 1. The server receives the text data sent from the terminal.

[1467] 2. The server uses a natural language processing (NLP) engine to analyze the text data and extract speech and thought patterns.

[1468] 3. The server uses an emotion engine to recognize the user's emotion from the voice data and adds the emotion information to the text data.

[1469] Model generation phase

[1470] 1. The server trains a generative AI model based on the features and emotional information extracted by the server.

[1471] 2. The server generates a trained generative artificial intelligence model and sends it to the terminal.

[1472] Model installation phase

[1473] 1. The device receives the generative AI model and installs it in the app.

[1474] 2. The device will check that the new model works.

[1475] Conversation Simulation Phase

[1476] 1. The user (elderly person) restarts the app and starts talking.

[1477] For example: "The weather is lovely today. Did you go for a walk?"

[1478] 2. The device collects the voice and converts it into text data using a voice recognition system.

[1479] 3. The device inputs this text data and emotional information into a generative artificial intelligence model.

[1480] 4. The device outputs the generated response to the user as speech.

[1481] For example: "Yes, Mom. The weather was nice today, so I went for a little walk."

[1482] Specific examples

[1483] For example, consider the case where a son uses this system for his mother who lives far away.

[1484] The user (son) talks to the app

[1485] The son says, "Hi, Mom. How was your day?" The app collects this speech and converts it into text using a speech recognition system. The converted text and voice data are then sent to the server.

[1486] Server analysis and model generation

[1487] The server analyzes the text data and extracts the son's speech patterns and thought patterns. The emotion engine also analyzes the voice data and recognizes the son's emotions. A generative AI model is trained based on the analysis results, and a model reflecting the son's speech patterns and emotions is generated. This model is then sent to the device.

[1488] Model installation and conversation simulation

[1489] Mom launches the app and says, "The weather is nice today. Did you go for a walk?" The device collects this voice and converts it into text data using a speech recognition system. The emotion engine then recognizes emotions from the input voice data. By inputting the text data and emotional information into a generative artificial intelligence model, an appropriate response is generated. This response is output as voice and played back as, "Yes, Mom. The weather is nice today, so I went for a short walk."

[1490] This system provides simulated conversations that take emotions into account, helping to alleviate feelings of loneliness and mental stress among the elderly and improving their quality of life.

[1491] The processing flow will be explained below.

[1492] Step 1:

[1493] The user launches the app on their smartphone, and the app interface appears, switching to audio recording mode.

[1494] Step 2:

[1495] The user speaks into the microphone, saying, "Hi, Mom. How was your day?"

[1496] Step 3:

[1497] The device collects the user's voice and stores it as digital audio data.

[1498] Step 4:

[1499] The device sends the collected voice data to the emotion engine, which recognizes the user's emotions from the voice. The emotion engine analyzes the emotions and saves the results as data.

[1500] Step 5:

[1501] The device sends the voice data, with the output of the emotion engine added, to the voice recognition system, which converts it into text data. The voice recognition system analyzes the voice and generates the text, "Hello, Mom. How was your day?"

[1502] Step 6:

[1503] The device then transmits the converted text data and emotion information to the server, where the data is transferred using a secure communication protocol.

[1504] Step 7:

[1505] The server receives the text data and emotion information sent from the device, and stores the received data in a data store for analysis.

[1506] Step 8:

[1507] The text data received by the server is analyzed using a natural language processing (NLP) engine to extract speaking style, frequently used phrases, emotional expressions, etc.

[1508] Step 9:

[1509] The server extracts the user's characteristics and emotional patterns from the analysis results and emotional information, and trains a generative AI model that uses machine learning algorithms to reflect the user's characteristics and emotions.

[1510] Step 10:

[1511] The server generates a trained generative AI model and sends it to the device. The generated model includes functions that can handle a variety of conversation scenarios.

[1512] Step 11:

[1513] The device receives the generative AI model sent from the server, and then installs the model into the app.

[1514] Step 12:

[1515] The operation of the generative AI model installed on the device is checked using a simple conversation scenario to verify whether the model generates accurate responses.

[1516] Step 13:

[1517] The user (an elderly person) restarts the app and starts a conversation with the generative AI model, saying, "The weather is nice today. Have you gone for a walk?"

[1518] Step 14:

[1519] The terminal collects the user's (elderly person's) voice, sends it to an emotion engine, adds emotional information, and then converts it into text data using a voice recognition system.

[1520] Step 15:

[1521] The device inputs the converted text data and emotion information into a generative artificial intelligence model, which generates an appropriate response based on this input.

[1522] Step 16:

[1523] The device will then output the generated response as speech, which will be "Yes, Mom. The weather was nice today, so I went for a short walk."

[1524] As a result, the system allows elderly people to enjoy simulated conversations with family and friends, and by adding emotion recognition functionality, it can provide more realistic and emotionally appropriate responses.

[1525] Example 2

[1526] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1527] In situations where it is difficult for elderly people to communicate directly with family and friends in nursing homes, there is a need to reduce their feelings of loneliness and mental burden and improve their quality of life through simulated conversations. However, conventional systems have difficulty in creating conversations that fully reflect the user's speaking style, thought patterns, and emotions, and have not been able to achieve simulated interactions that are close to real conversations.

[1528] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1529] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, means for transmitting the converted text data to the server, means for analyzing the text data and extracting user characteristics in the server, means for recognizing the user's emotions from the extracted characteristics and voice data and adding emotional information, means for training a generative AI model based on the extracted characteristics and emotional information, means for transmitting the trained generative AI model to a terminal, means for installing the generative AI model in the terminal, means for generating a response from the generative AI model based on a user's input, and means for outputting the generated response as voice. This makes it possible to provide a more natural and emotional simulated conversation that reflects the user's speaking style, thought patterns, and emotions.

[1530] "Users" refer to individuals who use this system to provide voice data, including seniors, their families, and friends.

[1531] "Voice data" refers to data that is a digital recording of the user's speech.

[1532] "Text data" refers to data obtained by converting voice data into character information.

[1533] "Server" refers to a remotely located computer system that receives, analyzes, generates models, and performs other processing.

[1534] "Features" refers to specific information about an utterance, such as the user's speaking style, thought patterns, and emotions.

[1535] "Emotion information" refers to information indicating emotions recognized from the user's voice data.

[1536] A "generative artificial intelligence model" refers to an artificial intelligence system that learns based on the user's characteristics and emotional information.

[1537] "Device" refers to a device directly operated by a user that stores collected voice data, converts it to text data, installs generative AI models, etc. Examples include smartphones and tablets.

[1538] "Response" refers to the content of the utterance generated by the generative artificial intelligence model based on the user's input.

[1539] A "voice recognition system" refers to a technology that converts voice data into text data.

[1540] A "voice synthesis system" refers to a technology that outputs generated text data as voice.

[1541] MODE FOR CARRYING OUT THE INVENTION

[1542] This invention is a system that provides simulated conversations and psychological support to elderly people, particularly those living in nursing homes where it is difficult to converse directly with family and friends. This system uses technology that collects user voice data, trains a generative artificial intelligence model based on the user's speaking style and thought patterns, and generates responses that reflect emotional information.

[1543] Specifically, the system operates as follows.

[1544] User data collection phase

[1545] A user launches an application on a device such as a smartphone or tablet and speaks into the microphone. For example, they might say, "Hello, Mom. How was your day?" The device collects and stores this speech. The hardware used could be a smartphone or tablet, and the software could be a voice recording application.

[1546] Next, the device converts the collected voice data into text data using a voice recognition system (such as Google Speech-to-Text API or IBM Watson Speech-to-Text), and then transmits the converted text data to a server via HTTP / HTTPS protocol.

[1547] Data analysis phase

[1548] The server receives the text data sent from the device and analyzes it using a natural language processing (NLP) engine (such as NLTK or spaCy) to extract the user's speaking style and thought patterns.

[1549] Next, the server uses an emotion engine (e.g., IBM Watson Tone Analyzer or Azure Cognitive Services) to recognize the user's emotion from the voice data and add the emotion information to the text data.

[1550] Model generation phase

[1551] Based on the results of the analysis, the server trains a generative AI model using the extracted features and emotion information. This process uses a machine learning framework (e.g., TensorFlow or PyTorch). The trained generative AI model is output as a file and sent back to the device via HTTP / HTTPS.

[1552] Model installation phase

[1553] The device installs the generative AI model received from the server into the application. This procedure includes application update functionality and an automatic installation process. The device also runs test scripts to verify the model's correct operation.

[1554] Conversation Simulation Phase

[1555] The user (e.g., an elderly person) launches the app again and says something like, "The weather is nice today. Did you go for a walk?" The device collects the audio and converts it into text again using a speech recognition system. The emotion engine also recognizes emotions from the input voice data. Finally, the device inputs the text data and emotion information into a generative artificial intelligence model to generate an appropriate response. This response is output as audio through a speech synthesis system (e.g., Google Text-to-Speech or Amazon Polly), and is played back as, "Yes, Mom. The weather is nice today, so I went for a short walk."

[1556] This system provides simulated conversations that take emotions into consideration, making it possible to reduce feelings of loneliness and mental stress, particularly among the elderly, and improve their quality of life.

[1557] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1558] Step 1:

[1559] The user launches the app and speaks into the microphone.

[1560] Input: Tap the app from the home screen of your smartphone or tablet to launch it. What you say (e.g., "Hi, Mom. How was your day?")

[1561] Specific behavior: The user taps the application to launch it and speaks into the microphone.

[1562] Output: Collected audio data

[1563] Step 2:

[1564] The device collects the user's voice and stores it as voice data.

[1565] Input: User speech, audio collected through a microphone

[1566] Specific operation: An application on the device activates the microphone, records and saves the user's voice.

[1567] Output: Saved audio data

[1568] Step 3:

[1569] The voice data collected by the terminal is converted into text data using a voice recognition system.

[1570] Input: Saved audio data

[1571] Specific operation: Calls the Google Speech-to-Text API or IBM Watson Speech-to-Text to convert voice data into text data.

[1572] Output: Converted text data

[1573] Step 4:

[1574] The terminal transmits the converted text data to the server.

[1575] Input: Converted text data

[1576] What it does: Sends data over an internet connection using HTTP / HTTPS protocols.

[1577] Output: Text data sent to the server

[1578] Step 5:

[1579] The server receives the text data sent from the terminal.

[1580] Input: Text data sent via HTTP / HTTPS protocol

[1581] What happens: The server receives the HTTP request and stores the data for analysis.

[1582] Output: Saved text data

[1583] Step 6:

[1584] The server uses a natural language processing (NLP) engine to analyze the text data and extract user characteristics.

[1585] Input: Saved text data

[1586] Specific operation: Analyzes text data using NLTK and spaCy to extract the user's speaking style and thought patterns.

[1587] Output: Extracted feature data

[1588] Step 7:

[1589] The server uses an emotion engine to recognize the user's emotion from the voice data and adds the emotion information to the text data.

[1590] Input: Audio data and text data

[1591] Specific operation: Emotions are analyzed using IBM Watson Tone Analyzer and Azure Cognitive Services, and emotional information is added to text data.

[1592] Output: Text data with emotional information added

[1593] Step 8:

[1594] The generative artificial intelligence model is trained based on the features and emotional information extracted by the server.

[1595] Input: Text data with extracted feature data and emotional information

[1596] Specific operations: Train generative artificial intelligence models using TensorFlow and PyTorch.

[1597] Output: A trained generative artificial intelligence model

[1598] Step 9:

[1599] The server generates a trained generative artificial intelligence model and transmits it to the terminal.

[1600] Input: A trained generative artificial intelligence model

[1601] Specific operation: The generated model is output in file format and sent to the terminal via HTTP / HTTPS protocol.

[1602] Output: The generative artificial intelligence model sent

[1603] Step 10:

[1604] The device receives the generative artificial intelligence model and installs it in the app.

[1605] Input: Generative AI model sent from the server

[1606] Specific operation: Install the model using the application's automatic update function.

[1607] Output: Installed generative artificial intelligence model

[1608] Step 11:

[1609] The device will check the operation of the new model.

[1610] Input: Installed generative artificial intelligence model

[1611] Specific behavior: Run test scripts to verify correct behavior of the model.

[1612] Output: Operation check result

[1613] Step 12:

[1614] The user (elderly person) restarts the app and speaks.

[1615] Input: Restart the app and what you say (e.g., "The weather is nice today. Have you gone for a walk?")

[1616] Specific behavior: The user launches the app and speaks into the microphone.

[1617] Output: Collected audio data

[1618] Step 13:

[1619] The device collects the voice and converts it into text data using a voice recognition system.

[1620] Input: Collected audio data

[1621] Specific operation: Converts into text data using Google Speech-to-Text API or IBM Watson Speech-to-Text.

[1622] Output: Converted text data

[1623] Step 14:

[1624] The device inputs text data and emotional information into a generative artificial intelligence model.

[1625] Input: Text data and emotion information

[1626] Specific operation: Text data and emotional information are input into a generative artificial intelligence model to generate a response.

[1627] Output: The generated response

[1628] Step 15:

[1629] The terminal outputs the generated response as speech to the user.

[1630] Input: Generated response text

[1631] Specific operation: Speech is synthesized using Google Text-to-Speech or Amazon Polly and output to the speaker.

[1632] Output: The response played as speech (e.g., "Yes, Mom. The weather was nice today, so I went for a short walk.")

[1633] These processing steps provide a simulated conversation that takes emotions into consideration, helping to alleviate feelings of loneliness and mental stress among the elderly and improving their quality of life.

[1634] (Application example 2)

[1635] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1636] One of the problems facing the elderly in modern society is loneliness and mental stress. In particular, there are many situations in which elderly people need support in physical stores, but they may not receive appropriate assistance. Furthermore, as opportunities for elderly people to enjoy face-to-face conversations with family and friends decrease, their quality of life may decline. To address these issues, a system is needed that can reduce feelings of loneliness for elderly people through simulated conversations and provide necessary guidance and support in physical stores.

[1637] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's voice data and extracting the user's characteristics and emotional information, means for training a generative AI model based on the extracted characteristics and emotional information, and means for transmitting the generated model to the interface device. This makes it possible to provide the elderly with simulated conversations with family or friends using a generative AI model that reproduces the user's speaking style, thought patterns, and emotions, and to further guide and support the elderly in physical stores.

[1638] "User's voice data" refers to information that is orally uttered by a user, and the content of that information is recorded as an acoustic signal.

[1639] "Text data" refers to data expressed as a string of characters based on voice data, and is the subject of natural language processing.

[1640] A "remote server" refers to a computer device installed in a remote location that is connected via a network such as the Internet, and that processes and stores data.

[1641] "Characteristic and emotional information" refers to information that indicates speaking style, thought patterns, and emotions extracted from voice data.

[1642] A "generative artificial intelligence model" refers to an artificial intelligence that generates responses to input data based on a trained algorithm.

[1643] An "interface device" refers to a device that allows a user and a system to exchange data with each other, such as a robot or a smart device.

[1644] "Response" refers to the reply or reaction that a generative artificial intelligence model generates based on user input.

[1645] A "physical store" refers to a commercial establishment located in a physical location where goods and services are offered.

[1646] System Overview

[1647] The system of the present invention allows elderly people to reduce feelings of loneliness and receive guidance and support through simulated conversations with family and friends in brick-and-mortar stores. The system collects and analyzes user voice data, and uses the results to train a generative artificial intelligence model to provide appropriate responses.

[1648] Hardware and software used

[1649] Hardware:

[1650] Remote Server: A powerful computing device that processes and stores data.

[1651] Interface devices: These include robots and smart devices used by the elderly. For example, robots are equipped with microphones, speakers, and voice recognition systems.

[1652] software:

[1653] Speech recognition systems (e.g., Google Speech-to-Text API)

[1654] Natural language processing engines (e.g., Google NLP, Microsoft Azure NLP)

[1655] Emotion recognition engine (e.g. IBM Watson Tone Analyzer)

[1656] Data processing and calculation

[1657] The server uses a speech recognition system to collect the user's voice data and convert it into text data. The text data is analyzed by the server to extract the user's characteristics and emotional information. In particular, a natural language processing engine is used to analyze the grammar and syntax of the text data, and an emotion recognition engine is used to add emotional information.

[1658] A generative AI model is trained based on the analyzed data. The trained model is sent to the interface device, where it is installed and tested. Finally, when the user speaks to the interface device, their speech is again collected and converted into text data, and the generative AI model generates an appropriate response.

[1659] Specific examples

[1660] For example, an elderly person in a physical store might speak to an interface device (robot) and say, "I'm not sure which product to buy today. What do you recommend?" This voice data is collected in real time and converted into text data using a voice recognition system. The text data is sent to a remote server and analyzed by a natural language processing engine and an emotion recognition engine. Based on the analysis results, a generative artificial intelligence model generates a response, and the robot responds aloud with, "That's a good choice. It's especially popular with the elderly."

[1661] Prompt Sentence Examples

[1662] We are building a simulated conversation system for elderly people. We need to analyze the user's voice data and provide conversations that reflect their speaking style and emotions. Please talk to them about the following topics:

[1663] 1. I'm not sure which product to buy today, what do you recommend?

[1664] 2. What do you think of this product?

[1665] Consider emotional information (e.g., happy, sad, confused).

[1666] In this way, seniors can receive the guidance and support they need even when they are in a physical store, which is expected to reduce feelings of loneliness and improve their quality of life.

[1667] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1668] Step 1:

[1669] The user speaks to the robot, which is an interface device. For example, they input a voice message such as, "I'm not sure which product to buy today. What do you recommend?" This voice message is sent to the robot via a microphone.

[1670] Input: User's voice data

[1671] Output: Audio data collected by the robot

[1672] Specific operation: The robot's microphone picks up the user's speech and saves it as audio data.

[1673] Step 2:

[1674] The robot converts the collected voice data into text data using a voice recognition system, utilizing voice recognition software such as the Google Speech-to-Text API.

[1675] Input: Collected audio data

[1676] Output: Text data converted by the speech recognition system

[1677] Specific operation: Audio data is sent to the server and converted into text data using the Google Speech-to-Text API.

[1678] Step 3:

[1679] The robot sends the converted text data to a remote server, which receives the data.

[1680] Input: Text data

[1681] Output: Text data sent to the remote server

[1682] Specific operation: The robot uploads text data to a remote server via the network.

[1683] Step 4:

[1684] The server analyzes the received text data and extracts user characteristics using a natural language processing (NLP) engine. It also uses an emotion recognition engine to obtain emotional information and add it to the text data.

[1685] Input: Text data, emotion information

[1686] Output: Analysis data with user characteristics and emotional information added

[1687] Specific operation: The grammar and syntax of text data is analyzed using Google NLP and Microsoft Azure NLP, and emotions are recognized using IBM Watson Tone Analyzer.

[1688] Step 5:

[1689] The server trains a generative artificial intelligence model based on the analyzed features and emotion information. After the model is trained, the remote server transmits the model to the interface device.

[1690] Input: Analysis data

[1691] Output: A trained generative artificial intelligence model

[1692] Specific operation: Learning is performed using the analysis data, a model is generated, and it is deployed to the interface device.

[1693] Step 6:

[1694] The robot, which acts as an interface device, receives the learned generative AI model and installs it into the application, while simultaneously verifying its operation.

[1695] Input: A trained generative artificial intelligence model

[1696] Output: Installed generative artificial intelligence model

[1697] Specific operation: The robot downloads the model from the remote server, installs it in the app, and checks its operation.

[1698] Step 7:

[1699] When the user speaks to the interface device again, the voice data is collected again and converted into text data by the voice recognition system. The text data and emotion information are input into the generative artificial intelligence model.

[1700] Input: User's voice data, converted text data

[1701] Output: Data input into the generative artificial intelligence model

[1702] How it works: The robot collects the audio again, converts it into text data using the Google Speech-to-Text API, and inputs it into the model.

[1703] Step 8:

[1704] The robot generates an appropriate response from a generative artificial intelligence model and outputs the response to the user as voice.

[1705] Input: Text data input to the generative artificial intelligence model

[1706] Output: The response output as audio

[1707] Specific operation: The generative artificial intelligence model generates an appropriate response and replies to the user audibly through the robot's speaker.

[1708] This step will provide seniors with guidance and support in physical stores, reducing feelings of loneliness and improving their quality of life.

[1709] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1710] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1711] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1712] [Fourth embodiment]

[1713] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1714] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1715] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1716] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1717] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1718] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1719] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1720] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1721] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1722] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1723] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1724] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1725] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1726] System Overview

[1727] This system allows elderly people who are unable to meet with family and friends due to an infectious disease pandemic to enjoy simulated conversations. The system collects user voice data, analyzes it to generate a generative artificial intelligence model, and installs the model on the elderly person's device to realize simulated conversations.

[1728] Program Overview

[1729] The program performs the following specific processing.

[1730] User data collection phase

[1731] 1. The user launches the app on their smartphone and speaks into the microphone.

[1732] For example: "Hi, Mom. How was your day?"

[1733] 2. The device collects the user's voice and saves it as voice data.

[1734] 3. The device uses a voice recognition system to convert the collected voice data into text data.

[1735] 4. The terminal sends the converted text data to the server.

[1736] Data analysis phase

[1737] 1. The server receives the text data sent from the terminal.

[1738] 2. The server analyzes the text data using a natural language processing (NLP) engine.

[1739] Specifically, it extracts sentence structure, speaking style, frequently used phrases, emotional expressions, etc.

[1740] 3. The server extracts and saves the user's characteristics from the analysis results.

[1741] Model generation phase

[1742] 1. The server trains a generative artificial intelligence model based on the extracted features.

[1743] This generative artificial intelligence model mimics specific speech and thought patterns.

[1744] 2. The server sends the trained and generated model to the device.

[1745] Model installation phase

[1746] 1. The terminal receives the generative artificial intelligence model sent from the server.

[1747] 2. The device installs the received model into the app.

[1748] 3. The terminal checks the operation of the installed model.

[1749] Conversation Simulation Phase

[1750] 1. The user (elderly person) launches the app and speaks into the microphone.

[1751] For example: "The weather is lovely today, did you go for a walk?"

[1752] 2. The device collects the user's voice and converts it into text data using a voice recognition system.

[1753] 3. The terminal inputs the converted text data into a generative artificial intelligence model.

[1754] 4. The device receives the response generated by the generative artificial intelligence model and outputs this response as speech.

[1755] For example: "Yes, Mom. The weather was nice today, so I went for a little walk."

[1756] Specific examples

[1757] For example, consider the case where a son uses this system for his mother who lives far away.

[1758] The user (son) talks to the app

[1759] The son says, "Hello, Mom. How was your day?" The app collects this speech and converts it into text using a speech recognition system. This text data is then sent to the server.

[1760] Server analysis and model generation

[1761] The server analyzes the text data and extracts the son's speaking style and thought patterns. Based on this, a generative AI model is trained to generate a model that reproduces the son's unique conversational style. This model is then sent to the device.

[1762] Model installation and conversation simulation

[1763] When a mother launches the app and says, "The weather is nice today. Did you go for a walk?", the device analyzes the speech and generates an appropriate response based on a generative artificial intelligence model. This response is output as a voice message saying, "Yes, Mom. The weather is nice today, so I went for a short walk."

[1764] This system allows elderly people to enjoy simulated conversations with family and friends, reducing feelings of loneliness and mental stress.

[1765] The processing flow will be explained below.

[1766] Step 1:

[1767] The user launches the app on their smartphone, and the app interface appears, switching to audio recording mode.

[1768] Step 2:

[1769] The user speaks into the microphone, for example, "Hi, Mom. How was your day?"

[1770] Step 3:

[1771] The device collects the user's voice and stores it as digital audio data.

[1772] Step 4:

[1773] The device sends the collected voice data to a voice recognition system, which converts it into text data. The voice recognition system analyzes the voice and generates the text, "Hello, Mom. How was your day?"

[1774] Step 5:

[1775] The terminal transmits the converted text data to the server, where the data is transferred using a secure communication protocol.

[1776] Step 6:

[1777] The server receives the text data sent from the device and stores it in a data store for analysis.

[1778] Step 7:

[1779] The server runs a natural language processing (NLP) engine to analyze the text data it receives, analyzing specific phrases, grammatical structures, emotional expressions, etc., and extracting the user's speaking style and thought patterns.

[1780] Step 8:

[1781] The server trains a generative artificial intelligence model based on the extracted features. The model uses machine learning algorithms to reflect the user's characteristics.

[1782] Step 9:

[1783] The server generates a trained generative AI model and sends it to the device. The generated model includes functions that can handle a variety of conversation scenarios.

[1784] Step 10:

[1785] The device receives the generative AI model sent from the server, and then installs the model into the app.

[1786] Step 11:

[1787] The operation of the generative AI model installed on the device is checked using a simple conversation scenario to verify whether the model generates accurate responses.

[1788] Step 12:

[1789] The user (elderly person) restarts the app and starts a conversation with the generative AI model, for example, saying, "The weather is nice today. Have you gone for a walk?"

[1790] Step 13:

[1791] The device collects the user's (elderly person's) voice and converts it into text data using a voice recognition system.

[1792] Step 14:

[1793] The device inputs the converted text data into a generative artificial intelligence model, which generates an appropriate response based on the input.

[1794] Step 15:

[1795] The device then plays back the generated response as speech, for example, "Yes, Mom. The weather was nice today, so I went for a short walk."

[1796] Through this series of steps, the system enables elderly people to enjoy simulated conversations with family and friends.

[1797] Example 1

[1798] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1799] Elderly people face the problem of feeling lonely and experiencing increased mental stress due to fewer opportunities to talk with family and friends due to infectious disease outbreaks and other factors. To solve this problem, a system that allows elderly people to enjoy simulated conversations is needed. Furthermore, an advanced, non-contact conversation system that can reproduce the user's speaking style and thought patterns is also needed.

[1800] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1801] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, means for transmitting the converted text data to the server, means for analyzing the text data and extracting user characteristics in the server, means for training a generative artificial intelligence model based on the extracted characteristics, means for transmitting the generated model to a terminal, means for installing the generative artificial intelligence model in the terminal, means for generating a response from the generative artificial intelligence model based on the user's voice input, means for outputting the generated response as voice, means for learning and saving the user's characteristic speaking style and thought patterns using the generative artificial intelligence model, and means for confirming the operation of the generative artificial intelligence model in the terminal. This allows elderly people to enjoy simulated conversations with family and friends, thereby reducing feelings of loneliness and mental burden.

[1802] "Voice data" refers to data that is a digital recording of a user's voice.

[1803] "Text data" refers to voice data converted into a string of characters using a voice recognition system.

[1804] A "server" is a computer system that processes and stores data and communicates with other computers over a network.

[1805] A "terminal" is a device operated by a user, such as a smartphone or tablet.

[1806] "Converting means" means the system or algorithm used to convert audio data into text data.

[1807] "Means of analysis" refers to software or algorithms that analyze text data and extract specific information, such as natural language processing engines.

[1808] "Means of feature extraction" refers to the technology that performs the process of recognizing and extracting certain patterns or features from data.

[1809] A "generative artificial intelligence model" is a machine learning model trained to mimic a user's characteristic speaking and thinking patterns.

[1810] "Means of learning" refers to techniques for using data to train generative artificial intelligence models.

[1811] "Means for installing" refers to the process of incorporating the generated artificial intelligence model into the software of the terminal.

[1812] A "response generator" is a system or algorithm that generates an appropriate reply based on user input.

[1813] The "means for outputting as voice" is a system that converts the generated text response into voice and outputs it.

[1814] "Means for operational verification" is a process for testing whether the installed artificial intelligence model is functioning correctly.

[1815] "Simulated conversation" refers to a simulated dialogue that is conducted in a format similar to a real conversation.

[1816] This invention is a system that collects user voice data, analyzes it to generate a generative artificial intelligence model that imitates specific speaking styles and thought patterns, and installs the model on a terminal to realize simulated conversations.

[1817] This system mainly consists of devices such as smartphones and tablets, a server responsible for data analysis, a voice recognition system, a natural language processing engine, and a generative artificial intelligence model.

[1818] User data collection

[1819] First, the user opens the app on their smartphone and speaks into the microphone. For example, you can use the following prompt:

[1820] "Hi, Mom. How was your day?"

[1821] The device collects this voice and stores it in local storage as audio data. The collected voice data is converted into text data using a speech recognition system such as Google Cloud Speech-to-Text or IBM Watson Speech to Text. The converted text data is then sent to the server as an HTTP POST request.

[1822] Data analysis and model generation on the server

[1823] The server receives the text data sent from the device and analyzes it using Python and a natural language processing engine such as NLTK or spaCy. The analysis includes analyzing sentence structure, extracting speaking characteristics, analyzing frequently used phrases, and analyzing emotional expressions. The server extracts user features from the analysis results and uses this feature data to train a generative artificial intelligence model (e.g., GPT-3). The generated model is then sent to the device.

[1824] Installing the model on the device and checking its operation

[1825] The device receives the generative AI model sent from the server and installs it in the app. The operation of the installed model is confirmed by generating test inputs on the device and verifying that the model's responses are accurate.

[1826] Conversation simulation for elderly people

[1827] Elderly people can launch the app and say things like:

[1828] "The weather is nice today, did you go for a walk?"

[1829] The device collects this speech and converts it into text data using a speech recognition system. The converted text data is input into a generative artificial intelligence model, and the response generated by the model is output as speech. For example, the following response may be returned:

[1830] "Yes, Mom. The weather was nice today, so I went for a little walk."

[1831] Using this system, elderly people can enjoy simulated conversations with family and friends, reducing feelings of loneliness and mental stress.

[1832] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1833] Step 1:

[1834] Data Collection Phase

[1835] 1. Start voice input

[1836] The user launches the app on their smartphone and speaks into the microphone.

[1837] Input: What the user says (e.g., "Hi, Mom. How was your day?")

[1838] Specific actions: Tap the voice input button in the app to start recording.

[1839] Output: Recorded audio data

[1840] 2. Saving audio data

[1841] The terminal collects the user's voice and stores it as voice data in local storage.

[1842] Input: Recorded audio data

[1843] Specific operation: When recording is finished, the audio data will be saved in WAV format to the device's local storage.

[1844] Output: Saved audio file

[1845] Step 2:

[1846] Data Transformation Phase

[1847] 3. Convert voice to text data

[1848] The terminal converts the collected voice data into text data using a voice recognition system.

[1849] Input: Saved audio file

[1850] What happens: The audio file is sent to a speech recognition API, such as Google Cloud Speech-to-Text, and converted to text.

[1851] Output: Converted text data

[1852] Step 3:

[1853] Data transmission phase

[1854] 4. Sending text data to the server

[1855] The terminal transmits the converted text data to the server.

[1856] Input: Text data

[1857] Specific behavior: Sends text data to the server as an HTTP POST request.

[1858] Output: Text data sent to the server

[1859] Step 4:

[1860] Data analysis phase

[1861] 5. Receiving text data

[1862] The server receives the text data sent from the terminal.

[1863] Input: Text data sent from the terminal

[1864] Specific behavior: The server accepts an HTTP POST request at an API endpoint and retrieves text data.

[1865] Output: Text data received on the server

[1866] 6. Analysis of Text Data

[1867] The server analyzes the text data using a natural language processing (NLP) engine.

[1868] Input: Text data received on the server

[1869] Specific operation: Text data is analyzed using Python, NLTK, spaCy, etc. to extract sentence structure, speaking style, frequently used phrases, emotional expressions, etc.

[1870] Output: Parsed data and feature information

[1871] Step 5:

[1872] Model generation phase

[1873] 7. Training generative AI models

[1874] The server trains a generative artificial intelligence model based on the extracted features.

[1875] Input: Analyzed data and feature information

[1876] Specific operation: Train a generative artificial intelligence model (e.g., GPT-3) using feature data.

[1877] Output: A trained generative artificial intelligence model

[1878] 8. Submitting the generative AI model

[1879] The server sends the generated model to the terminal.

[1880] Input: A trained generative artificial intelligence model

[1881] Specific operation: The model file is sent to the terminal as the payload of the HTTP response.

[1882] Output: Generative AI model sent to device

[1883] Step 6:

[1884] Model Install Phase

[1885] 9. Receiving the generative AI model

[1886] The terminal receives the generative artificial intelligence model sent from the server.

[1887] Input: Generative AI model received from the server

[1888] Specific behavior: Saves the model file to local storage.

[1889] Output: Generative AI model stored in internal storage

[1890] 10. Installing the generative AI model

[1891] The device installs the received generative artificial intelligence model into the app.

[1892] Input: Generative AI model stored in local storage

[1893] What happens: The model file is integrated into the app's internal database.

[1894] Output: Installed generative artificial intelligence model

[1895] 11. Checking the operation of the generative AI model

[1896] The terminal checks the operation of the installed model.

[1897] Input: Installed generative artificial intelligence model

[1898] What it does: Generate test inputs and verify that the model responds appropriately.

[1899] Output: A generative AI model that has been verified to work

[1900] Step 7:

[1901] Conversation Simulation Phase

[1902] 12. App launch and voice input

[1903] The user (elderly person) launches the app and speaks into the microphone.

[1904] Input: User utterance (e.g., "The weather is nice today, did you go for a walk?")

[1905] Specific actions: Tap the voice input button in the app to start recording.

[1906] Output: Recorded audio data

[1907] 13. Converting voice to text data

[1908] The terminal collects the user's voice and converts it into text data using a voice recognition system.

[1909] Input: Recorded audio data

[1910] What happens: The audio file is sent to a speech recognition API and converted to text.

[1911] Output: Converted text data

[1912] 14. Inputting text data into a generative AI model

[1913] The terminal inputs the converted text data into a generative artificial intelligence model.

[1914] Input: Converted text data

[1915] Specific behavior: Text data is passed to a model in the app.

[1916] Output: The response generated by the model

[1917] 15. Response generation and speech output

[1918] The terminal outputs the response generated by the generative artificial intelligence model as voice.

[1919] Input: The response generated by the model

[1920] Specific behavior: The response is converted into audio through a speech synthesis engine (e.g., Google Text-to-Speech) and played back to the user.

[1921] Output: A voice response the user hears (e.g., "Hi, Mom. The weather was nice today, so I went for a little walk.")

[1922] (Application example 1)

[1923] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1924] When elderly people live far from family and friends, they may feel lonely and have fewer opportunities to talk, which can increase their mental burden. Furthermore, even when using food delivery services, the service often ends up being limited to simply handing over the goods, with no real human interaction. The present invention aims to solve these problems and provide a system that allows elderly people to enjoy more fulfilling communication.

[1925] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1926] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, and means for transmitting the converted text data to the server, thereby enabling the elderly person and the virtual conversation host to enjoy natural conversation during food delivery.

[1927] "User voice data" means voice signals collected when a user speaks.

[1928] "Means of collection" refers to the method by which devices such as smartphones and tablets capture audio via microphones and then convert it into data.

[1929] "Means for converting into text data" refers to technology that uses a voice recognition system to convert voice data into text information.

[1930] "Means for sending to a server" refers to a method for sending text data to a cloud server or a remote computer system via the Internet.

[1931] "Means for analyzing text data" refers to technology that uses a natural language processing engine to analyze text data and extract conversation characteristics and emotions.

[1932] "Means for extracting user characteristics" refers to a method of analyzing speaking style, frequently used phrases, thought patterns, etc., to extract information specific to the user.

[1933] "Means for learning generative artificial intelligence models" refers to a method of using machine learning algorithms to create complex generative AI models based on user feature data.

[1934] "Means for transmitting the generated model to the terminal" refers to a method for transferring the generated AI model from the cloud to the user's device.

[1935] "Means for installing a generative artificial intelligence model on a terminal" refers to a method for installing a received AI model on a device and making it usable.

[1936] "Means for generating responses from a generative artificial intelligence model" refers to a technique that uses an AI model to generate responses to user input.

[1937] "Means for outputting a response as voice" refers to a technology that converts the generated text response into voice using a voice synthesis system and plays it aloud to the user.

[1938] "A means for a conversational agent to converse with a user on behalf of a delivery person when delivering a package" is a way for a conversational AI agent to interact with a user when delivering food or other deliveries.

[1939] System Overview

[1940] This system allows elderly people to enjoy conversations with food delivery services by having a conversation agent instead of the delivery person, thereby reducing feelings of loneliness. The system mainly consists of a server, a terminal, and a user.

[1941] Program Overview

[1942] User data collection phase

[1943] A user launches an app on their smartphone and speaks into the microphone, for example, saying something like: "Hello, Mom. How was your day?" The device collects the spoken voice and converts it into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text). The text data is then sent to the server.

[1944] Data analysis phase

[1945] The server receives the text data and analyzes it using a natural language processing (NLP) engine (e.g., OpenAI GPT-3). This analysis includes speaking style, frequently used phrases, emotional expressions, etc. The analysis results are extracted and saved as user characteristics.

[1946] Model generation phase

[1947] The server trains a generative artificial intelligence model based on the extracted features, which mimics specific speech and thought patterns, and sends the trained model to the device.

[1948] Model installation phase

[1949] The device receives the generative AI model sent from the server and installs it into the app. After installation, operation is checked.

[1950] Conversation Simulation Phase

[1951] When food is delivered, the user (elderly person) can launch the app and enjoy a conversation with the conversational agent. For example, if the user asks, "What kind of food has arrived today?", the device collects the speech and converts it into text data using a speech recognition system. The converted text data is input into a generative artificial intelligence model, which generates an appropriate response. The generated response is output as a voice such as, "Today's menu is an Italian set! It includes delicious pasta."

[1952] Hardware and software used

[1953] Hardware: smartphone, tablet, microphone

[1954] Software: Google Cloud Speech-to-Text API, OpenAI GPT-3, PyTorch, Hugging Face Transformers

[1955] Prompt Sentence Examples

[1956] The following is an example of a prompt statement to pass user input to the model:

[1957] User: Hello, what food arrived today?

[1958] Prompt the model:

[1959] "The user asks what kind of food has arrived. Example response: 'It's the Italian set today! It includes delicious pasta.'"

[1960] This system allows elderly people to enjoy conversation during food delivery, reducing the mental burden.

[1961] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1962] Step 1:

[1963] A user launches the app on their smartphone and speaks into the microphone, for example, "Hi, Mom. How was your day?"

[1964] Input: User's voice

[1965] Output: Audio data

[1966] How it works: Your smartphone's microphone picks up your voice, which is then collected by the app as digital audio data.

[1967] Step 2:

[1968] The device converts the collected voice data into text data using a voice recognition system (e.g., Google Cloud Speech-to-Text).

[1969] Input: Audio data

[1970] Output: Text data

[1971] How it works: Voice data is sent to a cloud-based voice recognition API, where it is processed and converted into text data in string format.

[1972] Step 3:

[1973] The terminal transmits the converted text data to the server.

[1974] Input: Text data

[1975] Output: Sends text data to the server

[1976] Specific operation: Text data is uploaded to a server via the Internet.

[1977] Step 4:

[1978] The server receives the text data and analyzes it using a natural language processing engine (e.g., OpenAI GPT-3).

[1979] Input: Text data

[1980] Output: Parsed user features

[1981] Specific operation: The server analyzes the text data and extracts characteristics such as speaking style, frequently used phrases, and emotional expressions.

[1982] Step 5:

[1983] The server trains a generative artificial intelligence model based on the extracted features.

[1984] Input: User characteristics

[1985] Output: A trained generative artificial intelligence model

[1986] How it works: A machine learning algorithm is run using the user's feature data to generate a customized generative AI model.

[1987] Step 6:

[1988] The server transmits the trained model to the terminal.

[1989] Input: A trained generative artificial intelligence model

[1990] Output: Sending the AI ​​model to the device

[1991] Specific operation: The generated AI model is sent to the device via the network.

[1992] Step 7:

[1993] The device receives the generative artificial intelligence model sent from the server and installs it within the app.

[1994] Input: Generative AI model

[1995] Output: Installed AI model

[1996] Specific operation: The received AI model is saved in the app and its operation is confirmed.

[1997] Step 8:

[1998] When a user (elderly person) receives a food delivery, they launch the app and speak into the microphone, for example, "What kind of food has arrived today?"

[1999] Input: User's voice

[2000] Output: Audio data

[2001] How it works: Your smartphone's microphone picks up your voice, which is then collected by the app as digital audio data.

[2002] Step 9:

[2003] The terminal converts the collected voice data into text data using a voice recognition system.

[2004] Input: Audio data

[2005] Output: Text data

[2006] What it does: The speech recognition system analyzes the audio data and converts it into text.

[2007] Step 10:

[2008] The terminal inputs the converted text data into a generative artificial intelligence model to generate an appropriate response.

[2009] Input: Text data

[2010] Output: Response text

[2011] Specific operation: The generative artificial intelligence model generates a response based on the input text data.

[2012] Step 11:

[2013] The device outputs the generated response as voice using a speech synthesis system (e.g., Google Cloud Text-to-Speech API).

[2014] Input: Response text

[2015] Output: Response voice

[2016] Specific operation: Text data is converted into audio data and output to the user through the speaker.

[2017] This will enable elderly people to enjoy fun conversations with a conversational agent during food delivery.

[2018] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2019] System Overview

[2020] The purpose of this invention is to enable elderly people in nursing homes to enjoy simulated conversations with family and friends when they are unable to meet in person. In particular, this invention analyzes the user's voice data and simulates conversations using a generative artificial intelligence model that learns the user's speaking style and thought patterns. In addition, by combining it with an emotion recognition engine, it is possible to recognize the user's emotions and reflect them in the content of the responses.

[2021] Program Overview

[2022] The program performs the following processing:

[2023] User data collection phase

[2024] 1. The user launches the app and speaks into the microphone.

[2025] For example: "Hi, Mom. How was your day?"

[2026] 2. The device collects the user's voice and saves it as voice data.

[2027] 3. The voice data collected by the device is converted into text data using a voice recognition system.

[2028] 4. The device sends the converted text data to the server.

[2029] Data analysis phase

[2030] 1. The server receives the text data sent from the terminal.

[2031] 2. The server uses a natural language processing (NLP) engine to analyze the text data and extract speech and thought patterns.

[2032] 3. The server uses an emotion engine to recognize the user's emotion from the voice data and adds the emotion information to the text data.

[2033] Model generation phase

[2034] 1. The server trains a generative AI model based on the features and emotional information extracted by the server.

[2035] 2. The server generates a trained generative artificial intelligence model and sends it to the terminal.

[2036] Model installation phase

[2037] 1. The device receives the generative AI model and installs it in the app.

[2038] 2. The device will check that the new model works.

[2039] Conversation Simulation Phase

[2040] 1. The user (elderly person) restarts the app and starts talking.

[2041] For example: "The weather is lovely today. Did you go for a walk?"

[2042] 2. The device collects the voice and converts it into text data using a voice recognition system.

[2043] 3. The device inputs this text data and emotional information into a generative artificial intelligence model.

[2044] 4. The device outputs the generated response to the user as speech.

[2045] For example: "Yes, Mom. The weather was nice today, so I went for a little walk."

[2046] Specific examples

[2047] For example, consider the case where a son uses this system for his mother who lives far away.

[2048] The user (son) talks to the app

[2049] The son says, "Hi, Mom. How was your day?" The app collects this speech and converts it into text using a speech recognition system. The converted text and voice data are then sent to the server.

[2050] Server analysis and model generation

[2051] The server analyzes the text data and extracts the son's speech patterns and thought patterns. The emotion engine also analyzes the voice data and recognizes the son's emotions. A generative AI model is trained based on the analysis results, and a model reflecting the son's speech patterns and emotions is generated. This model is then sent to the device.

[2052] Model installation and conversation simulation

[2053] Mom launches the app and says, "The weather is nice today. Did you go for a walk?" The device collects this voice and converts it into text data using a speech recognition system. The emotion engine then recognizes emotions from the input voice data. By inputting the text data and emotional information into a generative artificial intelligence model, an appropriate response is generated. This response is output as voice and played back as, "Yes, Mom. The weather is nice today, so I went for a short walk."

[2054] This system provides simulated conversations that take emotions into account, helping to alleviate feelings of loneliness and mental stress among the elderly and improving their quality of life.

[2055] The processing flow will be explained below.

[2056] Step 1:

[2057] The user launches the app on their smartphone, and the app interface appears, switching to audio recording mode.

[2058] Step 2:

[2059] The user speaks into the microphone, saying, "Hi, Mom. How was your day?"

[2060] Step 3:

[2061] The device collects the user's voice and stores it as digital audio data.

[2062] Step 4:

[2063] The device sends the collected voice data to the emotion engine, which recognizes the user's emotions from the voice. The emotion engine analyzes the emotions and saves the results as data.

[2064] Step 5:

[2065] The device sends the voice data, with the output of the emotion engine added, to the voice recognition system, which converts it into text data. The voice recognition system analyzes the voice and generates the text, "Hello, Mom. How was your day?"

[2066] Step 6:

[2067] The device then transmits the converted text data and emotion information to the server, where the data is transferred using a secure communication protocol.

[2068] Step 7:

[2069] The server receives the text data and emotion information sent from the device, and stores the received data in a data store for analysis.

[2070] Step 8:

[2071] The text data received by the server is analyzed using a natural language processing (NLP) engine to extract speaking style, frequently used phrases, emotional expressions, etc.

[2072] Step 9:

[2073] The server extracts the user's characteristics and emotional patterns from the analysis results and emotional information, and trains a generative AI model that uses machine learning algorithms to reflect the user's characteristics and emotions.

[2074] Step 10:

[2075] The server generates a trained generative AI model and sends it to the device. The generated model includes functions that can handle a variety of conversation scenarios.

[2076] Step 11:

[2077] The device receives the generative AI model sent from the server, and then installs the model into the app.

[2078] Step 12:

[2079] The operation of the generative AI model installed on the device is checked using a simple conversation scenario to verify whether the model generates accurate responses.

[2080] Step 13:

[2081] The user (an elderly person) restarts the app and starts a conversation with the generative AI model, saying, "The weather is nice today. Have you gone for a walk?"

[2082] Step 14:

[2083] The terminal collects the user's (elderly person's) voice, sends it to an emotion engine, adds emotional information, and then converts it into text data using a voice recognition system.

[2084] Step 15:

[2085] The device inputs the converted text data and emotion information into a generative artificial intelligence model, which generates an appropriate response based on this input.

[2086] Step 16:

[2087] The device will then output the generated response as speech, which will be "Yes, Mom. The weather was nice today, so I went for a short walk."

[2088] As a result, the system allows elderly people to enjoy simulated conversations with family and friends, and by adding emotion recognition functionality, it can provide more realistic and emotionally appropriate responses.

[2089] Example 2

[2090] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2091] In situations where it is difficult for elderly people to communicate directly with family and friends in nursing homes, there is a need to reduce their feelings of loneliness and mental burden and improve their quality of life through simulated conversations. However, conventional systems have difficulty in creating conversations that fully reflect the user's speaking style, thought patterns, and emotions, and have not been able to achieve simulated interactions that are close to real conversations.

[2092] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2093] In this invention, the server includes means for collecting user voice data, means for converting the collected voice data into text data, means for transmitting the converted text data to the server, means for analyzing the text data and extracting user characteristics in the server, means for recognizing the user's emotions from the extracted characteristics and voice data and adding emotional information, means for training a generative AI model based on the extracted characteristics and emotional information, means for transmitting the trained generative AI model to a terminal, means for installing the generative AI model in the terminal, means for generating a response from the generative AI model based on a user's input, and means for outputting the generated response as voice. This makes it possible to provide a more natural and emotional simulated conversation that reflects the user's speaking style, thought patterns, and emotions.

[2094] "Users" refer to individuals who use this system to provide voice data, including seniors, their families, and friends.

[2095] "Voice data" refers to data that is a digital recording of the user's speech.

[2096] "Text data" refers to data obtained by converting voice data into character information.

[2097] "Server" refers to a remotely located computer system that receives, analyzes, generates models, and performs other processing.

[2098] "Features" refers to specific information about an utterance, such as the user's speaking style, thought patterns, and emotions.

[2099] "Emotion information" refers to information indicating emotions recognized from the user's voice data.

[2100] A "generative artificial intelligence model" refers to an artificial intelligence system that learns based on the user's characteristics and emotional information.

[2101] "Device" refers to a device directly operated by a user that stores collected voice data, converts it to text data, installs generative AI models, etc. Examples include smartphones and tablets.

[2102] "Response" refers to the content of the utterance generated by the generative artificial intelligence model based on the user's input.

[2103] A "voice recognition system" refers to a technology that converts voice data into text data.

[2104] A "voice synthesis system" refers to a technology that outputs generated text data as voice.

[2105] MODE FOR CARRYING OUT THE INVENTION

[2106] This invention is a system that provides simulated conversations and psychological support to elderly people, particularly those living in nursing homes where it is difficult to converse directly with family and friends. This system uses technology that collects user voice data, trains a generative artificial intelligence model based on the user's speaking style and thought patterns, and generates responses that reflect emotional information.

[2107] Specifically, the system operates as follows.

[2108] User data collection phase

[2109] A user launches an application on a device such as a smartphone or tablet and speaks into the microphone. For example, they might say, "Hello, Mom. How was your day?" The device collects and stores this speech. The hardware used could be a smartphone or tablet, and the software could be a voice recording application.

[2110] Next, the device converts the collected voice data into text data using a voice recognition system (such as Google Speech-to-Text API or IBM Watson Speech-to-Text), and then transmits the converted text data to a server via HTTP / HTTPS protocol.

[2111] Data analysis phase

[2112] The server receives the text data sent from the device and analyzes it using a natural language processing (NLP) engine (such as NLTK or spaCy) to extract the user's speaking style and thought patterns.

[2113] Next, the server uses an emotion engine (e.g., IBM Watson Tone Analyzer or Azure Cognitive Services) to recognize the user's emotion from the voice data and add the emotion information to the text data.

[2114] Model generation phase

[2115] Based on the results of the analysis, the server trains a generative AI model using the extracted features and emotion information. This process uses a machine learning framework (e.g., TensorFlow or PyTorch). The trained generative AI model is output as a file and sent back to the device via HTTP / HTTPS.

[2116] Model installation phase

[2117] The device installs the generative AI model received from the server into the application. This procedure includes application update functionality and an automatic installation process. The device also runs test scripts to verify the model's correct operation.

[2118] Conversation Simulation Phase

[2119] The user (e.g., an elderly person) launches the app again and says something like, "The weather is nice today. Did you go for a walk?" The device collects the audio and converts it into text again using a speech recognition system. The emotion engine also recognizes emotions from the input voice data. Finally, the device inputs the text data and emotion information into a generative artificial intelligence model to generate an appropriate response. This response is output as audio through a speech synthesis system (e.g., Google Text-to-Speech or Amazon Polly), and is played back as, "Yes, Mom. The weather is nice today, so I went for a short walk."

[2120] This system provides simulated conversations that take emotions into consideration, making it possible to reduce feelings of loneliness and mental stress, particularly among the elderly, and improve their quality of life.

[2121] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2122] Step 1:

[2123] The user launches the app and speaks into the microphone.

[2124] Input: Tap the app from the home screen of your smartphone or tablet to launch it. What you say (e.g., "Hi, Mom. How was your day?")

[2125] Specific behavior: The user taps the application to launch it and speaks into the microphone.

[2126] Output: Collected audio data

[2127] Step 2:

[2128] The device collects the user's voice and stores it as voice data.

[2129] Input: User speech, audio collected through a microphone

[2130] Specific operation: An application on the device activates the microphone, records and saves the user's voice.

[2131] Output: Saved audio data

[2132] Step 3:

[2133] The voice data collected by the terminal is converted into text data using a voice recognition system.

[2134] Input: Saved audio data

[2135] Specific operation: Calls the Google Speech-to-Text API or IBM Watson Speech-to-Text to convert voice data into text data.

[2136] Output: Converted text data

[2137] Step 4:

[2138] The terminal transmits the converted text data to the server.

[2139] Input: Converted text data

[2140] What it does: Sends data over an internet connection using HTTP / HTTPS protocols.

[2141] Output: Text data sent to the server

[2142] Step 5:

[2143] The server receives the text data sent from the terminal.

[2144] Input: Text data sent via HTTP / HTTPS protocol

[2145] What happens: The server receives the HTTP request and stores the data for analysis.

[2146] Output: Saved text data

[2147] Step 6:

[2148] The server uses a natural language processing (NLP) engine to analyze the text data and extract user characteristics.

[2149] Input: Saved text data

[2150] Specific operation: Analyzes text data using NLTK and spaCy to extract the user's speaking style and thought patterns.

[2151] Output: Extracted feature data

[2152] Step 7:

[2153] The server uses an emotion engine to recognize the user's emotion from the voice data and adds the emotion information to the text data.

[2154] Input: Audio data and text data

[2155] Specific operation: Emotions are analyzed using IBM Watson Tone Analyzer and Azure Cognitive Services, and emotional information is added to text data.

[2156] Output: Text data with emotional information added

[2157] Step 8:

[2158] The generative artificial intelligence model is trained based on the features and emotional information extracted by the server.

[2159] Input: Text data with extracted feature data and emotional information

[2160] Specific operations: Train generative artificial intelligence models using TensorFlow and PyTorch.

[2161] Output: A trained generative artificial intelligence model

[2162] Step 9:

[2163] The server generates a trained generative artificial intelligence model and transmits it to the terminal.

[2164] Input: A trained generative artificial intelligence model

[2165] Specific operation: The generated model is output in file format and sent to the terminal via HTTP / HTTPS protocol.

[2166] Output: The generative artificial intelligence model sent

[2167] Step 10:

[2168] The device receives the generative artificial intelligence model and installs it in the app.

[2169] Input: Generative AI model sent from the server

[2170] Specific operation: Install the model using the application's automatic update function.

[2171] Output: Installed generative artificial intelligence model

[2172] Step 11:

[2173] The device will check the operation of the new model.

[2174] Input: Installed generative artificial intelligence model

[2175] Specific behavior: Run test scripts to verify correct behavior of the model.

[2176] Output: Operation check result

[2177] Step 12:

[2178] The user (elderly person) restarts the app and speaks.

[2179] Input: Restart the app and what you say (e.g., "The weather is nice today. Have you gone for a walk?")

[2180] Specific behavior: The user launches the app and speaks into the microphone.

[2181] Output: Collected audio data

[2182] Step 13:

[2183] The device collects the voice and converts it into text data using a voice recognition system.

[2184] Input: Collected audio data

[2185] Specific operation: Converts into text data using Google Speech-to-Text API or IBM Watson Speech-to-Text.

[2186] Output: Converted text data

[2187] Step 14:

[2188] The device inputs text data and emotional information into a generative artificial intelligence model.

[2189] Input: Text data and emotion information

[2190] Specific operation: Text data and emotional information are input into a generative artificial intelligence model to generate a response.

[2191] Output: The generated response

[2192] Step 15:

[2193] The terminal outputs the generated response as speech to the user.

[2194] Input: Generated response text

[2195] Specific operation: Speech is synthesized using Google Text-to-Speech or Amazon Polly and output to the speaker.

[2196] Output: The response played as speech (e.g., "Yes, Mom. The weather was nice today, so I went for a short walk.")

[2197] These processing steps provide a simulated conversation that takes emotions into consideration, helping to alleviate feelings of loneliness and mental stress among the elderly and improving their quality of life.

[2198] (Application example 2)

[2199] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2200] One of the problems facing the elderly in modern society is loneliness and mental stress. In particular, there are many situations in which elderly people need support in physical stores, but they may not receive appropriate assistance. Furthermore, as opportunities for elderly people to enjoy face-to-face conversations with family and friends decrease, their quality of life may decline. To address these issues, a system is needed that can reduce feelings of loneliness for elderly people through simulated conversations and provide necessary guidance and support in physical stores.

[2201] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's voice data and extracting the user's characteristics and emotional information, means for training a generative AI model based on the extracted characteristics and emotional information, and means for transmitting the generated model to the interface device. This makes it possible to provide the elderly with simulated conversations with family or friends using a generative AI model that reproduces the user's speaking style, thought patterns, and emotions, and to further guide and support the elderly in physical stores.

[2202] "User's voice data" refers to information that is orally uttered by a user, and the content of that information is recorded as an acoustic signal.

[2203] "Text data" refers to data expressed as a string of characters based on voice data, and is the subject of natural language processing.

[2204] A "remote server" refers to a computer device installed in a remote location that is connected via a network such as the Internet, and that processes and stores data.

[2205] "Characteristic and emotional information" refers to information that indicates speaking style, thought patterns, and emotions extracted from voice data.

[2206] A "generative artificial intelligence model" refers to an artificial intelligence that generates responses to input data based on a trained algorithm.

[2207] An "interface device" refers to a device that allows a user and a system to exchange data with each other, such as a robot or a smart device.

[2208] "Response" refers to the reply or reaction that a generative artificial intelligence model generates based on user input.

[2209] A "physical store" refers to a commercial establishment located in a physical location where goods and services are offered.

[2210] System Overview

[2211] The system of the present invention allows elderly people to reduce feelings of loneliness and receive guidance and support through simulated conversations with family and friends in brick-and-mortar stores. The system collects and analyzes user voice data, and uses the results to train a generative artificial intelligence model to provide appropriate responses.

[2212] Hardware and software used

[2213] Hardware:

[2214] Remote Server: A powerful computing device that processes and stores data.

[2215] Interface devices: These include robots and smart devices used by the elderly. For example, robots are equipped with microphones, speakers, and voice recognition systems.

[2216] software:

[2217] Speech recognition systems (e.g., Google Speech-to-Text API)

[2218] Natural language processing engines (e.g., Google NLP, Microsoft Azure NLP)

[2219] Emotion recognition engine (e.g. IBM Watson Tone Analyzer)

[2220] Data processing and calculation

[2221] The server uses a speech recognition system to collect the user's voice data and convert it into text data. The text data is analyzed by the server to extract the user's characteristics and emotional information. In particular, a natural language processing engine is used to analyze the grammar and syntax of the text data, and an emotion recognition engine is used to add emotional information.

[2222] A generative AI model is trained based on the analyzed data. The trained model is sent to the interface device, where it is installed and tested. Finally, when the user speaks to the interface device, their speech is again collected and converted into text data, and the generative AI model generates an appropriate response.

[2223] Specific examples

[2224] For example, an elderly person in a physical store might speak to an interface device (robot) and say, "I'm not sure which product to buy today. What do you recommend?" This voice data is collected in real time and converted into text data using a voice recognition system. The text data is sent to a remote server and analyzed by a natural language processing engine and an emotion recognition engine. Based on the analysis results, a generative artificial intelligence model generates a response, and the robot responds aloud with, "That's a good choice. It's especially popular with the elderly."

[2225] Prompt Sentence Examples

[2226] We are building a simulated conversation system for elderly people. We need to analyze the user's voice data and provide conversations that reflect their speaking style and emotions. Please talk to them about the following topics:

[2227] 1. I'm not sure which product to buy today, what do you recommend?

[2228] 2. What do you think of this product?

[2229] Consider emotional information (e.g., happy, sad, confused).

[2230] In this way, seniors can receive the guidance and support they need even when they are in a physical store, which is expected to reduce feelings of loneliness and improve their quality of life.

[2231] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2232] Step 1:

[2233] The user speaks to the robot, which is an interface device. For example, they input a voice message such as, "I'm not sure which product to buy today. What do you recommend?" This voice message is sent to the robot via a microphone.

[2234] Input: User's voice data

[2235] Output: Audio data collected by the robot

[2236] Specific operation: The robot's microphone picks up the user's speech and saves it as audio data.

[2237] Step 2:

[2238] The robot converts the collected voice data into text data using a voice recognition system, utilizing voice recognition software such as the Google Speech-to-Text API.

[2239] Input: Collected audio data

[2240] Output: Text data converted by the speech recognition system

[2241] Specific operation: Audio data is sent to the server and converted into text data using the Google Speech-to-Text API.

[2242] Step 3:

[2243] The robot sends the converted text data to a remote server, which receives the data.

[2244] Input: Text data

[2245] Output: Text data sent to the remote server

[2246] Specific operation: The robot uploads text data to a remote server via the network.

[2247] Step 4:

[2248] The server analyzes the received text data and extracts user characteristics using a natural language processing (NLP) engine. It also uses an emotion recognition engine to obtain emotional information and add it to the text data.

[2249] Input: Text data, emotion information

[2250] Output: Analysis data with user characteristics and emotional information added

[2251] Specific operation: The grammar and syntax of text data is analyzed using Google NLP and Microsoft Azure NLP, and emotions are recognized using IBM Watson Tone Analyzer.

[2252] Step 5:

[2253] The server trains a generative artificial intelligence model based on the analyzed features and emotion information. After the model is trained, the remote server transmits the model to the interface device.

[2254] Input: Analysis data

[2255] Output: A trained generative artificial intelligence model

[2256] Specific operation: Learning is performed using the analysis data, a model is generated, and it is deployed to the interface device.

[2257] Step 6:

[2258] The robot, which acts as an interface device, receives the learned generative AI model and installs it into the application, while simultaneously verifying its operation.

[2259] Input: A trained generative artificial intelligence model

[2260] Output: Installed generative artificial intelligence model

[2261] Specific operation: The robot downloads the model from the remote server, installs it in the app, and checks its operation.

[2262] Step 7:

[2263] When the user speaks to the interface device again, the voice data is collected again and converted into text data by the voice recognition system. The text data and emotion information are input into the generative artificial intelligence model.

[2264] Input: User's voice data, converted text data

[2265] Output: Data input into the generative artificial intelligence model

[2266] How it works: The robot collects the audio again, converts it into text data using the Google Speech-to-Text API, and inputs it into the model.

[2267] Step 8:

[2268] The robot generates an appropriate response from a generative artificial intelligence model and outputs the response to the user as voice.

[2269] Input: Text data input to the generative artificial intelligence model

[2270] Output: The response output as audio

[2271] Specific operation: The generative artificial intelligence model generates an appropriate response and replies to the user audibly through the robot's speaker.

[2272] This step will provide seniors with guidance and support in physical stores, reducing feelings of loneliness and improving their quality of life.

[2273] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2274] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2275] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2276] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2277] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2278] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2279] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2280] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2281] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2282] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2283] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2284] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2285] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2286] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2287] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2288] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2289] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2290] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2291] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2292] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2293] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2294] The following is further disclosed regarding the above embodiment.

[2295] (Claim 1)

[2296] means for collecting user voice data;

[2297] A means for converting the collected voice data into text data;

[2298] means for transmitting the converted text data to a server;

[2299] A means for analyzing the text data in the server and extracting user characteristics;

[2300] means for training a generative artificial intelligence model based on the extracted features;

[2301] A means for transmitting the generated model to a terminal;

[2302] a means for installing a generative artificial intelligence model on the terminal;

[2303] means for generating a response from a generative artificial intelligence model based on a user input;

[2304] means for outputting the generated response as speech;

[2305] A system including:

[2306] (Claim 2)

[2307] 2. The system of claim 1, wherein the generative artificial intelligence model reproduces the user's speaking style and thought patterns.

[2308] (Claim 3)

[2309] 2. The system according to claim 1, wherein the terminal provides the elderly user with a simulated conversation with a family member or friend.

[2310] "Example 1"

[2311] (Claim 1)

[2312] means for collecting user voice data;

[2313] A means for converting the collected voice data into text data;

[2314] means for transmitting the converted text data to a server;

[2315] A means for analyzing the text data in the server and extracting user characteristics;

[2316] means for training a generative artificial intelligence model based on the extracted features;

[2317] A means for transmitting the generated model to a terminal;

[2318] a means for installing a generative artificial intelligence model on the terminal;

[2319] means for generating a response from a generative artificial intelligence model based on a user's voice input;

[2320] means for outputting the generated response as speech;

[2321] A means for learning and storing a user's characteristic speaking style and thought patterns using a generative artificial intelligence model;

[2322] A means for verifying the operation of the generative artificial intelligence model on the terminal;

[2323] A system including:

[2324] (Claim 2)

[2325] 2. The system of claim 1, wherein the generative artificial intelligence model reproduces the user's speaking style and thought patterns.

[2326] (Claim 3)

[2327] 2. The system according to claim 1, wherein the terminal provides the elderly user with a simulated conversation with a family member or friend.

[2328] "Application Example 1"

[2329] (Claim 1)

[2330] means for collecting user voice data;

[2331] A means for converting the collected voice data into text data;

[2332] means for transmitting the converted text data to a server;

[2333] A means for analyzing the text data in the server and extracting user characteristics;

[2334] means for training a generative artificial intelligence model based on the extracted features;

[2335] A means for transmitting the generated model to a terminal;

[2336] a means for installing a generative artificial intelligence model on the terminal;

[2337] means for generating a response from a generative artificial intelligence model based on a user input;

[2338] means for outputting the generated response as speech;

[2339] A means for a conversation agent to converse with a user on behalf of a delivery person when delivering a delivery item;

[2340] A system including:

[2341] (Claim 2)

[2342] 2. The system of claim 1, wherein the generative artificial intelligence model reproduces the user's speaking style and thought patterns.

[2343] (Claim 3)

[2344] 2. The system according to claim 1, wherein the terminal provides the elderly user with a simulated conversation with a family member or friend.

[2345] "Example 2: Combining Emotion Engines"

[2346] (Claim 1)

[2347] means for collecting user voice data;

[2348] A means for converting the collected voice data into text data;

[2349] means for transmitting the converted text data to a server;

[2350] A means for analyzing the text data in the server and extracting user characteristics;

[2351] means for recognizing a user's emotion from the extracted features and voice data and adding emotion information;

[2352] a means for training a generative artificial intelligence model based on the extracted features and emotion information;

[2353] A means for transmitting the trained generative artificial intelligence model to a terminal;

[2354] a means for installing a generative artificial intelligence model on the terminal;

[2355] means for generating a response from a generative artificial intelligence model based on a user input;

[2356] means for outputting the generated response as speech;

[2357] A system including:

[2358] (Claim 2)

[2359] 2. The system of claim 1, wherein the generative artificial intelligence model reproduces the user's speech, thought patterns, and emotions.

[2360] (Claim 3)

[2361] 2. The system according to claim 1, wherein the terminal provides the elderly user with a simulated conversation with a family member or friend.

[2362] "Application example 2 when combining emotion engines"

[2363] (Claim 1)

[2364] means for collecting user voice data;

[2365] A means for converting the collected voice data into text data;

[2366] means for transmitting the converted text data to a remote server;

[2367] means for analyzing the text data in a remote server and extracting user characteristics and emotion information;

[2368] means for training a generative artificial intelligence model based on the extracted features and emotion information;

[2369] means for transmitting the generated model to an interface device;

[2370] means for installing a generative artificial intelligence model in the interface device;

[2371] means for generating a response from a generative artificial intelligence model based on a user input;

[2372] means for outputting the generated response as speech;

[2373] A system including:

[2374] (Claim 2)

[2375] 2. The system of claim 1, wherein the generative artificial intelligence model reproduces the user's speech, thought patterns, and emotions.

[2376] (Claim 3)

[2377] 2. The system according to claim 1, wherein the interface device provides the elderly user with a simulated conversation with a family member or friend, and provides guidance and assistance within a physical store. [Explanation of symbols]

[2378] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for collecting user voice data; A means for converting the collected voice data into text data; means for transmitting the converted text data to a server; A means for analyzing the text data in the server and extracting user characteristics; means for training a generative artificial intelligence model based on the extracted features; A means for transmitting the generated model to a terminal; a means for installing a generative artificial intelligence model on the terminal; means for generating a response from a generative artificial intelligence model based on a user input; means for outputting the generated response as speech; A system including:

2. 2. The system of claim 1, wherein the generative artificial intelligence model reproduces the user's speaking style and thought patterns.

3. 2. The system according to claim 1, wherein the terminal provides the elderly user with a simulated conversation with a family member or friend.

Images