System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting a user's interests and concerns; means for collecting a user's communication history; means for training a natural language processing model based on the collected interests and concerns and communication history; means for generating an optimized conversation using the generated natural language processing model; and means for analyzing a user's real-time reaction and updating the conversation based on the reaction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In couples and families living together, time and space differences due to work and lifestyles can lead to problems such as reduced communication and increased emotional distance. There is a need for methods to resolve these differences and achieve smoother communication and living arrangements. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including means for collecting user hobbies and interests, means for collecting user communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, and means for analyzing real-time user responses and updating the conversation based on the responses. This system can provide natural conversation as if users were in the same place, regardless of time or place, thereby eliminating time and space discrepancies in communal living.

[0006] "User" refers to an individual or group of individuals who uses the System.

[0007] "Hobbies and interests" refers to activities or topics that interest a user.

[0008] "Communication history" refers to a record of past messages and conversations between users.

[0009] "Natural language processing model" refers to a machine learning algorithm for analyzing and generating natural language.

[0010] "Training" refers to training a machine learning model based on collected data.

[0011] "Optimized conversation" refers to appropriate and natural dialogue content generated according to the user's situation and interests.

[0012] "Real-time reactions" refers to the immediate replies and emotional expressions that users give during a conversation.

[0013] "Analyzing" refers to analyzing collected data to extract useful information.

[0014] "Updating the conversation" refers to appropriately changing the conversation content based on the user's real-time reaction. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention relates to a system that uses generative AI to provide natural conversations in order to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[0037] Overall system overview

[0038] The system trains a natural language processing model based on user input such as hobbies, interests, and communication history, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses.

[0039] Data collection

[0040] 1. Collecting hobbies and interests

[0041] When a user registers for the service, they enter their hobbies and interests.

[0042] The terminal receives this input information and sends it to the server.

[0043] 2. Collection of communication history

[0044] With the user's consent, the server collects communication history and conversation history from messaging apps.

[0045] The server stores the collected data in a database.

[0046] Model training

[0047] The server trains a natural language processing model based on the user data collected.

[0048] The server iteratively trains the model with data to learn speech patterns optimized for individual users.

[0049] Conversation Generation

[0050] The user inputs a conversation topic or question into the terminal.

[0051] The device sends the input to the server.

[0052] The server uses the generative AI model to generate optimized conversations and sends the responses to the device.

[0053] The terminal displays the received reply to the user.

[0054] Real-time adjustments

[0055] 1. Real-time reaction analysis

[0056] Users react in real time to the generated conversation (e.g., by typing text or responding with voice).

[0057] The device receives real-time responses and transmits them to the server.

[0058] 2. Update the conversation

[0059] The server analyzes the user's real-time responses and sends feedback to the generative AI model.

[0060] The server updates the conversation content based on the feedback and sends it back to the device.

[0061] Specific examples

[0062] Travel plan example

[0063] 1. Input Phase

[0064] User A types into the terminal, "Where do you want to go on your next holiday?"

[0065] The device sends input to the server.

[0066] 2. Conversation generation phase

[0067] The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[0068] The terminal displays the generated conversation to User A.

[0069] 3. Real-time response

[0070] User A responds, "That's good, but I'm concerned about the budget."

[0071] The device sends this response to the server.

[0072] 4. Conversation Update Phase

[0073] The server analyzes the real-time responses and updates the suggestion with, "So, how about a trip to a hot spring in Hakone?"

[0074] The device displays the updated proposal to User A.

[0075] In this way, the system of the present invention can provide natural conversations that make people feel as if they are together in the same place, regardless of time or place, and can adjust to eliminate the sense of time and space discrepancies in communal living.

[0076] The processing flow will be explained below.

[0077] Step 1:

[0078] When users register for the service, they enter their hobbies and interests. The device receives this information and sends it to the server.

[0079] Step 2:

[0080] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[0081] Step 3:

[0082] The server trains a natural language processing model based on the user's hobbies, interests, and communication history collected by the server. The server repeatedly trains the model using data to learn conversation patterns optimized for each individual user.

[0083] Step 4:

[0084] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[0085] Step 5:

[0086] The server uses the generative AI model to generate an optimized conversation (e.g., "How about the trip to Hokkaido we talked about earlier?") and sends the response to the device. The device then displays the received response to the user.

[0087] Step 6:

[0088] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives the real-time response and sends it to the server.

[0089] Step 7:

[0090] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on the feedback.

[0091] Step 8:

[0092] The server sends the updated conversation (e.g., "So, how about a trip to a hot spring in Hakone?") to the device again. The device displays the updated proposal to the user.

[0093] In this way, users are provided with natural and optimized conversations regardless of time or place.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] In modern society, differences in time and space in communal living often lead to a decrease in communication and an increase in emotional distance, making it difficult for family and friends to interact smoothly and potentially leading to estrangement.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes means for inputting the user's hobbies and interests, means for collecting the user's communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and updating the conversation based on the reactions, and means for saving the collected data in a database. This makes it possible to provide natural conversation regardless of time or place, and to adjust so that time and space differences in communal living are not felt.

[0099] "User" refers to a person who uses the system.

[0100] "Hobbies and interests" refers to information about activities or topics that interest a user.

[0101] "Communication history" refers to a record of a user's past digital communications, such as messages, emails, and chats.

[0102] "Natural language processing model" refers to an algorithm or machine learning model for understanding and generating human language.

[0103] A "server" refers to a computer device that processes the entire system, stores and analyzes data, trains models, and so on.

[0104] "Terminal" refers to a hardware device that is directly operated by a user (e.g., smartphone, tablet, PC).

[0105] "Database" refers to a system for efficiently storing and managing collected data.

[0106] "Training a model" refers to the process of using collected data to refine and improve the performance of a natural language processing model.

[0107] "Optimized conversation" refers to conversation content that is easy to understand and relevant to the user, generated based on the user's hobbies, interests, and communication history.

[0108] "Real-time responses" refers to the instantaneous inputs and responses (e.g., text input, voice responses) a user makes while interacting with a system.

[0109] "Updating the conversation" refers to changing or improving the conversation content generated by the system based on the user's real-time reactions.

[0110] A "conversation generation scenario" refers to the flow and structure of a conversation with a user that is created based on collected data.

[0111] "Evaluating" refers to the process of determining how well the generated conversation matches the user's needs and expectations.

[0112] This invention relates to a system that utilizes a generative AI model to provide natural conversations in order to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[0113] Overall system overview

[0114] This system trains a natural language processing model based on user input such as hobbies, interests, and communication history, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses.

[0115] Data collection

[0116] Collecting hobbies and interests

[0117] When a user registers for the service, they enter their hobbies and interests.

[0118] The terminal receives this input information and sends it to the server.

[0119] Collection of communication history

[0120] With the user's consent, the server collects communication history and conversation history from messaging apps.

[0121] The server stores the collected data in a database.

[0122] Example: User B enters "watching movies" and "hiking" in the hobby field, and the server collects his past message data and stores it in a database.

[0123] Training the model

[0124] The server uses the stored user data to train a natural language processing model, using specific software such as TENSORFLOW (registered trademark) and PyTorch.

[0125] The server learns conversation patterns optimized for individual users.

[0126] Example: User B's hobby data and past communication history are used to train a natural language processing model (e.g., GPT-3®) to learn optimal conversation patterns about movies and hiking.

[0127] Conversation Generation

[0128] The user inputs a specific topic or question into the terminal.

[0129] The terminal sends the input contents to the server.

[0130] The server uses a trained generative AI model to generate optimized conversations.

[0131] The server sends the generated response to the terminal, which displays it to the user.

[0132] Example: User A types into his terminal, "Where do you want to go on your next holiday?" The server responds by saying, "How about that trip to Hokkaido we talked about earlier?" and the terminal displays this to User A.

[0133] Real-time adjustments

[0134] Real-time reaction analysis

[0135] Users react to generated conversations in real time.

[0136] The device receives the real-time response and sends it to the server.

[0137] Update the conversation

[0138] The server analyzes the user's real-time responses and sends feedback to the generative AI model.

[0139] The server updates the conversation content based on the feedback and sends it back to the device.

[0140] The terminal displays the updated content to the user.

[0141] Example: User A responds, "That sounds good, but I'm concerned about the budget," and the server analyzes this and updates the suggestion to, "Then, how about a trip to a hot spring in Hakone?", which the device displays to User A.

[0142] Examples of prompt statements

[0143] User A: "Where do you want to go on your next vacation?"

[0144] Server: "How was that trip to Hokkaido you were talking about earlier?"

[0145] User A: "That's good, but I'm concerned about the budget."

[0146] Server: "So, how about a trip to the hot springs in Hakone?"

[0147] This system allows for natural conversation regardless of time or place, eliminating the sense of time or space gaps in communal living and enabling smooth communication.

[0148] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0149] Step 1: User Registration

[0150] The user accesses the service for the first time and creates an account. The device displays a registration form to the user, where the user enters their name, email address, and hobbies and interests. The device then sends the information to the server.

[0151] Input: Name, email address, hobbies and interests

[0152] Output: User registration data (saved in database)

[0153] Step 2: Collect communication history

[0154] With the user's consent, the server collects communication history from messaging apps, emails, etc. The server stores this data in a database.

[0155] Input: User consent, communication history

[0156] Output: Collected communication history data (stored in a database)

[0157] Step 3: Save data

[0158] The server stores the user's hobbies and interests and the collected communication history in a database, confirms that the storage was successful, and sends a success message to the terminal.

[0159] Input: Hobbies and interests, communication history

[0160] Output: Data saved to the database, message that save was successful

[0161] Step 4: Train the model

[0162] The server uses the stored data to train a natural language processing model, specifically using TensorFlow and PyTorch to train the model and have it learn conversation patterns optimized for the user.

[0163] Input: User data in the database

[0164] Output: A trained natural language processing model

[0165] Step 5: Enter the conversation topic

[0166] The user inputs a conversation topic or question into the terminal, and the terminal transmits the input content to the server.

[0167] Input: A topic or question entered by the user

[0168] Output: Send data to the server

[0169] Step 6: Conversation generation

[0170] The server uses the trained generative AI model to generate optimized conversations, and sends the generated responses to the device, which then displays them to the user.

[0171] Input: The subject or question sent to the server

[0172] Output: Generated conversation (sent to terminal)

[0173] Step 7: Analyzing real-time responses

[0174] The user reacts to the generated conversation in real time, and the device transmits the reaction to the server.

[0175] Input: Real-time user response

[0176] Output: Sends response data to the server

[0177] Step 8: Update the conversation

[0178] The server analyzes the user's real-time responses and sends feedback to the generative AI model. The conversation content is updated based on that feedback, and the newly generated content is sent to the device. The device then displays the updated content to the user.

[0179] Input: Real-time user response

[0180] Output: Updated conversation (sent to terminal)

[0181] Specific operation example

[0182] 1. User A types into the terminal, "Where do you want to go on your next holiday?"

[0183] 2. The device sends this input to the server.

[0184] 3. The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[0185] 4. The device displays the generated conversation to User A.

[0186] 5. User A responds, "That's fine, but I'm concerned about the budget."

[0187] 6. The device sends this response to the server.

[0188] 7. The server analyzes the real-time response and updates the suggestion with, "How about a trip to a hot spring in Hakone?"

[0189] 8. The device displays the updated proposal to User A.

[0190] (Application example 1)

[0191] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0192] The present invention aims to improve customer satisfaction by facilitating smooth communication between customers and store clerks in brick-and-mortar stores. In particular, there is a need to strengthen relationships of trust with customers by providing personalized services and making appropriate product recommendations based on the customer's preferences and past purchase history. However, conventional technologies have had difficulty optimizing real-time conversations and generating responses tailored to individual customer needs.

[0193] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0194] In this invention, the server includes: means for collecting user hobbies and interests; means for collecting user communication history; means for training a natural language processing model based on the collected hobbies and interests and communication history; means for generating an optimized conversation using the generated natural language processing model; means for analyzing the user's real-time reactions and updating the conversation based on the reactions; means for generating a conversation between a customer and a store clerk to support interactions in a physical store; and means for displaying the generated conversation using an application installed on smart glasses or a head-mounted display. This enables smoother interactions with customers in physical stores and more personalized services to be provided.

[0195] The "means for collecting user's hobbies and interests" is a function for recording information on hobbies and interests entered by the user and saving it as data.

[0196] "Means for collecting a user's communication history" refers to a function for obtaining and analyzing past communication history, such as text messages and call records, with the user's consent.

[0197] "Means for training natural language processing models based on collected hobbies, interests, and communication history" refers to a function that uses collected user data to optimize natural language processing models using machine learning and deep learning algorithms.

[0198] "Means for generating optimized conversations using the generated natural language processing model" is a function that uses a trained natural language processing model to generate optimal conversations based on the user's interests and past communication history.

[0199] "Means for analyzing the user's real-time responses and updating the conversation based on those responses" refers to a function that analyzes the real-time text and voice responses entered by the user and updates the content of the conversation in a timely manner based on that information.

[0200] "A means for generating conversations between customers and store clerks to support interactions in physical stores" is a function that supports conversations between customers and store clerks in physical stores and makes appropriate product suggestions tailored to the customer's needs.

[0201] "Means for displaying the generated conversation using an application installed on smart glasses or a head-mounted display" refers to a function for displaying the generated conversation content using an application installed on a wearable device such as smart glasses or a head-mounted display.

[0202] MODE FOR CARRYING OUT THE INVENTION

[0203] The present invention is designed to facilitate communication between customers and store clerks in physical stores and to provide personalized services. Specific embodiments for carrying out the present invention will be described below.

[0204] Overall system overview

[0205] This system collects users' hobbies, interests, and communication history, and uses them to train a natural language processing model to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses. The system primarily consists of a server, terminals, and wearable devices such as smart glasses and head-mounted displays.

[0206] Data collection

[0207] Collecting hobbies and interests

[0208] When a user (customer) registers for the service, they enter their hobbies and interests into a terminal. This information is sent to the server via the terminal and stored in a database.

[0209] Collection of communication history

[0210] With the user's consent, the server collects communication history and past conversation history from messaging apps, and this data is also stored in the server's database.

[0211] Model training

[0212] The server trains a natural language processing model based on the collected user information, such as hobbies, interests, and communication history. This requires machine learning and deep learning algorithms. For example, a natural language processing model such as OpenAI's GPT-3 is applied.

[0213] Conversation Generation

[0214] The user (store clerk) uses an application installed on smart glasses or a head-mounted display to input conversation topics and questions. This input information is sent to the server via the device. The server uses a generative AI model to generate an optimized conversation and sends the response to the device. The device then displays the received response to the user (store clerk).

[0215] Examples of prompt statements

[0216] When the user (store clerk) types "Looking for new sneakers?", the server generates the following:

[0217] Generates "The Nike model you purchased last time is very popular."

[0218] Real-time adjustments

[0219] Real-time reaction analysis

[0220] The user (customer) responds to the generated conversation in real time (for example, by inputting text or responding with voice). The terminal receives the real-time response and sends it to the server.

[0221] Update the conversation

[0222] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on this feedback and sends it back to the device.

[0223] Examples of prompt statements

[0224] If the user (customer) responds with "I really like that model, but what else do you recommend?" the server generates:

[0225] "We have new color variations in stock, and we also recommend the latest Adidas models," says Genesis.

[0226] Examples of application in physical stores

[0227] This system will be used to support interactions between customers and store clerks in physical stores. For example, if a customer asks for an explanation of a specific product, the store clerk can use the system through the smart glasses to provide the customer with the most appropriate product explanation and suggestions. This allows customers to receive personalized service based on their preferences and past purchasing history, leading to increased customer satisfaction.

[0228] In this way, the system of the present invention significantly improves the quality of service in physical stores.

[0229] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0230] Step 1:

[0231] When a user (customer) registers for the service, they input information about their hobbies and interests through their device. The device then sends this information to the server, which then stores the data on their hobbies and interests in a database.

[0232] Input: User's hobbies and interests

[0233] Output: Hobbies and interests data stored in a database

[0234] Specific operation: The device acquires the user's hobbies and interests and sends them to the server.

[0235] Step 2:

[0236] With the user's consent, the server collects communication history, such as text messages and past conversation history, and stores it in a database.

[0237] Input: User's communication history

[0238] Output: Communication history stored in the database

[0239] Specific operation: The server collects and stores data from messaging apps and communication history.

[0240] Step 3:

[0241] The server uses the collected hobbies, interests, and communication history to train a natural language processing model, using machine learning algorithms to learn patterns from the training data.

[0242] Input: Hobbies and interests, communication history

[0243] Output: A trained natural language processing model

[0244] Specific operation: The server uses a natural language processing model such as GPT-3 to train the model based on the collected data.

[0245] Step 4:

[0246] The user (store clerk) inputs the conversation topic or question into the smart glasses or head-mounted display, which is then sent from the device to the server. The server uses a generative AI model to generate an optimized conversation and sends the response to the device.

[0247] Input: conversation topic or question

[0248] Output: Generated conversational responses

[0249] Specific operation: The device sends the conversation topic to the server, and the server uses a generative AI model to generate a conversation and provide a response.

[0250] Step 5:

[0251] The terminal receives and displays the generated conversation to the user (store clerk) through smart glasses or a head-mounted display.

[0252] Input: Generated conversation response

[0253] Output: The conversation displayed to the user

[0254] Specific operation: The device receives the generated conversation from the server and displays it on smart glasses or a head-mounted display.

[0255] Step 6:

[0256] The user (customer) responds to the generated conversation in real time and inputs the response into the terminal, which then transmits the real-time response to the server.

[0257] Input: Real-time response (text or voice)

[0258] Output: Real-time responses sent to the server

[0259] Specific operation: The process in which the terminal receives real-time responses from the user (customer) and sends them to the server.

[0260] Step 7:

[0261] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on this feedback and sends it back to the device.

[0262] Input: Analyzed real-time responses

[0263] Output: Updated conversation

[0264] Specific operation: The server analyzes real-time responses, updates the generative AI model, generates new conversation content, and sends it to the device.

[0265] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0266] This invention relates to a system that provides natural conversations by utilizing generative AI and an emotion engine to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[0267] Overall system overview

[0268] The system trains a natural language processing model based on user-entered hobbies, interests, communication history, and emotional information, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on real-time user reactions and emotional analysis.

[0269] Data collection

[0270] 1. Collecting hobbies and interests

[0271] When a user registers for the service, they enter their hobbies and interests.

[0272] The terminal receives this input information and sends it to the server.

[0273] 2. Collection of communication history

[0274] With the user's consent, the server collects past conversation history between users from communication history and messaging apps.

[0275] The server stores the collected data in a database.

[0276] 3. Collecting emotional information

[0277] The device analyzes the user's emotional information in real time using an emotion engine and sends it to the server.

[0278] Model training

[0279] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected.

[0280] The server iteratively trains the model with data to learn speech patterns optimized for individual users.

[0281] Conversation Generation

[0282] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[0283] The server uses the generative AI model to generate an optimized conversation (e.g., "How was the trip to Hokkaido we talked about earlier?") and sends the response to the device.

[0284] The terminal displays the received response to the user.

[0285] Real-time adjustments

[0286] 1. Real-time reaction analysis

[0287] Users respond in real time to generated conversations (e.g., "That's good, but I'm concerned about the budget").

[0288] The device receives the real-time response and sends it to the server.

[0289] 2. Emotional Information Analysis

[0290] The device uses an emotion engine to analyze emotional information from the user's facial expressions and voice in real time, such as facial expressions and tone of voice when the user shows interest.

[0291] The device transmits the analyzed emotional information to the server.

[0292] 3. Update the conversation

[0293] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model.

[0294] The server updates the conversation content based on the feedback and sends it back to the device.

[0295] Specific examples

[0296] Travel plan example

[0297] 1. Input Phase

[0298] User A types into the terminal, "Where do you want to go on your next holiday?"

[0299] The device sends input to the server.

[0300] 2. Conversation generation phase

[0301] The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[0302] The terminal displays the generated conversation to User A.

[0303] 3. Real-time reaction and emotional information collection

[0304] User A responds, "That's good, but I'm concerned about the budget." The device sends the response in real time to the server.

[0305] The device uses an emotion engine to analyze the user's facial expressions and voice to determine whether they are interested, and then sends that information to the server.

[0306] 4. Conversation Update Phase

[0307] The server analyzes real-time reactions and emotional information and updates the suggestion with, "So, how about a hot spring trip to Hakone?"

[0308] The device displays the updated proposal to User A.

[0309] In this way, the system of the present invention can provide natural and optimized conversations based on the user's emotions, regardless of time or place, and can adjust to eliminate any sense of time or space discrepancy in communal living.

[0310] The processing flow will be explained below.

[0311] Step 1:

[0312] When users register for the service, they enter their hobbies and interests. The device receives this information and sends it to the server.

[0313] Step 2:

[0314] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[0315] Step 3:

[0316] The device analyzes the user's emotional information in real time using an emotion engine and transmits it to the server, including the user's facial expressions, tone of voice, and physical movements.

[0317] Step 4:

[0318] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected by the server. The server repeatedly trains the model using data to learn conversation patterns optimized for each individual user.

[0319] Step 5:

[0320] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[0321] Step 6:

[0322] The server uses the generative AI model to generate an optimized conversation (e.g., "How about the trip to Hokkaido we talked about earlier?") and sends the response to the device. The device then displays the received response to the user.

[0323] Step 7:

[0324] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives the real-time response and sends it to the server.

[0325] Step 8:

[0326] The device analyzes emotional information from the user's facial expressions and voice in real time using an emotion engine and sends the information to the server.

[0327] Step 9:

[0328] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model, which then updates the conversation content based on the feedback.

[0329] Step 10:

[0330] The server sends the updated conversation (e.g., "So, how about a trip to a hot spring in Hakone?") to the device again. The device displays the updated proposal to the user.

[0331] In this way, users can experience natural and optimized conversations regardless of time or place, and feedback based on the user's emotions makes communication even more natural.

[0332] Example 2

[0333] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0334] In communal living, time and space differences can easily lead to decreased communication and increased emotional distance. Conventional systems do not provide natural conversations that effectively utilize users' hobbies, interests, communication history, and real-time emotional information, which is an obstacle to smooth communication. To solve this problem, a system is needed that can collect and analyze a wide range of user information and provide natural conversations based on the results.

[0335] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0336] In this invention, the server includes means for collecting user hobbies and interests, means for collecting user communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and collecting emotional information, and means for updating the conversation in real time based on the collected emotional information. This makes it possible to utilize the user's multifaceted information and have a natural and optimized conversation based on the real-time reactions and emotional information.

[0337] "User" means an individual or organization that uses the system.

[0338] "Hobbies and Interests" refers to specific activities or subjects that interest a user, which information the user enters into the system.

[0339] "Communication history" refers to a record of messages and conversations that a user has had with other users in the past.

[0340] A "natural language processing model" is a type of machine learning model for analyzing, understanding, and generating text written in natural language.

[0341] "Generated natural language processing model" refers to a natural language processing model trained on collected data.

[0342] "Optimized conversation" refers to the most appropriate and natural conversation content generated based on the user's hobbies, interests, communication history, and emotional information.

[0343] "Real-time response" refers to the instantaneous reactions and feedback a user gives while interacting with a system.

[0344] "Emotional information" refers to the emotional state analyzed from the user's facial expression, tone of voice, etc.

[0345] "Updating the conversation in real time" means dynamically changing the content of the conversation based on the user's real-time reactions and emotional information.

[0346] Overall system overview

[0347] This invention is a system designed to resolve time and space differences in communal living and provide smoother communication. This system trains a natural language processing model based on user-entered hobbies, interests, communication history, and emotional information, and generates optimized conversations.

[0348] Hardware and software used

[0349] Server: Responsible for storing data, training models, and generating conversations

[0350] Terminal: Responsible for inputting information from the user, collecting real-time emotion data, and displaying the generated conversation content

[0351] Emotion Engine: Software for analyzing emotional information from the user's facial expressions and voice

[0352] Natural language processing model: A machine learning model that generates conversations based on collected data.

[0353] Data collection

[0354] 1. Collecting hobbies and interests

[0355] When a user registers for the service, they input their hobbies and interests. The device receives this input information and sends it to the server.

[0356] 2. Collection of communication history

[0357] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[0358] 3. Collecting emotional information

[0359] The device analyzes the user's facial expressions and voice in real time using an emotion engine and sends the results to a server, which then stores the received emotion information in a database.

[0360] Model training

[0361] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected by the server. The data is used to repeatedly train the model, allowing it to learn conversation patterns optimized for specific users.

[0362] Conversation Generation

[0363] The user enters a conversation topic or question into the device (e.g., "Where do you want to go on your next vacation?"). The device sends this information to the server, which uses a generative AI model to generate an optimized response (e.g., "How about that trip to Hokkaido we were talking about earlier?"). The server sends the generated response to the device, which displays it to the user.

[0364] Real-time adjustments

[0365] 1. Real-time reaction analysis

[0366] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives this response and sends it to the server.

[0367] 2. Emotional Information Analysis

[0368] The device analyzes the user's facial expressions and tone of voice in real time to determine their emotional state, and sends the analysis results to the server.

[0369] 3. Update the conversation

[0370] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model. The server updates the conversation content based on this feedback and sends it back to the device (e.g., "So, how about a hot spring trip to Hakone?"). The device then displays the updated suggestions to the user.

[0371] Specific examples

[0372] 1. Input Phase

[0373] User A enters "Where do you want to go on your next holiday?" into the device. The device sends this information to the server.

[0374] 2. Conversation generation phase

[0375] The server generates a conversation based on past conversation history and hobby information, asking, "How about that trip to Hokkaido we talked about earlier?" The device displays the generated conversation to User A.

[0376] 3. Real-time reaction and emotional information collection

[0377] User A responds, "That's good, but I'm concerned about the budget." The device sends this response to the server. The device uses an emotion engine to analyze the user's facial expressions and voice, and sends that information to the server.

[0378] 4. Conversation Update Phase

[0379] The server analyzes the real-time reactions and emotional information and updates the suggestion to "How about a trip to a hot spring in Hakone?" The device displays the updated suggestion to User A.

[0380] Prompt Sentence Examples

[0381] "Where do you want to go on your next vacation? Get travel suggestions based on your hobbies and past communication history."

[0382] By implementing the present invention in this way, users can experience optimized, natural conversations regardless of time or place. Real-time emotion analysis also enables more appropriate and satisfying communication.

[0383] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0384] Step 1: User Registration

[0385] Input: A user accesses the system and enters the required information to create a new account.

[0386] Output: The terminal receives the user information and sends it to the server.

[0387] Specific behavior:

[0388] A user enters their name, email address, password, etc. into an input form.

[0389] The terminal sends this information to the server via the send button.

[0390] The server stores the received information in a database and creates a user account.

[0391] Step 2: Enter your hobbies and interests

[0392] Input: The user enters their hobbies and interests.

[0393] Output: The device receives this information and sends it to the server.

[0394] Specific behavior:

[0395] Users fill out a multiple-choice form to select activities or topics that interest them.

[0396] The terminal checks the input information and sends it to the server.

[0397] The server stores the received hobbies and interests in a database.

[0398] Step 3: Collect communication history

[0399] Input: Get user consent.

[0400] Output: The server collects past conversation history from communication history and messaging apps and stores it in a database.

[0401] Specific behavior:

[0402] The user presses the consent button regarding the collection of communication history.

[0403] The server collects past communication history through the messaging app's API.

[0404] The collected data is stored in a database.

[0405] Step 4: Collecting emotional information

[0406] Input: The device captures the user's facial expressions and voice in real time.

[0407] Output: The device analyzes these data using the emotion engine and sends the results to the server.

[0408] Specific behavior:

[0409] The device uses a camera and microphone to capture the user's facial expressions and voice.

[0410] The device uses an emotion engine to analyze facial expressions and voice in real time.

[0411] The analysis results are sent to a server, which stores the information in a database.

[0412] Step 5: Data Preparation

[0413] Input: The server collects information about the user's hobbies, interests, communication history, and emotional information.

[0414] Output: Combined into a dataset and used for model training.

[0415] Specific behavior:

[0416] The server extracts the necessary data from the database and performs preprocessing.

[0417] Integrate the data to create a dataset for model training.

[0418] Step 6: Model training

[0419] Input: Dataset.

[0420] Output: A trained natural language processing model.

[0421] Specific behavior:

[0422] The server uses the dataset to train a natural language processing model.

[0423] During the training process, the model parameters are optimized and the user is trained to learn optimized conversation patterns.

[0424] Step 7: User Input

[0425] Input: The user types a question or conversation topic into the device (e.g., "Where do you want to go on your next vacation?").

[0426] Output: The device receives input and sends it to the server.

[0427] Specific behavior:

[0428] The user enters a question into the input form on the terminal and presses the send button.

[0429] The terminal transmits the user input as text data to the server.

[0430] Step 8: Conversation Generation

[0431] Input: User questions and conversation topics.

[0432] Output: The server uses the generative AI model to generate an optimized response and sends it to the device.

[0433] Specific behavior:

[0434] The server receives the user's question and compares it with historical data.

[0435] Generative AI models are used to generate appropriate responses (e.g., "How about that trip to Hokkaido we talked about earlier?").

[0436] The generated response is sent to the terminal, which displays it to the user.

[0437] Step 9: Analyze real-time responses

[0438] Input: Real-time user response (e.g., "That sounds good, but I'm concerned about the budget").

[0439] Output: The device receives the response and sends it to the server.

[0440] Specific behavior:

[0441] Users provide real-time feedback on the generated conversation.

[0442] The device captures the response and sends it to the server.

[0443] Step 10: Emotional Analysis

[0444] Input: User facial and voice data.

[0445] Output: The analysis results of the emotion engine are sent to the server.

[0446] Specific behavior:

[0447] The device uses a camera and microphone to capture the user's facial expressions and voice data.

[0448] The emotion engine analyzes this data in real time and extracts emotional information.

[0449] The analysis results are sent to the server.

[0450] Step 11: Update your conversation

[0451] Input: Real-time user reaction and emotional information.

[0452] Output: Sends the updated conversation to the terminal.

[0453] Specific behavior:

[0454] The server analyzes real-time reactions and emotional information.

[0455] Feedback is fed into generative AI models to optimize and update conversation content.

[0456] The updated conversation content is sent to the terminal, which displays it to the user (e.g., "So, how about a trip to the hot springs in Hakone?").

[0457] (Application example 2)

[0458] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0459] In communal living, time and space differences can reduce communication and increase emotional distance. Furthermore, users' shopping experiences in virtual stores are not personalized, which can lead to unexpected frustration and stress. To address these issues, a system is needed that can generate optimized conversations based on users' hobbies, interests, real-time reactions, and emotional information, thereby improving the user's shopping experience.

[0460] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0461] In this invention, the server includes means for collecting user hobbies and interests, means for collecting user communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and updating the conversation based on the reactions, means for collecting and analyzing emotional information such as the user's facial expression information and tone of voice, and means for adjusting the conversation based on the analyzed emotional information. This makes it possible to realize more personalized and natural conversation based on the user's real-time emotions and reactions, and improve the user experience in the virtual store.

[0462] "Hobbies and interests" refers to the things, activities, and preferences that a user is interested in.

[0463] "Communication history" refers to a record of past conversations and messages between users.

[0464] A "natural language processing model" refers to a statistical machine learning model for understanding and generating human language.

[0465] "Real-time reactions" refers to the immediate reactions and responses that users give during a conversation.

[0466] "Facial expression information" refers to data relating to emotions obtained from the user's facial expressions.

[0467] "Voice tone" refers to information about the characteristics of the voice, such as the pitch and volume of the voice.

[0468] "Emotional information" refers to data that represents the user's emotional state, such as facial expressions and tone of voice.

[0469] The "means for adjusting the conversation" refers to a method for updating the generated conversation content based on the user's real-time reactions and emotional information.

[0470] To implement this invention, it is necessary to build a system that collects users' hobbies and interests, communication history, real-time reactions, and emotional information, and generates optimized conversations based on the collected information. This system includes the following main elements:

[0471] First, to collect the user's hobbies and interests, a method is required for the user to input these when logging in to the application. Also, with the user's consent, communication history is collected and this data is sent to the server, allowing the server to manage user information in an integrated manner.

[0472] The server trains a natural language processing model based on the collected hobbies, interests, and communication history. Specifically, it uses Python and TensorFlow to build and train a generative AI model. This model is used to learn the user's personalized conversation patterns and generate optimized conversations.

[0473] Next, to collect real-time user reactions and emotional information, devices such as smartphones and head-mounted displays must be equipped with a camera and microphone. These devices use OpenCV and emotion analysis libraries to analyze the user's facial expressions and tone of voice, and transmit this information to the server as emotional information.

[0474] Based on this emotional information and real-time reactions, the server provides feedback to the generative AI model to adjust and update the conversation content. Finally, the adjusted conversation content is sent to the device and displayed to the user.

[0475] As a concrete example, in a virtual shopping assistant scenario, if a user asks, "Do you recommend these shoes?", the server uses a generative AI model to respond, "How about these shoes that suit your style?" Furthermore, if the user responds, "I'd like to see reviews too," the server analyzes real-time emotional information and generates a tailored response, such as, "Here are reviews from other users. They look great!"

[0476] Example prompt sentence:

[0477] A user asked 'Do you recommend these shoes?'

[0478] The realization of this system will enable more personalized and natural conversations to be provided based on the user's real-time emotions and reactions, improving the user experience in virtual stores.

[0479] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0480] Step 1:

[0481] The device receives input from the user about hobbies and interests, which are entered when the user logs into the application, and transmits the input information to the server.

[0482] Input: User's hobbies and interests

[0483] Output: User's hobbies and interests sent to the server

[0484] Specific operation: Using the UI / UX on the device, the user fills in an input form about their hobbies and interests. By pressing the input button, the data is sent to the server.

[0485] Step 2:

[0486] The server collects the user's communication history and, with the user's consent, analyzes past conversation history from messaging apps and stores it in a database.

[0487] Input: User consent, communication history

[0488] Output: Communication history stored in the database

[0489] Specific operation: After the server confirms the user's consent, it obtains communication history from the messaging app and stores the analyzed data in a database.

[0490] Step 3:

[0491] The server trains a natural language processing model based on collected hobbies, interests, and communication history. It uses Python and TensorFlow to build and train a generative AI model that learns the user's personalized speech patterns.

[0492] Input: Hobbies and interests, communication history

[0493] Output: A trained natural language processing model

[0494] How it works: A generative AI model is built using TensorFlow, and the model is trained using collected data. After multiple learning cycles, the model learns optimal conversation patterns.

[0495] Step 4:

[0496] The device collects the user's real-time reactions and emotional information and sends it to the server. Using a smartphone or head-mounted display, facial expression information and tone of voice are collected from the camera and microphone, and analyzed using OpenCV and the emotion analysis library.

[0497] Input: Real-time user reactions, facial expressions, and tone of voice

[0498] Output: Emotion information sent to the server

[0499] Specific operation: The device's camera and microphone are used to record the user's reactions, facial expressions are analyzed using OpenCV, emotional states are detected using an emotion analysis library, and the data is sent to the server.

[0500] Step 5:

[0501] The server provides feedback to the generative AI model based on real-time reactions and emotional information, adjusting and updating the conversation content. The adjusted conversation content is then sent to the device and displayed to the user.

[0502] Input: Real-time reactions, emotional information

[0503] Output: Adjusted conversation

[0504] How it works: The server analyzes the received emotional information and reactions and provides feedback to the generative AI model. The model generates an adjusted conversation and sends it to the device. The device then displays the updated conversation to the user.

[0505] Step 6:

[0506] The device displays the adjusted conversation content to the user, who can then further react to the displayed conversation content, which becomes the starting point for the next conversation.

[0507] Input: Modified conversation

[0508] Output: The conversation as it appears to the user

[0509] Specific operation: The adjusted conversation content is presented to the user on the device display screen. The user reads it and gives a new response, which leads to the next conversation process.

[0510] This allows the system as a whole to generate more personalized and natural conversations based on the user's real-time emotions and reactions.

[0511] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0512] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0513] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0514] [Second embodiment]

[0515] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0516] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0517] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0518] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0519] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0520] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0521] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0522] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0523] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0524] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0525] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0526] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0527] This invention relates to a system that uses generative AI to provide natural conversations in order to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[0528] Overall system overview

[0529] The system trains a natural language processing model based on user input such as hobbies, interests, and communication history, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses.

[0530] Data collection

[0531] 1. Collecting hobbies and interests

[0532] When a user registers for the service, they enter their hobbies and interests.

[0533] The terminal receives this input information and sends it to the server.

[0534] 2. Collection of communication history

[0535] With the user's consent, the server collects communication history and conversation history from messaging apps.

[0536] The server stores the collected data in a database.

[0537] Model training

[0538] The server trains a natural language processing model based on the user data collected.

[0539] The server iteratively trains the model with data to learn speech patterns optimized for individual users.

[0540] Conversation Generation

[0541] The user inputs a conversation topic or question into the terminal.

[0542] The device sends the input to the server.

[0543] The server uses the generative AI model to generate optimized conversations and sends the responses to the device.

[0544] The terminal displays the received reply to the user.

[0545] Real-time adjustments

[0546] 1. Real-time reaction analysis

[0547] Users react in real time to the generated conversation (e.g., by typing text or responding with voice).

[0548] The device receives real-time responses and transmits them to the server.

[0549] 2. Update the conversation

[0550] The server analyzes the user's real-time responses and sends feedback to the generative AI model.

[0551] The server updates the conversation content based on the feedback and sends it back to the device.

[0552] Specific examples

[0553] Travel plan example

[0554] 1. Input Phase

[0555] User A types into the terminal, "Where do you want to go on your next holiday?"

[0556] The device sends input to the server.

[0557] 2. Conversation generation phase

[0558] The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[0559] The terminal displays the generated conversation to User A.

[0560] 3. Real-time response

[0561] User A responds, "That's good, but I'm concerned about the budget."

[0562] The device sends this response to the server.

[0563] 4. Conversation Update Phase

[0564] The server analyzes the real-time responses and updates the suggestion with, "So, how about a trip to a hot spring in Hakone?"

[0565] The device displays the updated proposal to User A.

[0566] In this way, the system of the present invention can provide natural conversations that make people feel as if they are together in the same place, regardless of time or place, and can adjust to eliminate the sense of time and space discrepancies in communal living.

[0567] The processing flow will be explained below.

[0568] Step 1:

[0569] When users register for the service, they enter their hobbies and interests. The device receives this information and sends it to the server.

[0570] Step 2:

[0571] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[0572] Step 3:

[0573] The server trains a natural language processing model based on the user's hobbies, interests, and communication history collected by the server. The server repeatedly trains the model using data to learn conversation patterns optimized for each individual user.

[0574] Step 4:

[0575] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[0576] Step 5:

[0577] The server uses the generative AI model to generate an optimized conversation (e.g., "How about the trip to Hokkaido we talked about earlier?") and sends the response to the device. The device then displays the received response to the user.

[0578] Step 6:

[0579] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives the real-time response and sends it to the server.

[0580] Step 7:

[0581] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on the feedback.

[0582] Step 8:

[0583] The server sends the updated conversation (e.g., "So, how about a trip to a hot spring in Hakone?") to the device again. The device displays the updated proposal to the user.

[0584] In this way, users are provided with natural and optimized conversations regardless of time or place.

[0585] Example 1

[0586] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0587] In modern society, differences in time and space in communal living often lead to a decrease in communication and an increase in emotional distance, making it difficult for family and friends to interact smoothly and potentially leading to estrangement.

[0588] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0589] In this invention, the server includes means for inputting the user's hobbies and interests, means for collecting the user's communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and updating the conversation based on the reactions, and means for saving the collected data in a database. This makes it possible to provide natural conversation regardless of time or place, and to adjust so that time and space differences in communal living are not felt.

[0590] "User" refers to a person who uses the system.

[0591] "Hobbies and interests" refers to information about activities or topics that interest a user.

[0592] "Communication history" refers to a record of a user's past digital communications, such as messages, emails, and chats.

[0593] "Natural language processing model" refers to an algorithm or machine learning model for understanding and generating human language.

[0594] A "server" refers to a computer device that processes the entire system, stores and analyzes data, trains models, and so on.

[0595] "Terminal" refers to a hardware device that is directly operated by a user (e.g., smartphone, tablet, PC).

[0596] "Database" refers to a system for efficiently storing and managing collected data.

[0597] "Training a model" refers to the process of using collected data to refine and improve the performance of a natural language processing model.

[0598] "Optimized conversation" refers to conversation content that is easy to understand and relevant to the user, generated based on the user's hobbies, interests, and communication history.

[0599] "Real-time responses" refers to the instantaneous inputs and responses (e.g., text input, voice responses) a user makes while interacting with a system.

[0600] "Updating the conversation" refers to changing or improving the conversation content generated by the system based on the user's real-time reactions.

[0601] A "conversation generation scenario" refers to the flow and structure of a conversation with a user that is created based on collected data.

[0602] "Evaluating" refers to the process of determining how well the generated conversation matches the user's needs and expectations.

[0603] This invention relates to a system that utilizes a generative AI model to provide natural conversations in order to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[0604] Overall system overview

[0605] This system trains a natural language processing model based on user input such as hobbies, interests, and communication history, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses.

[0606] Data collection

[0607] Collecting hobbies and interests

[0608] When a user registers for the service, they enter their hobbies and interests.

[0609] The terminal receives this input information and sends it to the server.

[0610] Collection of communication history

[0611] With the user's consent, the server collects communication history and conversation history from messaging apps.

[0612] The server stores the collected data in a database.

[0613] Example: User B enters "watching movies" and "hiking" in the hobby field, and the server collects his past message data and stores it in a database.

[0614] Training the model

[0615] The server uses the stored user data to train a natural language processing model, using TensorFlow and PyTorch as the specific software.

[0616] The server learns conversation patterns optimized for individual users.

[0617] Example: User B's hobby data and past communication history are used to train a natural language processing model (e.g., GPT-3) to learn optimal conversation patterns about movies and hiking.

[0618] Conversation Generation

[0619] The user inputs a specific topic or question into the terminal.

[0620] The terminal sends the input contents to the server.

[0621] The server uses a trained generative AI model to generate optimized conversations.

[0622] The server sends the generated response to the terminal, which displays it to the user.

[0623] Example: User A types into his terminal, "Where do you want to go on your next holiday?" The server responds by saying, "How about that trip to Hokkaido we talked about earlier?" and the terminal displays this to User A.

[0624] Real-time adjustments

[0625] Real-time reaction analysis

[0626] Users react to generated conversations in real time.

[0627] The device receives the real-time response and sends it to the server.

[0628] Update the conversation

[0629] The server analyzes the user's real-time responses and sends feedback to the generative AI model.

[0630] The server updates the conversation content based on the feedback and sends it back to the device.

[0631] The terminal displays the updated content to the user.

[0632] Example: User A responds, "That sounds good, but I'm concerned about the budget," and the server analyzes this and updates the suggestion to, "Then, how about a trip to a hot spring in Hakone?", which the device displays to User A.

[0633] Examples of prompt statements

[0634] User A: "Where do you want to go on your next vacation?"

[0635] Server: "How was that trip to Hokkaido you were talking about earlier?"

[0636] User A: "That's good, but I'm concerned about the budget."

[0637] Server: "So, how about a trip to the hot springs in Hakone?"

[0638] This system allows for natural conversation regardless of time or place, eliminating the sense of time or space gaps in communal living and enabling smooth communication.

[0639] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0640] Step 1: User Registration

[0641] The user accesses the service for the first time and creates an account. The device displays a registration form to the user, where the user enters their name, email address, and hobbies and interests. The device then sends the information to the server.

[0642] Input: Name, email address, hobbies and interests

[0643] Output: User registration data (saved in database)

[0644] Step 2: Collect communication history

[0645] With the user's consent, the server collects communication history from messaging apps, emails, etc. The server stores this data in a database.

[0646] Input: User consent, communication history

[0647] Output: Collected communication history data (stored in a database)

[0648] Step 3: Save data

[0649] The server stores the user's hobbies and interests and the collected communication history in a database, confirms that the storage was successful, and sends a success message to the terminal.

[0650] Input: Hobbies and interests, communication history

[0651] Output: Data saved to the database, message that save was successful

[0652] Step 4: Train the model

[0653] The server uses the stored data to train a natural language processing model, specifically using TensorFlow and PyTorch to train the model and have it learn conversation patterns optimized for the user.

[0654] Input: User data in the database

[0655] Output: A trained natural language processing model

[0656] Step 5: Enter the conversation topic

[0657] The user inputs a conversation topic or question into the terminal, and the terminal transmits the input content to the server.

[0658] Input: A topic or question entered by the user

[0659] Output: Send data to the server

[0660] Step 6: Conversation generation

[0661] The server uses the trained generative AI model to generate optimized conversations, and sends the generated responses to the device, which then displays them to the user.

[0662] Input: The subject or question sent to the server

[0663] Output: Generated conversation (sent to terminal)

[0664] Step 7: Analyzing real-time responses

[0665] The user reacts to the generated conversation in real time, and the device transmits the reaction to the server.

[0666] Input: Real-time user response

[0667] Output: Sends response data to the server

[0668] Step 8: Update the conversation

[0669] The server analyzes the user's real-time responses and sends feedback to the generative AI model. The conversation content is updated based on that feedback, and the newly generated content is sent to the device. The device then displays the updated content to the user.

[0670] Input: Real-time user response

[0671] Output: Updated conversation (sent to terminal)

[0672] Specific operation example

[0673] 1. User A types into the terminal, "Where do you want to go on your next holiday?"

[0674] 2. The device sends this input to the server.

[0675] 3. The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[0676] 4. The device displays the generated conversation to User A.

[0677] 5. User A responds, "That's fine, but I'm concerned about the budget."

[0678] 6. The device sends this response to the server.

[0679] 7. The server analyzes the real-time response and updates the suggestion with, "How about a trip to a hot spring in Hakone?"

[0680] 8. The device displays the updated proposal to User A.

[0681] (Application example 1)

[0682] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0683] The present invention aims to improve customer satisfaction by facilitating smooth communication between customers and store clerks in brick-and-mortar stores. In particular, there is a need to strengthen relationships of trust with customers by providing personalized services and making appropriate product recommendations based on the customer's preferences and past purchase history. However, conventional technologies have had difficulty optimizing real-time conversations and generating responses tailored to individual customer needs.

[0684] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0685] In this invention, the server includes: means for collecting user hobbies and interests; means for collecting user communication history; means for training a natural language processing model based on the collected hobbies and interests and communication history; means for generating an optimized conversation using the generated natural language processing model; means for analyzing the user's real-time reactions and updating the conversation based on the reactions; means for generating a conversation between a customer and a store clerk to support interactions in a physical store; and means for displaying the generated conversation using an application installed on smart glasses or a head-mounted display. This enables smoother interactions with customers in physical stores and more personalized services to be provided.

[0686] The "means for collecting user's hobbies and interests" is a function for recording information on hobbies and interests entered by the user and saving it as data.

[0687] "Means for collecting a user's communication history" refers to a function for obtaining and analyzing past communication history, such as text messages and call records, with the user's consent.

[0688] "Means for training natural language processing models based on collected hobbies, interests, and communication history" refers to a function that uses collected user data to optimize natural language processing models using machine learning and deep learning algorithms.

[0689] "Means for generating optimized conversations using the generated natural language processing model" is a function that uses a trained natural language processing model to generate optimal conversations based on the user's interests and past communication history.

[0690] "Means for analyzing the user's real-time responses and updating the conversation based on those responses" refers to a function that analyzes the real-time text and voice responses entered by the user and updates the content of the conversation in a timely manner based on that information.

[0691] "A means for generating conversations between customers and store clerks to support interactions in physical stores" is a function that supports conversations between customers and store clerks in physical stores and makes appropriate product suggestions tailored to the customer's needs.

[0692] "Means for displaying the generated conversation using an application installed on smart glasses or a head-mounted display" refers to a function for displaying the generated conversation content using an application installed on a wearable device such as smart glasses or a head-mounted display.

[0693] MODE FOR CARRYING OUT THE INVENTION

[0694] The present invention is designed to facilitate communication between customers and store clerks in physical stores and to provide personalized services. Specific embodiments for carrying out the present invention will be described below.

[0695] Overall system overview

[0696] This system collects users' hobbies, interests, and communication history, and uses them to train a natural language processing model to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses. The system primarily consists of a server, terminals, and wearable devices such as smart glasses and head-mounted displays.

[0697] Data collection

[0698] Collecting hobbies and interests

[0699] When a user (customer) registers for the service, they enter their hobbies and interests into a terminal. This information is sent to the server via the terminal and stored in a database.

[0700] Collection of communication history

[0701] With the user's consent, the server collects communication history and past conversation history from messaging apps, and this data is also stored in the server's database.

[0702] Model training

[0703] The server trains a natural language processing model based on the collected user information, such as hobbies, interests, and communication history. This requires machine learning and deep learning algorithms, such as OpenAI's GPT-3.

[0704] Conversation Generation

[0705] The user (store clerk) uses an application installed on smart glasses or a head-mounted display to input conversation topics and questions. This input information is sent to the server via the device. The server uses a generative AI model to generate an optimized conversation and sends the response to the device. The device then displays the received response to the user (store clerk).

[0706] Examples of prompt statements

[0707] When the user (store clerk) types "Looking for new sneakers?", the server generates the following:

[0708] Generates "The Nike model you purchased last time is very popular."

[0709] Real-time adjustments

[0710] Real-time reaction analysis

[0711] The user (customer) responds to the generated conversation in real time (for example, by inputting text or responding with voice). The terminal receives the real-time response and sends it to the server.

[0712] Update the conversation

[0713] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on this feedback and sends it back to the device.

[0714] Examples of prompt statements

[0715] If the user (customer) responds with "I really like that model, but what else do you recommend?" the server generates:

[0716] "We have new color variations in stock, and we also recommend the latest Adidas models," says Genesis.

[0717] Examples of application in physical stores

[0718] This system will be used to support interactions between customers and store clerks in physical stores. For example, if a customer asks for an explanation of a specific product, the store clerk can use the system through the smart glasses to provide the customer with the most appropriate product explanation and suggestions. This allows customers to receive personalized service based on their preferences and past purchasing history, leading to increased customer satisfaction.

[0719] In this way, the system of the present invention significantly improves the quality of service in physical stores.

[0720] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0721] Step 1:

[0722] When a user (customer) registers for the service, they input information about their hobbies and interests through their device. The device then sends this information to the server, which then stores the data on their hobbies and interests in a database.

[0723] Input: User's hobbies and interests

[0724] Output: Hobbies and interests data stored in a database

[0725] Specific operation: The device acquires the user's hobbies and interests and sends them to the server.

[0726] Step 2:

[0727] With the user's consent, the server collects communication history, such as text messages and past conversation history, and stores it in a database.

[0728] Input: User's communication history

[0729] Output: Communication history stored in the database

[0730] Specific operation: The server collects and stores data from messaging apps and communication history.

[0731] Step 3:

[0732] The server uses the collected hobbies, interests, and communication history to train a natural language processing model, using machine learning algorithms to learn patterns from the training data.

[0733] Input: Hobbies and interests, communication history

[0734] Output: A trained natural language processing model

[0735] Specific operation: The server uses a natural language processing model such as GPT-3 to train the model based on the collected data.

[0736] Step 4:

[0737] The user (store clerk) inputs the conversation topic or question into the smart glasses or head-mounted display, which is then sent from the device to the server. The server uses a generative AI model to generate an optimized conversation and sends the response to the device.

[0738] Input: conversation topic or question

[0739] Output: Generated conversational responses

[0740] Specific operation: The device sends the conversation topic to the server, and the server uses a generative AI model to generate a conversation and provide a response.

[0741] Step 5:

[0742] The terminal receives and displays the generated conversation to the user (store clerk) through smart glasses or a head-mounted display.

[0743] Input: Generated conversation response

[0744] Output: The conversation displayed to the user

[0745] Specific operation: The device receives the generated conversation from the server and displays it on smart glasses or a head-mounted display.

[0746] Step 6:

[0747] The user (customer) responds to the generated conversation in real time and inputs the response into the terminal, which then transmits the real-time response to the server.

[0748] Input: Real-time response (text or voice)

[0749] Output: Real-time responses sent to the server

[0750] Specific operation: The process in which the terminal receives real-time responses from the user (customer) and sends them to the server.

[0751] Step 7:

[0752] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on this feedback and sends it back to the device.

[0753] Input: Analyzed real-time responses

[0754] Output: Updated conversation

[0755] Specific operation: The server analyzes real-time responses, updates the generative AI model, generates new conversation content, and sends it to the device.

[0756] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0757] This invention relates to a system that provides natural conversations by utilizing generative AI and an emotion engine to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[0758] Overall system overview

[0759] The system trains a natural language processing model based on user-entered hobbies, interests, communication history, and emotional information, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on real-time user reactions and emotional analysis.

[0760] Data collection

[0761] 1. Collecting hobbies and interests

[0762] When a user registers for the service, they enter their hobbies and interests.

[0763] The terminal receives this input information and sends it to the server.

[0764] 2. Collection of communication history

[0765] With the user's consent, the server collects past conversation history between users from communication history and messaging apps.

[0766] The server stores the collected data in a database.

[0767] 3. Collecting emotional information

[0768] The device analyzes the user's emotional information in real time using an emotion engine and sends it to the server.

[0769] Model training

[0770] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected.

[0771] The server iteratively trains the model with data to learn speech patterns optimized for individual users.

[0772] Conversation Generation

[0773] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[0774] The server uses the generative AI model to generate an optimized conversation (e.g., "How was the trip to Hokkaido we talked about earlier?") and sends the response to the device.

[0775] The terminal displays the received response to the user.

[0776] Real-time adjustments

[0777] 1. Real-time reaction analysis

[0778] Users respond in real time to generated conversations (e.g., "That's good, but I'm concerned about the budget").

[0779] The device receives the real-time response and sends it to the server.

[0780] 2. Emotional Information Analysis

[0781] The device uses an emotion engine to analyze emotional information from the user's facial expressions and voice in real time, such as facial expressions and tone of voice when the user shows interest.

[0782] The device transmits the analyzed emotional information to the server.

[0783] 3. Update the conversation

[0784] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model.

[0785] The server updates the conversation content based on the feedback and sends it back to the device.

[0786] Specific examples

[0787] Travel plan example

[0788] 1. Input Phase

[0789] User A types into the terminal, "Where do you want to go on your next holiday?"

[0790] The device sends input to the server.

[0791] 2. Conversation generation phase

[0792] The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[0793] The terminal displays the generated conversation to User A.

[0794] 3. Real-time reaction and emotional information collection

[0795] User A responds, "That's good, but I'm concerned about the budget." The device sends the response in real time to the server.

[0796] The device uses an emotion engine to analyze the user's facial expressions and voice to determine whether they are interested, and then sends that information to the server.

[0797] 4. Conversation Update Phase

[0798] The server analyzes real-time reactions and emotional information and updates the suggestion with, "So, how about a hot spring trip to Hakone?"

[0799] The device displays the updated proposal to User A.

[0800] In this way, the system of the present invention can provide natural and optimized conversations based on the user's emotions, regardless of time or place, and can adjust to eliminate any sense of time or space discrepancy in communal living.

[0801] The processing flow will be explained below.

[0802] Step 1:

[0803] When users register for the service, they enter their hobbies and interests. The device receives this information and sends it to the server.

[0804] Step 2:

[0805] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[0806] Step 3:

[0807] The device analyzes the user's emotional information in real time using an emotion engine and transmits it to the server, including the user's facial expressions, tone of voice, and physical movements.

[0808] Step 4:

[0809] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected by the server. The server repeatedly trains the model using data to learn conversation patterns optimized for each individual user.

[0810] Step 5:

[0811] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[0812] Step 6:

[0813] The server uses the generative AI model to generate an optimized conversation (e.g., "How about the trip to Hokkaido we talked about earlier?") and sends the response to the device. The device then displays the received response to the user.

[0814] Step 7:

[0815] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives the real-time response and sends it to the server.

[0816] Step 8:

[0817] The device analyzes emotional information from the user's facial expressions and voice in real time using an emotion engine and sends the information to the server.

[0818] Step 9:

[0819] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model, which then updates the conversation content based on the feedback.

[0820] Step 10:

[0821] The server sends the updated conversation (e.g., "So, how about a trip to a hot spring in Hakone?") to the device again. The device displays the updated proposal to the user.

[0822] In this way, users can experience natural and optimized conversations regardless of time or place, and feedback based on the user's emotions makes communication even more natural.

[0823] Example 2

[0824] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0825] In communal living, time and space differences can easily lead to decreased communication and increased emotional distance. Conventional systems do not provide natural conversations that effectively utilize users' hobbies, interests, communication history, and real-time emotional information, which is an obstacle to smooth communication. To solve this problem, a system is needed that can collect and analyze a wide range of user information and provide natural conversations based on the results.

[0826] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0827] In this invention, the server includes means for collecting user hobbies and interests, means for collecting user communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and collecting emotional information, and means for updating the conversation in real time based on the collected emotional information. This makes it possible to utilize the user's multifaceted information and have a natural and optimized conversation based on the real-time reactions and emotional information.

[0828] "User" means an individual or organization that uses the system.

[0829] "Hobbies and Interests" refers to specific activities or subjects that interest a user, which information the user enters into the system.

[0830] "Communication history" refers to a record of messages and conversations that a user has had with other users in the past.

[0831] A "natural language processing model" is a type of machine learning model for analyzing, understanding, and generating text written in natural language.

[0832] "Generated natural language processing model" refers to a natural language processing model trained on collected data.

[0833] "Optimized conversation" refers to the most appropriate and natural conversation content generated based on the user's hobbies, interests, communication history, and emotional information.

[0834] "Real-time response" refers to the instantaneous reactions and feedback a user gives while interacting with a system.

[0835] "Emotional information" refers to the emotional state analyzed from the user's facial expression, tone of voice, etc.

[0836] "Updating the conversation in real time" means dynamically changing the content of the conversation based on the user's real-time reactions and emotional information.

[0837] Overall system overview

[0838] This invention is a system designed to resolve time and space differences in communal living and provide smoother communication. This system trains a natural language processing model based on user-entered hobbies, interests, communication history, and emotional information, and generates optimized conversations.

[0839] Hardware and software used

[0840] Server: Responsible for storing data, training models, and generating conversations

[0841] Terminal: Responsible for inputting information from the user, collecting real-time emotion data, and displaying the generated conversation content

[0842] Emotion Engine: Software for analyzing emotional information from the user's facial expressions and voice

[0843] Natural language processing model: A machine learning model that generates conversations based on collected data.

[0844] Data collection

[0845] 1. Collecting hobbies and interests

[0846] When a user registers for the service, they input their hobbies and interests. The device receives this input information and sends it to the server.

[0847] 2. Collection of communication history

[0848] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[0849] 3. Collecting emotional information

[0850] The device analyzes the user's facial expressions and voice in real time using an emotion engine and sends the results to a server, which then stores the received emotion information in a database.

[0851] Model training

[0852] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected by the server. The data is used to repeatedly train the model, allowing it to learn conversation patterns optimized for specific users.

[0853] Conversation Generation

[0854] The user enters a conversation topic or question into the device (e.g., "Where do you want to go on your next vacation?"). The device sends this information to the server, which uses a generative AI model to generate an optimized response (e.g., "How about that trip to Hokkaido we were talking about earlier?"). The server sends the generated response to the device, which displays it to the user.

[0855] Real-time adjustments

[0856] 1. Real-time reaction analysis

[0857] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives this response and sends it to the server.

[0858] 2. Emotional Information Analysis

[0859] The device analyzes the user's facial expressions and tone of voice in real time to determine their emotional state, and sends the analysis results to the server.

[0860] 3. Update the conversation

[0861] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model. The server updates the conversation content based on this feedback and sends it back to the device (e.g., "So, how about a hot spring trip to Hakone?"). The device then displays the updated suggestions to the user.

[0862] Specific examples

[0863] 1. Input Phase

[0864] User A enters "Where do you want to go on your next holiday?" into the device. The device sends this information to the server.

[0865] 2. Conversation generation phase

[0866] The server generates a conversation based on past conversation history and hobby information, asking, "How about that trip to Hokkaido we talked about earlier?" The device displays the generated conversation to User A.

[0867] 3. Real-time reaction and emotional information collection

[0868] User A responds, "That's good, but I'm concerned about the budget." The device sends this response to the server. The device uses an emotion engine to analyze the user's facial expressions and voice, and sends that information to the server.

[0869] 4. Conversation Update Phase

[0870] The server analyzes the real-time reactions and emotional information and updates the suggestion to "How about a trip to a hot spring in Hakone?" The device displays the updated suggestion to User A.

[0871] Prompt Sentence Examples

[0872] "Where do you want to go on your next vacation? Get travel suggestions based on your hobbies and past communication history."

[0873] By implementing the present invention in this way, users can experience optimized, natural conversations regardless of time or place. Real-time emotion analysis also enables more appropriate and satisfying communication.

[0874] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0875] Step 1: User Registration

[0876] Input: A user accesses the system and enters the required information to create a new account.

[0877] Output: The terminal receives the user information and sends it to the server.

[0878] Specific behavior:

[0879] A user enters their name, email address, password, etc. into an input form.

[0880] The terminal sends this information to the server via the send button.

[0881] The server stores the received information in a database and creates a user account.

[0882] Step 2: Enter your hobbies and interests

[0883] Input: The user enters their hobbies and interests.

[0884] Output: The device receives this information and sends it to the server.

[0885] Specific behavior:

[0886] Users fill out a multiple-choice form to select activities or topics that interest them.

[0887] The terminal checks the input information and sends it to the server.

[0888] The server stores the received hobbies and interests in a database.

[0889] Step 3: Collect communication history

[0890] Input: Get user consent.

[0891] Output: The server collects past conversation history from communication history and messaging apps and stores it in a database.

[0892] Specific behavior:

[0893] The user presses the consent button regarding the collection of communication history.

[0894] The server collects past communication history through the messaging app's API.

[0895] The collected data is stored in a database.

[0896] Step 4: Collecting emotional information

[0897] Input: The device captures the user's facial expressions and voice in real time.

[0898] Output: The device analyzes these data using the emotion engine and sends the results to the server.

[0899] Specific behavior:

[0900] The device uses a camera and microphone to capture the user's facial expressions and voice.

[0901] The device uses an emotion engine to analyze facial expressions and voice in real time.

[0902] The analysis results are sent to a server, which stores the information in a database.

[0903] Step 5: Data Preparation

[0904] Input: The server collects information about the user's hobbies, interests, communication history, and emotional information.

[0905] Output: Combined into a dataset and used for model training.

[0906] Specific behavior:

[0907] The server extracts the necessary data from the database and performs preprocessing.

[0908] Integrate the data to create a dataset for model training.

[0909] Step 6: Model training

[0910] Input: Dataset.

[0911] Output: A trained natural language processing model.

[0912] Specific behavior:

[0913] The server uses the dataset to train a natural language processing model.

[0914] During the training process, the model parameters are optimized and the user is trained to learn optimized conversation patterns.

[0915] Step 7: User Input

[0916] Input: The user types a question or conversation topic into the device (e.g., "Where do you want to go on your next vacation?").

[0917] Output: The device receives input and sends it to the server.

[0918] Specific behavior:

[0919] The user enters a question into the input form on the terminal and presses the send button.

[0920] The terminal transmits the user input as text data to the server.

[0921] Step 8: Conversation Generation

[0922] Input: User questions and conversation topics.

[0923] Output: The server uses the generative AI model to generate an optimized response and sends it to the device.

[0924] Specific behavior:

[0925] The server receives the user's question and compares it with historical data.

[0926] Generative AI models are used to generate appropriate responses (e.g., "How about that trip to Hokkaido we talked about earlier?").

[0927] The generated response is sent to the terminal, which displays it to the user.

[0928] Step 9: Analyze real-time responses

[0929] Input: Real-time user response (e.g., "That sounds good, but I'm concerned about the budget").

[0930] Output: The device receives the response and sends it to the server.

[0931] Specific behavior:

[0932] Users provide real-time feedback on the generated conversation.

[0933] The device captures the response and sends it to the server.

[0934] Step 10: Emotional Analysis

[0935] Input: User facial and voice data.

[0936] Output: The analysis results of the emotion engine are sent to the server.

[0937] Specific behavior:

[0938] The device uses a camera and microphone to capture the user's facial expressions and voice data.

[0939] The emotion engine analyzes this data in real time and extracts emotional information.

[0940] The analysis results are sent to the server.

[0941] Step 11: Update your conversation

[0942] Input: Real-time user reaction and emotional information.

[0943] Output: Sends the updated conversation to the terminal.

[0944] Specific behavior:

[0945] The server analyzes real-time reactions and emotional information.

[0946] Feedback is fed into generative AI models to optimize and update conversation content.

[0947] The updated conversation content is sent to the terminal, which displays it to the user (e.g., "So, how about a trip to the hot springs in Hakone?").

[0948] (Application example 2)

[0949] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0950] In communal living, time and space differences can reduce communication and increase emotional distance. Furthermore, users' shopping experiences in virtual stores are not personalized, which can lead to unexpected frustration and stress. To address these issues, a system is needed that can generate optimized conversations based on users' hobbies, interests, real-time reactions, and emotional information, thereby improving the user's shopping experience.

[0951] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0952] In this invention, the server includes means for collecting user hobbies and interests, means for collecting user communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and updating the conversation based on the reactions, means for collecting and analyzing emotional information such as the user's facial expression information and tone of voice, and means for adjusting the conversation based on the analyzed emotional information. This makes it possible to realize more personalized and natural conversation based on the user's real-time emotions and reactions, and improve the user experience in the virtual store.

[0953] "Hobbies and interests" refers to the things, activities, and preferences that a user is interested in.

[0954] "Communication history" refers to a record of past conversations and messages between users.

[0955] A "natural language processing model" refers to a statistical machine learning model for understanding and generating human language.

[0956] "Real-time reactions" refers to the immediate reactions and responses that users give during a conversation.

[0957] "Facial expression information" refers to data relating to emotions obtained from the user's facial expressions.

[0958] "Voice tone" refers to information about the characteristics of the voice, such as the pitch and volume of the voice.

[0959] "Emotional information" refers to data that represents the user's emotional state, such as facial expressions and tone of voice.

[0960] The "means for adjusting the conversation" refers to a method for updating the generated conversation content based on the user's real-time reactions and emotional information.

[0961] To implement this invention, it is necessary to build a system that collects users' hobbies and interests, communication history, real-time reactions, and emotional information, and generates optimized conversations based on the collected information. This system includes the following main elements:

[0962] First, to collect the user's hobbies and interests, a method is required for the user to input these when logging in to the application. Also, with the user's consent, communication history is collected and this data is sent to the server, allowing the server to manage user information in an integrated manner.

[0963] The server trains a natural language processing model based on the collected hobbies, interests, and communication history. Specifically, it uses Python and TensorFlow to build and train a generative AI model. This model is used to learn the user's personalized conversation patterns and generate optimized conversations.

[0964] Next, to collect real-time user reactions and emotional information, devices such as smartphones and head-mounted displays must be equipped with a camera and microphone. These devices use OpenCV and emotion analysis libraries to analyze the user's facial expressions and tone of voice, and transmit this information to the server as emotional information.

[0965] Based on this emotional information and real-time reactions, the server provides feedback to the generative AI model to adjust and update the conversation content. Finally, the adjusted conversation content is sent to the device and displayed to the user.

[0966] As a concrete example, in a virtual shopping assistant scenario, if a user asks, "Do you recommend these shoes?", the server uses a generative AI model to respond, "How about these shoes that suit your style?" Furthermore, if the user responds, "I'd like to see reviews too," the server analyzes real-time emotional information and generates a tailored response, such as, "Here are reviews from other users. They look great!"

[0967] Example prompt sentence:

[0968] A user asked 'Do you recommend these shoes?'

[0969] The realization of this system will enable more personalized and natural conversations to be provided based on the user's real-time emotions and reactions, improving the user experience in virtual stores.

[0970] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0971] Step 1:

[0972] The device receives input from the user about hobbies and interests, which are entered when the user logs into the application, and transmits the input information to the server.

[0973] Input: User's hobbies and interests

[0974] Output: User's hobbies and interests sent to the server

[0975] Specific operation: Using the UI / UX on the device, the user fills in an input form about their hobbies and interests. By pressing the input button, the data is sent to the server.

[0976] Step 2:

[0977] The server collects the user's communication history and, with the user's consent, analyzes past conversation history from messaging apps and stores it in a database.

[0978] Input: User consent, communication history

[0979] Output: Communication history stored in the database

[0980] Specific operation: After the server confirms the user's consent, it obtains communication history from the messaging app and stores the analyzed data in a database.

[0981] Step 3:

[0982] The server trains a natural language processing model based on collected hobbies, interests, and communication history. It uses Python and TensorFlow to build and train a generative AI model that learns the user's personalized speech patterns.

[0983] Input: Hobbies and interests, communication history

[0984] Output: A trained natural language processing model

[0985] How it works: A generative AI model is built using TensorFlow, and the model is trained using collected data. After multiple learning cycles, the model learns optimal conversation patterns.

[0986] Step 4:

[0987] The device collects the user's real-time reactions and emotional information and sends it to the server. Using a smartphone or head-mounted display, facial expression information and tone of voice are collected from the camera and microphone, and analyzed using OpenCV and the emotion analysis library.

[0988] Input: Real-time user reactions, facial expressions, and tone of voice

[0989] Output: Emotion information sent to the server

[0990] Specific operation: The device's camera and microphone are used to record the user's reactions, facial expressions are analyzed using OpenCV, emotional states are detected using an emotion analysis library, and the data is sent to the server.

[0991] Step 5:

[0992] The server provides feedback to the generative AI model based on real-time reactions and emotional information, adjusting and updating the conversation content. The adjusted conversation content is then sent to the device and displayed to the user.

[0993] Input: Real-time reactions, emotional information

[0994] Output: Adjusted conversation

[0995] How it works: The server analyzes the received emotional information and reactions and provides feedback to the generative AI model. The model generates an adjusted conversation and sends it to the device. The device then displays the updated conversation to the user.

[0996] Step 6:

[0997] The device displays the adjusted conversation content to the user, who can then further react to the displayed conversation content, which becomes the starting point for the next conversation.

[0998] Input: Modified conversation

[0999] Output: The conversation as it appears to the user

[1000] Specific operation: The adjusted conversation content is presented to the user on the device display screen. The user reads it and gives a new response, which leads to the next conversation process.

[1001] This allows the system as a whole to generate more personalized and natural conversations based on the user's real-time emotions and reactions.

[1002] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1003] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1004] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1005] [Third embodiment]

[1006] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1007] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1008] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1009] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1010] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1011] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1012] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1013] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1014] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1015] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1016] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1017] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1018] This invention relates to a system that uses generative AI to provide natural conversations in order to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[1019] Overall system overview

[1020] The system trains a natural language processing model based on user input such as hobbies, interests, and communication history, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses.

[1021] Data collection

[1022] 1. Collecting hobbies and interests

[1023] When a user registers for the service, they enter their hobbies and interests.

[1024] The terminal receives this input information and sends it to the server.

[1025] 2. Collection of communication history

[1026] With the user's consent, the server collects communication history and conversation history from messaging apps.

[1027] The server stores the collected data in a database.

[1028] Model training

[1029] The server trains a natural language processing model based on the user data collected.

[1030] The server iteratively trains the model with data to learn speech patterns optimized for individual users.

[1031] Conversation Generation

[1032] The user inputs a conversation topic or question into the terminal.

[1033] The device sends the input to the server.

[1034] The server uses the generative AI model to generate optimized conversations and sends the responses to the device.

[1035] The terminal displays the received reply to the user.

[1036] Real-time adjustments

[1037] 1. Real-time reaction analysis

[1038] Users react in real time to the generated conversation (e.g., by typing text or responding with voice).

[1039] The device receives real-time responses and transmits them to the server.

[1040] 2. Update the conversation

[1041] The server analyzes the user's real-time responses and sends feedback to the generative AI model.

[1042] The server updates the conversation content based on the feedback and sends it back to the device.

[1043] Specific examples

[1044] Travel plan example

[1045] 1. Input Phase

[1046] User A types into the terminal, "Where do you want to go on your next holiday?"

[1047] The device sends input to the server.

[1048] 2. Conversation generation phase

[1049] The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[1050] The terminal displays the generated conversation to User A.

[1051] 3. Real-time response

[1052] User A responds, "That's good, but I'm concerned about the budget."

[1053] The device sends this response to the server.

[1054] 4. Conversation Update Phase

[1055] The server analyzes the real-time responses and updates the suggestion with, "So, how about a trip to a hot spring in Hakone?"

[1056] The device displays the updated proposal to User A.

[1057] In this way, the system of the present invention can provide natural conversations that make people feel as if they are together in the same place, regardless of time or place, and can adjust to eliminate the sense of time and space discrepancies in communal living.

[1058] The processing flow will be explained below.

[1059] Step 1:

[1060] When users register for the service, they enter their hobbies and interests. The device receives this information and sends it to the server.

[1061] Step 2:

[1062] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[1063] Step 3:

[1064] The server trains a natural language processing model based on the user's hobbies, interests, and communication history collected by the server. The server repeatedly trains the model using data to learn conversation patterns optimized for each individual user.

[1065] Step 4:

[1066] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[1067] Step 5:

[1068] The server uses the generative AI model to generate an optimized conversation (e.g., "How about the trip to Hokkaido we talked about earlier?") and sends the response to the device. The device then displays the received response to the user.

[1069] Step 6:

[1070] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives the real-time response and sends it to the server.

[1071] Step 7:

[1072] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on the feedback.

[1073] Step 8:

[1074] The server sends the updated conversation (e.g., "So, how about a trip to a hot spring in Hakone?") to the device again. The device displays the updated proposal to the user.

[1075] In this way, users are provided with natural and optimized conversations regardless of time or place.

[1076] Example 1

[1077] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1078] In modern society, differences in time and space in communal living often lead to a decrease in communication and an increase in emotional distance, making it difficult for family and friends to interact smoothly and potentially leading to estrangement.

[1079] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1080] In this invention, the server includes means for inputting the user's hobbies and interests, means for collecting the user's communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and updating the conversation based on the reactions, and means for saving the collected data in a database. This makes it possible to provide natural conversation regardless of time or place, and to adjust so that time and space differences in communal living are not felt.

[1081] "User" refers to a person who uses the system.

[1082] "Hobbies and interests" refers to information about activities or topics that interest a user.

[1083] "Communication history" refers to a record of a user's past digital communications, such as messages, emails, and chats.

[1084] "Natural language processing model" refers to an algorithm or machine learning model for understanding and generating human language.

[1085] A "server" refers to a computer device that processes the entire system, stores and analyzes data, trains models, and so on.

[1086] "Terminal" refers to a hardware device that is directly operated by a user (e.g., smartphone, tablet, PC).

[1087] "Database" refers to a system for efficiently storing and managing collected data.

[1088] "Training a model" refers to the process of using collected data to refine and improve the performance of a natural language processing model.

[1089] "Optimized conversation" refers to conversation content that is easy to understand and relevant to the user, generated based on the user's hobbies, interests, and communication history.

[1090] "Real-time responses" refers to the instantaneous inputs and responses (e.g., text input, voice responses) a user makes while interacting with a system.

[1091] "Updating the conversation" refers to changing or improving the conversation content generated by the system based on the user's real-time reactions.

[1092] A "conversation generation scenario" refers to the flow and structure of a conversation with a user that is created based on collected data.

[1093] "Evaluating" refers to the process of determining how well the generated conversation matches the user's needs and expectations.

[1094] This invention relates to a system that utilizes a generative AI model to provide natural conversations in order to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[1095] Overall system overview

[1096] This system trains a natural language processing model based on user input such as hobbies, interests, and communication history, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses.

[1097] Data collection

[1098] Collecting hobbies and interests

[1099] When a user registers for the service, they enter their hobbies and interests.

[1100] The terminal receives this input information and sends it to the server.

[1101] Collection of communication history

[1102] With the user's consent, the server collects communication history and conversation history from messaging apps.

[1103] The server stores the collected data in a database.

[1104] Example: User B enters "watching movies" and "hiking" in the hobby field, and the server collects his past message data and stores it in a database.

[1105] Training the model

[1106] The server uses the stored user data to train a natural language processing model, using TensorFlow and PyTorch as the specific software.

[1107] The server learns conversation patterns optimized for individual users.

[1108] Example: User B's hobby data and past communication history are used to train a natural language processing model (e.g., GPT-3) to learn optimal conversation patterns about movies and hiking.

[1109] Conversation Generation

[1110] The user inputs a specific topic or question into the terminal.

[1111] The terminal sends the input contents to the server.

[1112] The server uses a trained generative AI model to generate optimized conversations.

[1113] The server sends the generated response to the terminal, which displays it to the user.

[1114] Example: User A types into his terminal, "Where do you want to go on your next holiday?" The server responds by saying, "How about that trip to Hokkaido we talked about earlier?" and the terminal displays this to User A.

[1115] Real-time adjustments

[1116] Real-time reaction analysis

[1117] Users react to generated conversations in real time.

[1118] The device receives the real-time response and sends it to the server.

[1119] Update the conversation

[1120] The server analyzes the user's real-time responses and sends feedback to the generative AI model.

[1121] The server updates the conversation content based on the feedback and sends it back to the device.

[1122] The terminal displays the updated content to the user.

[1123] Example: User A responds, "That sounds good, but I'm concerned about the budget," and the server analyzes this and updates the suggestion to, "Then, how about a trip to a hot spring in Hakone?", which the device displays to User A.

[1124] Examples of prompt statements

[1125] User A: "Where do you want to go on your next vacation?"

[1126] Server: "How was that trip to Hokkaido you were talking about earlier?"

[1127] User A: "That's good, but I'm concerned about the budget."

[1128] Server: "So, how about a trip to the hot springs in Hakone?"

[1129] This system allows for natural conversation regardless of time or place, eliminating the sense of time or space gaps in communal living and enabling smooth communication.

[1130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1131] Step 1: User Registration

[1132] The user accesses the service for the first time and creates an account. The device displays a registration form to the user, where the user enters their name, email address, and hobbies and interests. The device then sends the information to the server.

[1133] Input: Name, email address, hobbies and interests

[1134] Output: User registration data (saved in database)

[1135] Step 2: Collect communication history

[1136] With the user's consent, the server collects communication history from messaging apps, emails, etc. The server stores this data in a database.

[1137] Input: User consent, communication history

[1138] Output: Collected communication history data (stored in a database)

[1139] Step 3: Save data

[1140] The server stores the user's hobbies and interests and the collected communication history in a database, confirms that the storage was successful, and sends a success message to the terminal.

[1141] Input: Hobbies and interests, communication history

[1142] Output: Data saved to the database, message that save was successful

[1143] Step 4: Train the model

[1144] The server uses the stored data to train a natural language processing model, specifically using TensorFlow and PyTorch to train the model and have it learn conversation patterns optimized for the user.

[1145] Input: User data in the database

[1146] Output: A trained natural language processing model

[1147] Step 5: Enter the conversation topic

[1148] The user inputs a conversation topic or question into the terminal, and the terminal transmits the input content to the server.

[1149] Input: A topic or question entered by the user

[1150] Output: Send data to the server

[1151] Step 6: Conversation generation

[1152] The server uses the trained generative AI model to generate optimized conversations, and sends the generated responses to the device, which then displays them to the user.

[1153] Input: The subject or question sent to the server

[1154] Output: Generated conversation (sent to terminal)

[1155] Step 7: Analyzing real-time responses

[1156] The user reacts to the generated conversation in real time, and the device transmits the reaction to the server.

[1157] Input: Real-time user response

[1158] Output: Sends response data to the server

[1159] Step 8: Update the conversation

[1160] The server analyzes the user's real-time responses and sends feedback to the generative AI model. The conversation content is updated based on that feedback, and the newly generated content is sent to the device. The device then displays the updated content to the user.

[1161] Input: Real-time user response

[1162] Output: Updated conversation (sent to terminal)

[1163] Specific operation example

[1164] 1. User A types into the terminal, "Where do you want to go on your next holiday?"

[1165] 2. The device sends this input to the server.

[1166] 3. The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[1167] 4. The device displays the generated conversation to User A.

[1168] 5. User A responds, "That's fine, but I'm concerned about the budget."

[1169] 6. The device sends this response to the server.

[1170] 7. The server analyzes the real-time response and updates the suggestion with, "How about a trip to a hot spring in Hakone?"

[1171] 8. The device displays the updated proposal to User A.

[1172] (Application example 1)

[1173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1174] The present invention aims to improve customer satisfaction by facilitating smooth communication between customers and store clerks in brick-and-mortar stores. In particular, there is a need to strengthen relationships of trust with customers by providing personalized services and making appropriate product recommendations based on the customer's preferences and past purchase history. However, conventional technologies have had difficulty optimizing real-time conversations and generating responses tailored to individual customer needs.

[1175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1176] In this invention, the server includes: means for collecting user hobbies and interests; means for collecting user communication history; means for training a natural language processing model based on the collected hobbies and interests and communication history; means for generating an optimized conversation using the generated natural language processing model; means for analyzing the user's real-time reactions and updating the conversation based on the reactions; means for generating a conversation between a customer and a store clerk to support interactions in a physical store; and means for displaying the generated conversation using an application installed on smart glasses or a head-mounted display. This enables smoother interactions with customers in physical stores and more personalized services to be provided.

[1177] The "means for collecting user's hobbies and interests" is a function for recording information on hobbies and interests entered by the user and saving it as data.

[1178] "Means for collecting a user's communication history" refers to a function for obtaining and analyzing past communication history, such as text messages and call records, with the user's consent.

[1179] "Means for training natural language processing models based on collected hobbies, interests, and communication history" refers to a function that uses collected user data to optimize natural language processing models using machine learning and deep learning algorithms.

[1180] "Means for generating optimized conversations using the generated natural language processing model" is a function that uses a trained natural language processing model to generate optimal conversations based on the user's interests and past communication history.

[1181] "Means for analyzing the user's real-time responses and updating the conversation based on those responses" refers to a function that analyzes the real-time text and voice responses entered by the user and updates the content of the conversation in a timely manner based on that information.

[1182] "A means for generating conversations between customers and store clerks to support interactions in physical stores" is a function that supports conversations between customers and store clerks in physical stores and makes appropriate product suggestions tailored to the customer's needs.

[1183] "Means for displaying the generated conversation using an application installed on smart glasses or a head-mounted display" refers to a function for displaying the generated conversation content using an application installed on a wearable device such as smart glasses or a head-mounted display.

[1184] MODE FOR CARRYING OUT THE INVENTION

[1185] The present invention is designed to facilitate communication between customers and store clerks in physical stores and to provide personalized services. Specific embodiments for carrying out the present invention will be described below.

[1186] Overall system overview

[1187] This system collects users' hobbies, interests, and communication history, and uses them to train a natural language processing model to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses. The system primarily consists of a server, terminals, and wearable devices such as smart glasses and head-mounted displays.

[1188] Data collection

[1189] Collecting hobbies and interests

[1190] When a user (customer) registers for the service, they enter their hobbies and interests into a terminal. This information is sent to the server via the terminal and stored in a database.

[1191] Collection of communication history

[1192] With the user's consent, the server collects communication history and past conversation history from messaging apps, and this data is also stored in the server's database.

[1193] Model training

[1194] The server trains a natural language processing model based on the collected user information, such as hobbies, interests, and communication history. This requires machine learning and deep learning algorithms, such as OpenAI's GPT-3.

[1195] Conversation Generation

[1196] The user (store clerk) uses an application installed on smart glasses or a head-mounted display to input conversation topics and questions. This input information is sent to the server via the device. The server uses a generative AI model to generate an optimized conversation and sends the response to the device. The device then displays the received response to the user (store clerk).

[1197] Examples of prompt statements

[1198] When the user (store clerk) types "Looking for new sneakers?", the server generates the following:

[1199] Generates "The Nike model you purchased last time is very popular."

[1200] Real-time adjustments

[1201] Real-time reaction analysis

[1202] The user (customer) responds to the generated conversation in real time (for example, by inputting text or responding with voice). The terminal receives the real-time response and sends it to the server.

[1203] Update the conversation

[1204] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on this feedback and sends it back to the device.

[1205] Examples of prompt statements

[1206] If the user (customer) responds with "I really like that model, but what else do you recommend?" the server generates:

[1207] "We have new color variations in stock, and we also recommend the latest Adidas models," says Genesis.

[1208] Examples of application in physical stores

[1209] This system will be used to support interactions between customers and store clerks in physical stores. For example, if a customer asks for an explanation of a specific product, the store clerk can use the system through the smart glasses to provide the customer with the most appropriate product explanation and suggestions. This allows customers to receive personalized service based on their preferences and past purchasing history, leading to increased customer satisfaction.

[1210] In this way, the system of the present invention significantly improves the quality of service in physical stores.

[1211] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1212] Step 1:

[1213] When a user (customer) registers for the service, they input information about their hobbies and interests through their device. The device then sends this information to the server, which then stores the data on their hobbies and interests in a database.

[1214] Input: User's hobbies and interests

[1215] Output: Hobbies and interests data stored in a database

[1216] Specific operation: The device acquires the user's hobbies and interests and sends them to the server.

[1217] Step 2:

[1218] With the user's consent, the server collects communication history, such as text messages and past conversation history, and stores it in a database.

[1219] Input: User's communication history

[1220] Output: Communication history stored in the database

[1221] Specific operation: The server collects and stores data from messaging apps and communication history.

[1222] Step 3:

[1223] The server uses the collected hobbies, interests, and communication history to train a natural language processing model, using machine learning algorithms to learn patterns from the training data.

[1224] Input: Hobbies and interests, communication history

[1225] Output: A trained natural language processing model

[1226] Specific operation: The server uses a natural language processing model such as GPT-3 to train the model based on the collected data.

[1227] Step 4:

[1228] The user (store clerk) inputs the conversation topic or question into the smart glasses or head-mounted display, which is then sent from the device to the server. The server uses a generative AI model to generate an optimized conversation and sends the response to the device.

[1229] Input: conversation topic or question

[1230] Output: Generated conversational responses

[1231] Specific operation: The device sends the conversation topic to the server, and the server uses a generative AI model to generate a conversation and provide a response.

[1232] Step 5:

[1233] The terminal receives and displays the generated conversation to the user (store clerk) through smart glasses or a head-mounted display.

[1234] Input: Generated conversation response

[1235] Output: The conversation displayed to the user

[1236] Specific operation: The device receives the generated conversation from the server and displays it on smart glasses or a head-mounted display.

[1237] Step 6:

[1238] The user (customer) responds to the generated conversation in real time and inputs the response into the terminal, which then transmits the real-time response to the server.

[1239] Input: Real-time response (text or voice)

[1240] Output: Real-time responses sent to the server

[1241] Specific operation: The process in which the terminal receives real-time responses from the user (customer) and sends them to the server.

[1242] Step 7:

[1243] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on this feedback and sends it back to the device.

[1244] Input: Analyzed real-time responses

[1245] Output: Updated conversation

[1246] Specific operation: The server analyzes real-time responses, updates the generative AI model, generates new conversation content, and sends it to the device.

[1247] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1248] This invention relates to a system that provides natural conversations by utilizing generative AI and an emotion engine to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[1249] Overall system overview

[1250] The system trains a natural language processing model based on user-entered hobbies, interests, communication history, and emotional information, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on real-time user reactions and emotional analysis.

[1251] Data collection

[1252] 1. Collecting hobbies and interests

[1253] When a user registers for the service, they enter their hobbies and interests.

[1254] The terminal receives this input information and sends it to the server.

[1255] 2. Collection of communication history

[1256] With the user's consent, the server collects past conversation history between users from communication history and messaging apps.

[1257] The server stores the collected data in a database.

[1258] 3. Collecting emotional information

[1259] The device analyzes the user's emotional information in real time using an emotion engine and sends it to the server.

[1260] Model training

[1261] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected.

[1262] The server iteratively trains the model with data to learn speech patterns optimized for individual users.

[1263] Conversation Generation

[1264] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[1265] The server uses the generative AI model to generate an optimized conversation (e.g., "How was the trip to Hokkaido we talked about earlier?") and sends the response to the device.

[1266] The terminal displays the received response to the user.

[1267] Real-time adjustments

[1268] 1. Real-time reaction analysis

[1269] Users respond in real time to generated conversations (e.g., "That's good, but I'm concerned about the budget").

[1270] The device receives the real-time response and sends it to the server.

[1271] 2. Emotional Information Analysis

[1272] The device uses an emotion engine to analyze emotional information from the user's facial expressions and voice in real time, such as facial expressions and tone of voice when the user shows interest.

[1273] The device transmits the analyzed emotional information to the server.

[1274] 3. Update the conversation

[1275] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model.

[1276] The server updates the conversation content based on the feedback and sends it back to the device.

[1277] Specific examples

[1278] Travel plan example

[1279] 1. Input Phase

[1280] User A types into the terminal, "Where do you want to go on your next holiday?"

[1281] The device sends input to the server.

[1282] 2. Conversation generation phase

[1283] The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[1284] The terminal displays the generated conversation to User A.

[1285] 3. Real-time reaction and emotional information collection

[1286] User A responds, "That's good, but I'm concerned about the budget." The device sends the response in real time to the server.

[1287] The device uses an emotion engine to analyze the user's facial expressions and voice to determine whether they are interested, and then sends that information to the server.

[1288] 4. Conversation Update Phase

[1289] The server analyzes real-time reactions and emotional information and updates the suggestion with, "So, how about a hot spring trip to Hakone?"

[1290] The device displays the updated proposal to User A.

[1291] In this way, the system of the present invention can provide natural and optimized conversations based on the user's emotions, regardless of time or place, and can adjust to eliminate any sense of time or space discrepancy in communal living.

[1292] The processing flow will be explained below.

[1293] Step 1:

[1294] When users register for the service, they enter their hobbies and interests. The device receives this information and sends it to the server.

[1295] Step 2:

[1296] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[1297] Step 3:

[1298] The device analyzes the user's emotional information in real time using an emotion engine and transmits it to the server, including the user's facial expressions, tone of voice, and physical movements.

[1299] Step 4:

[1300] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected by the server. The server repeatedly trains the model using data to learn conversation patterns optimized for each individual user.

[1301] Step 5:

[1302] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[1303] Step 6:

[1304] The server uses the generative AI model to generate an optimized conversation (e.g., "How about the trip to Hokkaido we talked about earlier?") and sends the response to the device. The device then displays the received response to the user.

[1305] Step 7:

[1306] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives the real-time response and sends it to the server.

[1307] Step 8:

[1308] The device analyzes emotional information from the user's facial expressions and voice in real time using an emotion engine and sends the information to the server.

[1309] Step 9:

[1310] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model, which then updates the conversation content based on the feedback.

[1311] Step 10:

[1312] The server sends the updated conversation (e.g., "So, how about a trip to a hot spring in Hakone?") to the device again. The device displays the updated proposal to the user.

[1313] In this way, users can experience natural and optimized conversations regardless of time or place, and feedback based on the user's emotions makes communication even more natural.

[1314] Example 2

[1315] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1316] In communal living, time and space differences can easily lead to decreased communication and increased emotional distance. Conventional systems do not provide natural conversations that effectively utilize users' hobbies, interests, communication history, and real-time emotional information, which is an obstacle to smooth communication. To solve this problem, a system is needed that can collect and analyze a wide range of user information and provide natural conversations based on the results.

[1317] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1318] In this invention, the server includes means for collecting user hobbies and interests, means for collecting user communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and collecting emotional information, and means for updating the conversation in real time based on the collected emotional information. This makes it possible to utilize the user's multifaceted information and have a natural and optimized conversation based on the real-time reactions and emotional information.

[1319] "User" means an individual or organization that uses the system.

[1320] "Hobbies and Interests" refers to specific activities or subjects that interest a user, which information the user enters into the system.

[1321] "Communication history" refers to a record of messages and conversations that a user has had with other users in the past.

[1322] A "natural language processing model" is a type of machine learning model for analyzing, understanding, and generating text written in natural language.

[1323] "Generated natural language processing model" refers to a natural language processing model trained on collected data.

[1324] "Optimized conversation" refers to the most appropriate and natural conversation content generated based on the user's hobbies, interests, communication history, and emotional information.

[1325] "Real-time response" refers to the instantaneous reactions and feedback a user gives while interacting with a system.

[1326] "Emotional information" refers to the emotional state analyzed from the user's facial expression, tone of voice, etc.

[1327] "Updating the conversation in real time" means dynamically changing the content of the conversation based on the user's real-time reactions and emotional information.

[1328] Overall system overview

[1329] This invention is a system designed to resolve time and space differences in communal living and provide smoother communication. This system trains a natural language processing model based on user-entered hobbies, interests, communication history, and emotional information, and generates optimized conversations.

[1330] Hardware and software used

[1331] Server: Responsible for storing data, training models, and generating conversations

[1332] Terminal: Responsible for inputting information from the user, collecting real-time emotion data, and displaying the generated conversation content

[1333] Emotion Engine: Software for analyzing emotional information from the user's facial expressions and voice

[1334] Natural language processing model: A machine learning model that generates conversations based on collected data.

[1335] Data collection

[1336] 1. Collecting hobbies and interests

[1337] When a user registers for the service, they input their hobbies and interests. The device receives this input information and sends it to the server.

[1338] 2. Collection of communication history

[1339] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[1340] 3. Collecting emotional information

[1341] The device analyzes the user's facial expressions and voice in real time using an emotion engine and sends the results to a server, which then stores the received emotion information in a database.

[1342] Model training

[1343] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected by the server. The data is used to repeatedly train the model, allowing it to learn conversation patterns optimized for specific users.

[1344] Conversation Generation

[1345] The user enters a conversation topic or question into the device (e.g., "Where do you want to go on your next vacation?"). The device sends this information to the server, which uses a generative AI model to generate an optimized response (e.g., "How about that trip to Hokkaido we were talking about earlier?"). The server sends the generated response to the device, which displays it to the user.

[1346] Real-time adjustments

[1347] 1. Real-time reaction analysis

[1348] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives this response and sends it to the server.

[1349] 2. Emotional Information Analysis

[1350] The device analyzes the user's facial expressions and tone of voice in real time to determine their emotional state, and sends the analysis results to the server.

[1351] 3. Update the conversation

[1352] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model. The server updates the conversation content based on this feedback and sends it back to the device (e.g., "So, how about a hot spring trip to Hakone?"). The device then displays the updated suggestions to the user.

[1353] Specific examples

[1354] 1. Input Phase

[1355] User A enters "Where do you want to go on your next holiday?" into the device. The device sends this information to the server.

[1356] 2. Conversation generation phase

[1357] The server generates a conversation based on past conversation history and hobby information, asking, "How about that trip to Hokkaido we talked about earlier?" The device displays the generated conversation to User A.

[1358] 3. Real-time reaction and emotional information collection

[1359] User A responds, "That's good, but I'm concerned about the budget." The device sends this response to the server. The device uses an emotion engine to analyze the user's facial expressions and voice, and sends that information to the server.

[1360] 4. Conversation Update Phase

[1361] The server analyzes the real-time reactions and emotional information and updates the suggestion to "How about a trip to a hot spring in Hakone?" The device displays the updated suggestion to User A.

[1362] Prompt Sentence Examples

[1363] "Where do you want to go on your next vacation? Get travel suggestions based on your hobbies and past communication history."

[1364] By implementing the present invention in this way, users can experience optimized, natural conversations regardless of time or place. Real-time emotion analysis also enables more appropriate and satisfying communication.

[1365] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1366] Step 1: User Registration

[1367] Input: A user accesses the system and enters the required information to create a new account.

[1368] Output: The terminal receives the user information and sends it to the server.

[1369] Specific behavior:

[1370] A user enters their name, email address, password, etc. into an input form.

[1371] The terminal sends this information to the server via the send button.

[1372] The server stores the received information in a database and creates a user account.

[1373] Step 2: Enter your hobbies and interests

[1374] Input: The user enters their hobbies and interests.

[1375] Output: The device receives this information and sends it to the server.

[1376] Specific behavior:

[1377] Users fill out a multiple-choice form to select activities or topics that interest them.

[1378] The terminal checks the input information and sends it to the server.

[1379] The server stores the received hobbies and interests in a database.

[1380] Step 3: Collect communication history

[1381] Input: Get user consent.

[1382] Output: The server collects past conversation history from communication history and messaging apps and stores it in a database.

[1383] Specific behavior:

[1384] The user presses the consent button regarding the collection of communication history.

[1385] The server collects past communication history through the messaging app's API.

[1386] The collected data is stored in a database.

[1387] Step 4: Collecting emotional information

[1388] Input: The device captures the user's facial expressions and voice in real time.

[1389] Output: The device analyzes these data using the emotion engine and sends the results to the server.

[1390] Specific behavior:

[1391] The device uses a camera and microphone to capture the user's facial expressions and voice.

[1392] The device uses an emotion engine to analyze facial expressions and voice in real time.

[1393] The analysis results are sent to a server, which stores the information in a database.

[1394] Step 5: Data Preparation

[1395] Input: The server collects information about the user's hobbies, interests, communication history, and emotional information.

[1396] Output: Combined into a dataset and used for model training.

[1397] Specific behavior:

[1398] The server extracts the necessary data from the database and performs preprocessing.

[1399] Integrate the data to create a dataset for model training.

[1400] Step 6: Model training

[1401] Input: Dataset.

[1402] Output: A trained natural language processing model.

[1403] Specific behavior:

[1404] The server uses the dataset to train a natural language processing model.

[1405] During the training process, the model parameters are optimized and the user is trained to learn optimized conversation patterns.

[1406] Step 7: User Input

[1407] Input: The user types a question or conversation topic into the device (e.g., "Where do you want to go on your next vacation?").

[1408] Output: The device receives input and sends it to the server.

[1409] Specific behavior:

[1410] The user enters a question into the input form on the terminal and presses the send button.

[1411] The terminal transmits the user input as text data to the server.

[1412] Step 8: Conversation Generation

[1413] Input: User questions and conversation topics.

[1414] Output: The server uses the generative AI model to generate an optimized response and sends it to the device.

[1415] Specific behavior:

[1416] The server receives the user's question and compares it with historical data.

[1417] Generative AI models are used to generate appropriate responses (e.g., "How about that trip to Hokkaido we talked about earlier?").

[1418] The generated response is sent to the terminal, which displays it to the user.

[1419] Step 9: Analyze real-time responses

[1420] Input: Real-time user response (e.g., "That sounds good, but I'm concerned about the budget").

[1421] Output: The device receives the response and sends it to the server.

[1422] Specific behavior:

[1423] Users provide real-time feedback on the generated conversation.

[1424] The device captures the response and sends it to the server.

[1425] Step 10: Emotional Analysis

[1426] Input: User facial and voice data.

[1427] Output: The analysis results of the emotion engine are sent to the server.

[1428] Specific behavior:

[1429] The device uses a camera and microphone to capture the user's facial expressions and voice data.

[1430] The emotion engine analyzes this data in real time and extracts emotional information.

[1431] The analysis results are sent to the server.

[1432] Step 11: Update your conversation

[1433] Input: Real-time user reaction and emotional information.

[1434] Output: Sends the updated conversation to the terminal.

[1435] Specific behavior:

[1436] The server analyzes real-time reactions and emotional information.

[1437] Feedback is fed into generative AI models to optimize and update conversation content.

[1438] The updated conversation content is sent to the terminal, which displays it to the user (e.g., "So, how about a trip to the hot springs in Hakone?").

[1439] (Application example 2)

[1440] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1441] In communal living, time and space differences can reduce communication and increase emotional distance. Furthermore, users' shopping experiences in virtual stores are not personalized, which can lead to unexpected frustration and stress. To address these issues, a system is needed that can generate optimized conversations based on users' hobbies, interests, real-time reactions, and emotional information, thereby improving the user's shopping experience.

[1442] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1443] In this invention, the server includes means for collecting user hobbies and interests, means for collecting user communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and updating the conversation based on the reactions, means for collecting and analyzing emotional information such as the user's facial expression information and tone of voice, and means for adjusting the conversation based on the analyzed emotional information. This makes it possible to realize more personalized and natural conversation based on the user's real-time emotions and reactions, and improve the user experience in the virtual store.

[1444] "Hobbies and interests" refers to the things, activities, and preferences that a user is interested in.

[1445] "Communication history" refers to a record of past conversations and messages between users.

[1446] A "natural language processing model" refers to a statistical machine learning model for understanding and generating human language.

[1447] "Real-time reactions" refers to the immediate reactions and responses that users give during a conversation.

[1448] "Facial expression information" refers to data relating to emotions obtained from the user's facial expressions.

[1449] "Voice tone" refers to information about the characteristics of the voice, such as the pitch and volume of the voice.

[1450] "Emotional information" refers to data that represents the user's emotional state, such as facial expressions and tone of voice.

[1451] The "means for adjusting the conversation" refers to a method for updating the generated conversation content based on the user's real-time reactions and emotional information.

[1452] To implement this invention, it is necessary to build a system that collects users' hobbies and interests, communication history, real-time reactions, and emotional information, and generates optimized conversations based on the collected information. This system includes the following main elements:

[1453] First, to collect the user's hobbies and interests, a method is required for the user to input these when logging in to the application. Also, with the user's consent, communication history is collected and this data is sent to the server, allowing the server to manage user information in an integrated manner.

[1454] The server trains a natural language processing model based on the collected hobbies, interests, and communication history. Specifically, it uses Python and TensorFlow to build and train a generative AI model. This model is used to learn the user's personalized conversation patterns and generate optimized conversations.

[1455] Next, to collect real-time user reactions and emotional information, devices such as smartphones and head-mounted displays must be equipped with a camera and microphone. These devices use OpenCV and emotion analysis libraries to analyze the user's facial expressions and tone of voice, and transmit this information to the server as emotional information.

[1456] Based on this emotional information and real-time reactions, the server provides feedback to the generative AI model to adjust and update the conversation content. Finally, the adjusted conversation content is sent to the device and displayed to the user.

[1457] As a concrete example, in a virtual shopping assistant scenario, if a user asks, "Do you recommend these shoes?", the server uses a generative AI model to respond, "How about these shoes that suit your style?" Furthermore, if the user responds, "I'd like to see reviews too," the server analyzes real-time emotional information and generates a tailored response, such as, "Here are reviews from other users. They look great!"

[1458] Example prompt sentence:

[1459] A user asked 'Do you recommend these shoes?'

[1460] The realization of this system will enable more personalized and natural conversations to be provided based on the user's real-time emotions and reactions, improving the user experience in virtual stores.

[1461] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1462] Step 1:

[1463] The device receives input from the user about hobbies and interests, which are entered when the user logs into the application, and transmits the input information to the server.

[1464] Input: User's hobbies and interests

[1465] Output: User's hobbies and interests sent to the server

[1466] Specific operation: Using the UI / UX on the device, the user fills in an input form about their hobbies and interests. By pressing the input button, the data is sent to the server.

[1467] Step 2:

[1468] The server collects the user's communication history and, with the user's consent, analyzes past conversation history from messaging apps and stores it in a database.

[1469] Input: User consent, communication history

[1470] Output: Communication history stored in the database

[1471] Specific operation: After the server confirms the user's consent, it obtains communication history from the messaging app and stores the analyzed data in a database.

[1472] Step 3:

[1473] The server trains a natural language processing model based on collected hobbies, interests, and communication history. It uses Python and TensorFlow to build and train a generative AI model that learns the user's personalized speech patterns.

[1474] Input: Hobbies and interests, communication history

[1475] Output: A trained natural language processing model

[1476] How it works: A generative AI model is built using TensorFlow, and the model is trained using collected data. After multiple learning cycles, the model learns optimal conversation patterns.

[1477] Step 4:

[1478] The device collects the user's real-time reactions and emotional information and sends it to the server. Using a smartphone or head-mounted display, facial expression information and tone of voice are collected from the camera and microphone, and analyzed using OpenCV and the emotion analysis library.

[1479] Input: Real-time user reactions, facial expressions, and tone of voice

[1480] Output: Emotion information sent to the server

[1481] Specific operation: The device's camera and microphone are used to record the user's reactions, facial expressions are analyzed using OpenCV, emotional states are detected using an emotion analysis library, and the data is sent to the server.

[1482] Step 5:

[1483] The server provides feedback to the generative AI model based on real-time reactions and emotional information, adjusting and updating the conversation content. The adjusted conversation content is then sent to the device and displayed to the user.

[1484] Input: Real-time reactions, emotional information

[1485] Output: Adjusted conversation

[1486] How it works: The server analyzes the received emotional information and reactions and provides feedback to the generative AI model. The model generates an adjusted conversation and sends it to the device. The device then displays the updated conversation to the user.

[1487] Step 6:

[1488] The device displays the adjusted conversation content to the user, who can then further react to the displayed conversation content, which becomes the starting point for the next conversation.

[1489] Input: Modified conversation

[1490] Output: The conversation as it appears to the user

[1491] Specific operation: The adjusted conversation content is presented to the user on the device display screen. The user reads it and gives a new response, which leads to the next conversation process.

[1492] This allows the system as a whole to generate more personalized and natural conversations based on the user's real-time emotions and reactions.

[1493] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1494] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1495] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1496] [Fourth embodiment]

[1497] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1498] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1499] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1500] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1501] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1502] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1503] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1504] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1505] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1506] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1507] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1508] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1509] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1510] This invention relates to a system that uses generative AI to provide natural conversations in order to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[1511] Overall system overview

[1512] The system trains a natural language processing model based on user input such as hobbies, interests, and communication history, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses.

[1513] Data collection

[1514] 1. Collecting hobbies and interests

[1515] When a user registers for the service, they enter their hobbies and interests.

[1516] The terminal receives this input information and sends it to the server.

[1517] 2. Collection of communication history

[1518] With the user's consent, the server collects communication history and conversation history from messaging apps.

[1519] The server stores the collected data in a database.

[1520] Model training

[1521] The server trains a natural language processing model based on the user data collected.

[1522] The server iteratively trains the model with data to learn speech patterns optimized for individual users.

[1523] Conversation Generation

[1524] The user inputs a conversation topic or question into the terminal.

[1525] The device sends the input to the server.

[1526] The server uses the generative AI model to generate optimized conversations and sends the responses to the device.

[1527] The terminal displays the received reply to the user.

[1528] Real-time adjustments

[1529] 1. Real-time reaction analysis

[1530] Users react in real time to the generated conversation (e.g., by typing text or responding with voice).

[1531] The device receives real-time responses and transmits them to the server.

[1532] 2. Update the conversation

[1533] The server analyzes the user's real-time responses and sends feedback to the generative AI model.

[1534] The server updates the conversation content based on the feedback and sends it back to the device.

[1535] Specific examples

[1536] Travel plan example

[1537] 1. Input Phase

[1538] User A types into the terminal, "Where do you want to go on your next holiday?"

[1539] The device sends input to the server.

[1540] 2. Conversation generation phase

[1541] The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[1542] The terminal displays the generated conversation to User A.

[1543] 3. Real-time response

[1544] User A responds, "That's good, but I'm concerned about the budget."

[1545] The device sends this response to the server.

[1546] 4. Conversation Update Phase

[1547] The server analyzes the real-time responses and updates the suggestion with, "So, how about a trip to a hot spring in Hakone?"

[1548] The device displays the updated proposal to User A.

[1549] In this way, the system of the present invention can provide natural conversations that make people feel as if they are together in the same place, regardless of time or place, and can adjust to eliminate the sense of time and space discrepancies in communal living.

[1550] The processing flow will be explained below.

[1551] Step 1:

[1552] When users register for the service, they enter their hobbies and interests. The device receives this information and sends it to the server.

[1553] Step 2:

[1554] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[1555] Step 3:

[1556] The server trains a natural language processing model based on the user's hobbies, interests, and communication history collected by the server. The server repeatedly trains the model using data to learn conversation patterns optimized for each individual user.

[1557] Step 4:

[1558] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[1559] Step 5:

[1560] The server uses the generative AI model to generate an optimized conversation (e.g., "How about the trip to Hokkaido we talked about earlier?") and sends the response to the device. The device then displays the received response to the user.

[1561] Step 6:

[1562] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives the real-time response and sends it to the server.

[1563] Step 7:

[1564] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on the feedback.

[1565] Step 8:

[1566] The server sends the updated conversation (e.g., "So, how about a trip to a hot spring in Hakone?") to the device again. The device displays the updated proposal to the user.

[1567] In this way, users are provided with natural and optimized conversations regardless of time or place.

[1568] Example 1

[1569] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1570] In modern society, differences in time and space in communal living often lead to a decrease in communication and an increase in emotional distance, making it difficult for family and friends to interact smoothly and potentially leading to estrangement.

[1571] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1572] In this invention, the server includes means for inputting the user's hobbies and interests, means for collecting the user's communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and updating the conversation based on the reactions, and means for saving the collected data in a database. This makes it possible to provide natural conversation regardless of time or place, and to adjust so that time and space differences in communal living are not felt.

[1573] "User" refers to a person who uses the system.

[1574] "Hobbies and interests" refers to information about activities or topics that interest a user.

[1575] "Communication history" refers to a record of a user's past digital communications, such as messages, emails, and chats.

[1576] "Natural language processing model" refers to an algorithm or machine learning model for understanding and generating human language.

[1577] A "server" refers to a computer device that processes the entire system, stores and analyzes data, trains models, and so on.

[1578] "Terminal" refers to a hardware device that is directly operated by a user (e.g., smartphone, tablet, PC).

[1579] "Database" refers to a system for efficiently storing and managing collected data.

[1580] "Training a model" refers to the process of using collected data to refine and improve the performance of a natural language processing model.

[1581] "Optimized conversation" refers to conversation content that is easy to understand and relevant to the user, generated based on the user's hobbies, interests, and communication history.

[1582] "Real-time responses" refers to the instantaneous inputs and responses (e.g., text input, voice responses) a user makes while interacting with a system.

[1583] "Updating the conversation" refers to changing or improving the conversation content generated by the system based on the user's real-time reactions.

[1584] A "conversation generation scenario" refers to the flow and structure of a conversation with a user that is created based on collected data.

[1585] "Evaluating" refers to the process of determining how well the generated conversation matches the user's needs and expectations.

[1586] This invention relates to a system that utilizes a generative AI model to provide natural conversations in order to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[1587] Overall system overview

[1588] This system trains a natural language processing model based on user input such as hobbies, interests, and communication history, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses.

[1589] Data collection

[1590] Collecting hobbies and interests

[1591] When a user registers for the service, they enter their hobbies and interests.

[1592] The terminal receives this input information and sends it to the server.

[1593] Collection of communication history

[1594] With the user's consent, the server collects communication history and conversation history from messaging apps.

[1595] The server stores the collected data in a database.

[1596] Example: User B enters "watching movies" and "hiking" in the hobby field, and the server collects his past message data and stores it in a database.

[1597] Training the model

[1598] The server uses the stored user data to train a natural language processing model, using TensorFlow and PyTorch as the specific software.

[1599] The server learns conversation patterns optimized for individual users.

[1600] Example: User B's hobby data and past communication history are used to train a natural language processing model (e.g., GPT-3) to learn optimal conversation patterns about movies and hiking.

[1601] Conversation Generation

[1602] The user inputs a specific topic or question into the terminal.

[1603] The terminal sends the input contents to the server.

[1604] The server uses a trained generative AI model to generate optimized conversations.

[1605] The server sends the generated response to the terminal, which displays it to the user.

[1606] Example: User A types into his terminal, "Where do you want to go on your next holiday?" The server responds by saying, "How about that trip to Hokkaido we talked about earlier?" and the terminal displays this to User A.

[1607] Real-time adjustments

[1608] Real-time reaction analysis

[1609] Users react to generated conversations in real time.

[1610] The device receives the real-time response and sends it to the server.

[1611] Update the conversation

[1612] The server analyzes the user's real-time responses and sends feedback to the generative AI model.

[1613] The server updates the conversation content based on the feedback and sends it back to the device.

[1614] The terminal displays the updated content to the user.

[1615] Example: User A responds, "That sounds good, but I'm concerned about the budget," and the server analyzes this and updates the suggestion to, "Then, how about a trip to a hot spring in Hakone?", which the device displays to User A.

[1616] Examples of prompt statements

[1617] User A: "Where do you want to go on your next vacation?"

[1618] Server: "How was that trip to Hokkaido you were talking about earlier?"

[1619] User A: "That's good, but I'm concerned about the budget."

[1620] Server: "So, how about a trip to the hot springs in Hakone?"

[1621] This system allows for natural conversation regardless of time or place, eliminating the sense of time or space gaps in communal living and enabling smooth communication.

[1622] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1623] Step 1: User Registration

[1624] The user accesses the service for the first time and creates an account. The device displays a registration form to the user, where the user enters their name, email address, and hobbies and interests. The device then sends the information to the server.

[1625] Input: Name, email address, hobbies and interests

[1626] Output: User registration data (saved in database)

[1627] Step 2: Collect communication history

[1628] With the user's consent, the server collects communication history from messaging apps, emails, etc. The server stores this data in a database.

[1629] Input: User consent, communication history

[1630] Output: Collected communication history data (stored in a database)

[1631] Step 3: Save data

[1632] The server stores the user's hobbies and interests and the collected communication history in a database, confirms that the storage was successful, and sends a success message to the terminal.

[1633] Input: Hobbies and interests, communication history

[1634] Output: Data saved to the database, message that save was successful

[1635] Step 4: Train the model

[1636] The server uses the stored data to train a natural language processing model, specifically using TensorFlow and PyTorch to train the model and have it learn conversation patterns optimized for the user.

[1637] Input: User data in the database

[1638] Output: A trained natural language processing model

[1639] Step 5: Enter the conversation topic

[1640] The user inputs a conversation topic or question into the terminal, and the terminal transmits the input content to the server.

[1641] Input: A topic or question entered by the user

[1642] Output: Send data to the server

[1643] Step 6: Conversation generation

[1644] The server uses the trained generative AI model to generate optimized conversations, and sends the generated responses to the device, which then displays them to the user.

[1645] Input: The subject or question sent to the server

[1646] Output: Generated conversation (sent to terminal)

[1647] Step 7: Analyzing real-time responses

[1648] The user reacts to the generated conversation in real time, and the device transmits the reaction to the server.

[1649] Input: Real-time user response

[1650] Output: Sends response data to the server

[1651] Step 8: Update the conversation

[1652] The server analyzes the user's real-time responses and sends feedback to the generative AI model. The conversation content is updated based on that feedback, and the newly generated content is sent to the device. The device then displays the updated content to the user.

[1653] Input: Real-time user response

[1654] Output: Updated conversation (sent to terminal)

[1655] Specific operation example

[1656] 1. User A types into the terminal, "Where do you want to go on your next holiday?"

[1657] 2. The device sends this input to the server.

[1658] 3. The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[1659] 4. The device displays the generated conversation to User A.

[1660] 5. User A responds, "That's fine, but I'm concerned about the budget."

[1661] 6. The device sends this response to the server.

[1662] 7. The server analyzes the real-time response and updates the suggestion with, "How about a trip to a hot spring in Hakone?"

[1663] 8. The device displays the updated proposal to User A.

[1664] (Application example 1)

[1665] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1666] The present invention aims to improve customer satisfaction by facilitating smooth communication between customers and store clerks in brick-and-mortar stores. In particular, there is a need to strengthen relationships of trust with customers by providing personalized services and making appropriate product recommendations based on the customer's preferences and past purchase history. However, conventional technologies have had difficulty optimizing real-time conversations and generating responses tailored to individual customer needs.

[1667] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1668] In this invention, the server includes: means for collecting user hobbies and interests; means for collecting user communication history; means for training a natural language processing model based on the collected hobbies and interests and communication history; means for generating an optimized conversation using the generated natural language processing model; means for analyzing the user's real-time reactions and updating the conversation based on the reactions; means for generating a conversation between a customer and a store clerk to support interactions in a physical store; and means for displaying the generated conversation using an application installed on smart glasses or a head-mounted display. This enables smoother interactions with customers in physical stores and more personalized services to be provided.

[1669] The "means for collecting user's hobbies and interests" is a function for recording information on hobbies and interests entered by the user and saving it as data.

[1670] "Means for collecting a user's communication history" refers to a function for obtaining and analyzing past communication history, such as text messages and call records, with the user's consent.

[1671] "Means for training natural language processing models based on collected hobbies, interests, and communication history" refers to a function that uses collected user data to optimize natural language processing models using machine learning and deep learning algorithms.

[1672] "Means for generating optimized conversations using the generated natural language processing model" is a function that uses a trained natural language processing model to generate optimal conversations based on the user's interests and past communication history.

[1673] "Means for analyzing the user's real-time responses and updating the conversation based on those responses" refers to a function that analyzes the real-time text and voice responses entered by the user and updates the content of the conversation in a timely manner based on that information.

[1674] "A means for generating conversations between customers and store clerks to support interactions in physical stores" is a function that supports conversations between customers and store clerks in physical stores and makes appropriate product suggestions tailored to the customer's needs.

[1675] "Means for displaying the generated conversation using an application installed on smart glasses or a head-mounted display" refers to a function for displaying the generated conversation content using an application installed on a wearable device such as smart glasses or a head-mounted display.

[1676] MODE FOR CARRYING OUT THE INVENTION

[1677] The present invention is designed to facilitate communication between customers and store clerks in physical stores and to provide personalized services. Specific embodiments for carrying out the present invention will be described below.

[1678] Overall system overview

[1679] This system collects users' hobbies, interests, and communication history, and uses them to train a natural language processing model to generate optimized conversations. It also has the ability to update the conversation content based on the user's real-time responses. The system primarily consists of a server, terminals, and wearable devices such as smart glasses and head-mounted displays.

[1680] Data collection

[1681] Collecting hobbies and interests

[1682] When a user (customer) registers for the service, they enter their hobbies and interests into a terminal. This information is sent to the server via the terminal and stored in a database.

[1683] Collection of communication history

[1684] With the user's consent, the server collects communication history and past conversation history from messaging apps, and this data is also stored in the server's database.

[1685] Model training

[1686] The server trains a natural language processing model based on the collected user information, such as hobbies, interests, and communication history. This requires machine learning and deep learning algorithms, such as OpenAI's GPT-3.

[1687] Conversation Generation

[1688] The user (store clerk) uses an application installed on smart glasses or a head-mounted display to input conversation topics and questions. This input information is sent to the server via the device. The server uses a generative AI model to generate an optimized conversation and sends the response to the device. The device then displays the received response to the user (store clerk).

[1689] Examples of prompt statements

[1690] When the user (store clerk) types "Looking for new sneakers?", the server generates the following:

[1691] Generates "The Nike model you purchased last time is very popular."

[1692] Real-time adjustments

[1693] Real-time reaction analysis

[1694] The user (customer) responds to the generated conversation in real time (for example, by inputting text or responding with voice). The terminal receives the real-time response and sends it to the server.

[1695] Update the conversation

[1696] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on this feedback and sends it back to the device.

[1697] Examples of prompt statements

[1698] If the user (customer) responds with "I really like that model, but what else do you recommend?" the server generates:

[1699] "We have new color variations in stock, and we also recommend the latest Adidas models," says Genesis.

[1700] Examples of application in physical stores

[1701] This system will be used to support interactions between customers and store clerks in physical stores. For example, if a customer asks for an explanation of a specific product, the store clerk can use the system through the smart glasses to provide the customer with the most appropriate product explanation and suggestions. This allows customers to receive personalized service based on their preferences and past purchasing history, leading to increased customer satisfaction.

[1702] In this way, the system of the present invention significantly improves the quality of service in physical stores.

[1703] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1704] Step 1:

[1705] When a user (customer) registers for the service, they input information about their hobbies and interests through their device. The device then sends this information to the server, which then stores the data on their hobbies and interests in a database.

[1706] Input: User's hobbies and interests

[1707] Output: Hobbies and interests data stored in a database

[1708] Specific operation: The device acquires the user's hobbies and interests and sends them to the server.

[1709] Step 2:

[1710] With the user's consent, the server collects communication history, such as text messages and past conversation history, and stores it in a database.

[1711] Input: User's communication history

[1712] Output: Communication history stored in the database

[1713] Specific operation: The server collects and stores data from messaging apps and communication history.

[1714] Step 3:

[1715] The server uses the collected hobbies, interests, and communication history to train a natural language processing model, using machine learning algorithms to learn patterns from the training data.

[1716] Input: Hobbies and interests, communication history

[1717] Output: A trained natural language processing model

[1718] Specific operation: The server uses a natural language processing model such as GPT-3 to train the model based on the collected data.

[1719] Step 4:

[1720] The user (store clerk) inputs the conversation topic or question into the smart glasses or head-mounted display, which is then sent from the device to the server. The server uses a generative AI model to generate an optimized conversation and sends the response to the device.

[1721] Input: conversation topic or question

[1722] Output: Generated conversational responses

[1723] Specific operation: The device sends the conversation topic to the server, and the server uses a generative AI model to generate a conversation and provide a response.

[1724] Step 5:

[1725] The terminal receives and displays the generated conversation to the user (store clerk) through smart glasses or a head-mounted display.

[1726] Input: Generated conversation response

[1727] Output: The conversation displayed to the user

[1728] Specific operation: The device receives the generated conversation from the server and displays it on smart glasses or a head-mounted display.

[1729] Step 6:

[1730] The user (customer) responds to the generated conversation in real time and inputs the response into the terminal, which then transmits the real-time response to the server.

[1731] Input: Real-time response (text or voice)

[1732] Output: Real-time responses sent to the server

[1733] Specific operation: The process in which the terminal receives real-time responses from the user (customer) and sends them to the server.

[1734] Step 7:

[1735] The server analyzes the user's real-time responses and sends feedback to the generative AI model, which then updates the conversation content based on this feedback and sends it back to the device.

[1736] Input: Analyzed real-time responses

[1737] Output: Updated conversation

[1738] Specific operation: The server analyzes real-time responses, updates the generative AI model, generates new conversation content, and sends it to the device.

[1739] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1740] This invention relates to a system that provides natural conversations by utilizing generative AI and an emotion engine to resolve the problem of time and space differences in communal living that can lead to reduced communication and increased emotional distance. Specific embodiments of the invention are described below.

[1741] Overall system overview

[1742] The system trains a natural language processing model based on user-entered hobbies, interests, communication history, and emotional information, and uses it to generate optimized conversations. It also has the ability to update the conversation content based on real-time user reactions and emotional analysis.

[1743] Data collection

[1744] 1. Collecting hobbies and interests

[1745] When a user registers for the service, they enter their hobbies and interests.

[1746] The terminal receives this input information and sends it to the server.

[1747] 2. Collection of communication history

[1748] With the user's consent, the server collects past conversation history between users from communication history and messaging apps.

[1749] The server stores the collected data in a database.

[1750] 3. Collecting emotional information

[1751] The device analyzes the user's emotional information in real time using an emotion engine and sends it to the server.

[1752] Model training

[1753] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected.

[1754] The server iteratively trains the model with data to learn speech patterns optimized for individual users.

[1755] Conversation Generation

[1756] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[1757] The server uses the generative AI model to generate an optimized conversation (e.g., "How was the trip to Hokkaido we talked about earlier?") and sends the response to the device.

[1758] The terminal displays the received response to the user.

[1759] Real-time adjustments

[1760] 1. Real-time reaction analysis

[1761] Users respond in real time to generated conversations (e.g., "That's good, but I'm concerned about the budget").

[1762] The device receives the real-time response and sends it to the server.

[1763] 2. Emotional Information Analysis

[1764] The device uses an emotion engine to analyze emotional information from the user's facial expressions and voice in real time, such as facial expressions and tone of voice when the user shows interest.

[1765] The device transmits the analyzed emotional information to the server.

[1766] 3. Update the conversation

[1767] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model.

[1768] The server updates the conversation content based on the feedback and sends it back to the device.

[1769] Specific examples

[1770] Travel plan example

[1771] 1. Input Phase

[1772] User A types into the terminal, "Where do you want to go on your next holiday?"

[1773] The device sends input to the server.

[1774] 2. Conversation generation phase

[1775] The server generates a question based on past conversation history and hobby information: "How about that trip to Hokkaido we talked about earlier?"

[1776] The terminal displays the generated conversation to User A.

[1777] 3. Real-time reaction and emotional information collection

[1778] User A responds, "That's good, but I'm concerned about the budget." The device sends the response in real time to the server.

[1779] The device uses an emotion engine to analyze the user's facial expressions and voice to determine whether they are interested, and then sends that information to the server.

[1780] 4. Conversation Update Phase

[1781] The server analyzes real-time reactions and emotional information and updates the suggestion with, "So, how about a hot spring trip to Hakone?"

[1782] The device displays the updated proposal to User A.

[1783] In this way, the system of the present invention can provide natural and optimized conversations based on the user's emotions, regardless of time or place, and can adjust to eliminate any sense of time or space discrepancy in communal living.

[1784] The processing flow will be explained below.

[1785] Step 1:

[1786] When users register for the service, they enter their hobbies and interests. The device receives this information and sends it to the server.

[1787] Step 2:

[1788] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[1789] Step 3:

[1790] The device analyzes the user's emotional information in real time using an emotion engine and transmits it to the server, including the user's facial expressions, tone of voice, and physical movements.

[1791] Step 4:

[1792] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected by the server. The server repeatedly trains the model using data to learn conversation patterns optimized for each individual user.

[1793] Step 5:

[1794] The device receives conversation topics and questions (e.g., "Where do you want to go on your next holiday?") entered by the user into the device and sends them to the server.

[1795] Step 6:

[1796] The server uses the generative AI model to generate an optimized conversation (e.g., "How about the trip to Hokkaido we talked about earlier?") and sends the response to the device. The device then displays the received response to the user.

[1797] Step 7:

[1798] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives the real-time response and sends it to the server.

[1799] Step 8:

[1800] The device analyzes emotional information from the user's facial expressions and voice in real time using an emotion engine and sends the information to the server.

[1801] Step 9:

[1802] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model, which then updates the conversation content based on the feedback.

[1803] Step 10:

[1804] The server sends the updated conversation (e.g., "So, how about a trip to a hot spring in Hakone?") to the device again. The device displays the updated proposal to the user.

[1805] In this way, users can experience natural and optimized conversations regardless of time or place, and feedback based on the user's emotions makes communication even more natural.

[1806] Example 2

[1807] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1808] In communal living, time and space differences can easily lead to decreased communication and increased emotional distance. Conventional systems do not provide natural conversations that effectively utilize users' hobbies, interests, communication history, and real-time emotional information, which is an obstacle to smooth communication. To solve this problem, a system is needed that can collect and analyze a wide range of user information and provide natural conversations based on the results.

[1809] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1810] In this invention, the server includes means for collecting user hobbies and interests, means for collecting user communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and collecting emotional information, and means for updating the conversation in real time based on the collected emotional information. This makes it possible to utilize the user's multifaceted information and have a natural and optimized conversation based on the real-time reactions and emotional information.

[1811] "User" means an individual or organization that uses the system.

[1812] "Hobbies and Interests" refers to specific activities or subjects that interest a user, which information the user enters into the system.

[1813] "Communication history" refers to a record of messages and conversations that a user has had with other users in the past.

[1814] A "natural language processing model" is a type of machine learning model for analyzing, understanding, and generating text written in natural language.

[1815] "Generated natural language processing model" refers to a natural language processing model trained on collected data.

[1816] "Optimized conversation" refers to the most appropriate and natural conversation content generated based on the user's hobbies, interests, communication history, and emotional information.

[1817] "Real-time response" refers to the instantaneous reactions and feedback a user gives while interacting with a system.

[1818] "Emotional information" refers to the emotional state analyzed from the user's facial expression, tone of voice, etc.

[1819] "Updating the conversation in real time" means dynamically changing the content of the conversation based on the user's real-time reactions and emotional information.

[1820] Overall system overview

[1821] This invention is a system designed to resolve time and space differences in communal living and provide smoother communication. This system trains a natural language processing model based on user-entered hobbies, interests, communication history, and emotional information, and generates optimized conversations.

[1822] Hardware and software used

[1823] Server: Responsible for storing data, training models, and generating conversations

[1824] Terminal: Responsible for inputting information from the user, collecting real-time emotion data, and displaying the generated conversation content

[1825] Emotion Engine: Software for analyzing emotional information from the user's facial expressions and voice

[1826] Natural language processing model: A machine learning model that generates conversations based on collected data.

[1827] Data collection

[1828] 1. Collecting hobbies and interests

[1829] When a user registers for the service, they input their hobbies and interests. The device receives this input information and sends it to the server.

[1830] 2. Collection of communication history

[1831] With the user's consent, the server collects past conversation history between users from communication history and messaging apps, and stores the collected data in a database.

[1832] 3. Collecting emotional information

[1833] The device analyzes the user's facial expressions and voice in real time using an emotion engine and sends the results to a server, which then stores the received emotion information in a database.

[1834] Model training

[1835] The server trains a natural language processing model based on the user's hobbies, interests, communication history, and emotional information collected by the server. The data is used to repeatedly train the model, allowing it to learn conversation patterns optimized for specific users.

[1836] Conversation Generation

[1837] The user enters a conversation topic or question into the device (e.g., "Where do you want to go on your next vacation?"). The device sends this information to the server, which uses a generative AI model to generate an optimized response (e.g., "How about that trip to Hokkaido we were talking about earlier?"). The server sends the generated response to the device, which displays it to the user.

[1838] Real-time adjustments

[1839] 1. Real-time reaction analysis

[1840] The user responds to the generated conversation in real time (e.g., "That's good, but I'm concerned about the budget."). The device receives this response and sends it to the server.

[1841] 2. Emotional Information Analysis

[1842] The device analyzes the user's facial expressions and tone of voice in real time to determine their emotional state, and sends the analysis results to the server.

[1843] 3. Update the conversation

[1844] The server analyzes the user's real-time reactions and emotional information and sends feedback to the generative AI model. The server updates the conversation content based on this feedback and sends it back to the device (e.g., "So, how about a hot spring trip to Hakone?"). The device then displays the updated suggestions to the user.

[1845] Specific examples

[1846] 1. Input Phase

[1847] User A enters "Where do you want to go on your next holiday?" into the device. The device sends this information to the server.

[1848] 2. Conversation generation phase

[1849] The server generates a conversation based on past conversation history and hobby information, asking, "How about that trip to Hokkaido we talked about earlier?" The device displays the generated conversation to User A.

[1850] 3. Real-time reaction and emotional information collection

[1851] User A responds, "That's good, but I'm concerned about the budget." The device sends this response to the server. The device uses an emotion engine to analyze the user's facial expressions and voice, and sends that information to the server.

[1852] 4. Conversation Update Phase

[1853] The server analyzes the real-time reactions and emotional information and updates the suggestion to "How about a trip to a hot spring in Hakone?" The device displays the updated suggestion to User A.

[1854] Prompt Sentence Examples

[1855] "Where do you want to go on your next vacation? Get travel suggestions based on your hobbies and past communication history."

[1856] By implementing the present invention in this way, users can experience optimized, natural conversations regardless of time or place. Real-time emotion analysis also enables more appropriate and satisfying communication.

[1857] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1858] Step 1: User Registration

[1859] Input: A user accesses the system and enters the required information to create a new account.

[1860] Output: The terminal receives the user information and sends it to the server.

[1861] Specific behavior:

[1862] A user enters their name, email address, password, etc. into an input form.

[1863] The terminal sends this information to the server via the send button.

[1864] The server stores the received information in a database and creates a user account.

[1865] Step 2: Enter your hobbies and interests

[1866] Input: The user enters their hobbies and interests.

[1867] Output: The device receives this information and sends it to the server.

[1868] Specific behavior:

[1869] Users fill out a multiple-choice form to select activities or topics that interest them.

[1870] The terminal checks the input information and sends it to the server.

[1871] The server stores the received hobbies and interests in a database.

[1872] Step 3: Collect communication history

[1873] Input: Get user consent.

[1874] Output: The server collects past conversation history from communication history and messaging apps and stores it in a database.

[1875] Specific behavior:

[1876] The user presses the consent button regarding the collection of communication history.

[1877] The server collects past communication history through the messaging app's API.

[1878] The collected data is stored in a database.

[1879] Step 4: Collecting emotional information

[1880] Input: The device captures the user's facial expressions and voice in real time.

[1881] Output: The device analyzes these data using the emotion engine and sends the results to the server.

[1882] Specific behavior:

[1883] The device uses a camera and microphone to capture the user's facial expressions and voice.

[1884] The device uses an emotion engine to analyze facial expressions and voice in real time.

[1885] The analysis results are sent to a server, which stores the information in a database.

[1886] Step 5: Data Preparation

[1887] Input: The server collects information about the user's hobbies, interests, communication history, and emotional information.

[1888] Output: Combined into a dataset and used for model training.

[1889] Specific behavior:

[1890] The server extracts the necessary data from the database and performs preprocessing.

[1891] Integrate the data to create a dataset for model training.

[1892] Step 6: Model training

[1893] Input: Dataset.

[1894] Output: A trained natural language processing model.

[1895] Specific behavior:

[1896] The server uses the dataset to train a natural language processing model.

[1897] During the training process, the model parameters are optimized and the user is trained to learn optimized conversation patterns.

[1898] Step 7: User Input

[1899] Input: The user types a question or conversation topic into the device (e.g., "Where do you want to go on your next vacation?").

[1900] Output: The device receives input and sends it to the server.

[1901] Specific behavior:

[1902] The user enters a question into the input form on the terminal and presses the send button.

[1903] The terminal transmits the user input as text data to the server.

[1904] Step 8: Conversation Generation

[1905] Input: User questions and conversation topics.

[1906] Output: The server uses the generative AI model to generate an optimized response and sends it to the device.

[1907] Specific behavior:

[1908] The server receives the user's question and compares it with historical data.

[1909] Generative AI models are used to generate appropriate responses (e.g., "How about that trip to Hokkaido we talked about earlier?").

[1910] The generated response is sent to the terminal, which displays it to the user.

[1911] Step 9: Analyze real-time responses

[1912] Input: Real-time user response (e.g., "That sounds good, but I'm concerned about the budget").

[1913] Output: The device receives the response and sends it to the server.

[1914] Specific behavior:

[1915] Users provide real-time feedback on the generated conversation.

[1916] The device captures the response and sends it to the server.

[1917] Step 10: Emotional Analysis

[1918] Input: User facial and voice data.

[1919] Output: The analysis results of the emotion engine are sent to the server.

[1920] Specific behavior:

[1921] The device uses a camera and microphone to capture the user's facial expressions and voice data.

[1922] The emotion engine analyzes this data in real time and extracts emotional information.

[1923] The analysis results are sent to the server.

[1924] Step 11: Update your conversation

[1925] Input: Real-time user reaction and emotional information.

[1926] Output: Sends the updated conversation to the terminal.

[1927] Specific behavior:

[1928] The server analyzes real-time reactions and emotional information.

[1929] Feedback is fed into generative AI models to optimize and update conversation content.

[1930] The updated conversation content is sent to the terminal, which displays it to the user (e.g., "So, how about a trip to the hot springs in Hakone?").

[1931] (Application example 2)

[1932] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1933] In communal living, time and space differences can reduce communication and increase emotional distance. Furthermore, users' shopping experiences in virtual stores are not personalized, which can lead to unexpected frustration and stress. To address these issues, a system is needed that can generate optimized conversations based on users' hobbies, interests, real-time reactions, and emotional information, thereby improving the user's shopping experience.

[1934] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1935] In this invention, the server includes means for collecting user hobbies and interests, means for collecting user communication history, means for training a natural language processing model based on the collected hobbies and interests and communication history, means for generating an optimized conversation using the generated natural language processing model, means for analyzing the user's real-time reactions and updating the conversation based on the reactions, means for collecting and analyzing emotional information such as the user's facial expression information and tone of voice, and means for adjusting the conversation based on the analyzed emotional information. This makes it possible to realize more personalized and natural conversation based on the user's real-time emotions and reactions, and improve the user experience in the virtual store.

[1936] "Hobbies and interests" refers to the things, activities, and preferences that a user is interested in.

[1937] "Communication history" refers to a record of past conversations and messages between users.

[1938] A "natural language processing model" refers to a statistical machine learning model for understanding and generating human language.

[1939] "Real-time reactions" refers to the immediate reactions and responses that users give during a conversation.

[1940] "Facial expression information" refers to data relating to emotions obtained from the user's facial expressions.

[1941] "Voice tone" refers to information about the characteristics of the voice, such as the pitch and volume of the voice.

[1942] "Emotional information" refers to data that represents the user's emotional state, such as facial expressions and tone of voice.

[1943] The "means for adjusting the conversation" refers to a method for updating the generated conversation content based on the user's real-time reactions and emotional information.

[1944] To implement this invention, it is necessary to build a system that collects users' hobbies and interests, communication history, real-time reactions, and emotional information, and generates optimized conversations based on the collected information. This system includes the following main elements:

[1945] First, to collect the user's hobbies and interests, a method is required for the user to input these when logging in to the application. Also, with the user's consent, communication history is collected and this data is sent to the server, allowing the server to manage user information in an integrated manner.

[1946] The server trains a natural language processing model based on the collected hobbies, interests, and communication history. Specifically, it uses Python and TensorFlow to build and train a generative AI model. This model is used to learn the user's personalized conversation patterns and generate optimized conversations.

[1947] Next, to collect real-time user reactions and emotional information, devices such as smartphones and head-mounted displays must be equipped with a camera and microphone. These devices use OpenCV and emotion analysis libraries to analyze the user's facial expressions and tone of voice, and transmit this information to the server as emotional information.

[1948] Based on this emotional information and real-time reactions, the server provides feedback to the generative AI model to adjust and update the conversation content. Finally, the adjusted conversation content is sent to the device and displayed to the user.

[1949] As a concrete example, in a virtual shopping assistant scenario, if a user asks, "Do you recommend these shoes?", the server uses a generative AI model to respond, "How about these shoes that suit your style?" Furthermore, if the user responds, "I'd like to see reviews too," the server analyzes real-time emotional information and generates a tailored response, such as, "Here are reviews from other users. They look great!"

[1950] Example prompt sentence:

[1951] A user asked 'Do you recommend these shoes?'

[1952] The realization of this system will enable more personalized and natural conversations to be provided based on the user's real-time emotions and reactions, improving the user experience in virtual stores.

[1953] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1954] Step 1:

[1955] The device receives input from the user about hobbies and interests, which are entered when the user logs into the application, and transmits the input information to the server.

[1956] Input: User's hobbies and interests

[1957] Output: User's hobbies and interests sent to the server

[1958] Specific operation: Using the UI / UX on the device, the user fills in an input form about their hobbies and interests. By pressing the input button, the data is sent to the server.

[1959] Step 2:

[1960] The server collects the user's communication history and, with the user's consent, analyzes past conversation history from messaging apps and stores it in a database.

[1961] Input: User consent, communication history

[1962] Output: Communication history stored in the database

[1963] Specific operation: After the server confirms the user's consent, it obtains communication history from the messaging app and stores the analyzed data in a database.

[1964] Step 3:

[1965] The server trains a natural language processing model based on collected hobbies, interests, and communication history. It uses Python and TensorFlow to build and train a generative AI model that learns the user's personalized speech patterns.

[1966] Input: Hobbies and interests, communication history

[1967] Output: A trained natural language processing model

[1968] How it works: A generative AI model is built using TensorFlow, and the model is trained using collected data. After multiple learning cycles, the model learns optimal conversation patterns.

[1969] Step 4:

[1970] The device collects the user's real-time reactions and emotional information and sends it to the server. Using a smartphone or head-mounted display, facial expression information and tone of voice are collected from the camera and microphone, and analyzed using OpenCV and the emotion analysis library.

[1971] Input: Real-time user reactions, facial expressions, and tone of voice

[1972] Output: Emotion information sent to the server

[1973] Specific operation: The device's camera and microphone are used to record the user's reactions, facial expressions are analyzed using OpenCV, emotional states are detected using an emotion analysis library, and the data is sent to the server.

[1974] Step 5:

[1975] The server provides feedback to the generative AI model based on real-time reactions and emotional information, adjusting and updating the conversation content. The adjusted conversation content is then sent to the device and displayed to the user.

[1976] Input: Real-time reactions, emotional information

[1977] Output: Adjusted conversation

[1978] How it works: The server analyzes the received emotional information and reactions and provides feedback to the generative AI model. The model generates an adjusted conversation and sends it to the device. The device then displays the updated conversation to the user.

[1979] Step 6:

[1980] The device displays the adjusted conversation content to the user, who can then further react to the displayed conversation content, which becomes the starting point for the next conversation.

[1981] Input: Modified conversation

[1982] Output: The conversation as it appears to the user

[1983] Specific operation: The adjusted conversation content is presented to the user on the device display screen. The user reads it and gives a new response, which leads to the next conversation process.

[1984] This allows the system as a whole to generate more personalized and natural conversations based on the user's real-time emotions and reactions.

[1985] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1986] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1987] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1988] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1989] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1990] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1991] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1992] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1993] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1994] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1995] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1996] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1997] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1998] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1999] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2000] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2001] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2002] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2003] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2004] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2005] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2006] The following is further disclosed regarding the above embodiment.

[2007] (Claim 1)

[2008] a means for collecting user hobbies and interests;

[2009] means for collecting a user's communication history;

[2010] A means for training a natural language processing model based on the collected hobbies, interests, and communication history;

[2011] a means for generating an optimized conversation using the generated natural language processing model;

[2012] means for analyzing real-time user responses and updating the conversation based on the responses;

[2013] A system including:

[2014] (Claim 2)

[2015] 10. The system of claim 1, further comprising means for evaluating the naturalness of the generated speech based on the user's individual speech patterns and preferences.

[2016] (Claim 3)

[2017] 10. The system of claim 1, further comprising means for storing the collected data in a database and using the database to create a conversation generation scenario.

[2018] "Example 1"

[2019] (Claim 1)

[2020] a means for inputting the user's hobbies and interests;

[2021] means for collecting a user's communication history;

[2022] a means for training a natural language processing model based on the collected hobbies, interests, and communication history;

[2023] a means for generating an optimized conversation using the generated natural language processing model;

[2024] means for analyzing real-time user responses and updating the conversation based on the responses;

[2025] a means for storing the collected data in a database;

[2026] A system including:

[2027] (Claim 2)

[2028] 10. The system of claim 1, further comprising means for evaluating the naturalness of the generated speech based on the user's individual speech patterns and preferences.

[2029] (Claim 3)

[2030] 10. The system of claim 1, further comprising means for creating a conversation generation scenario using the collected data.

[2031] "Application Example 1"

[2032] (Claim 1)

[2033] a means for collecting user hobbies and interests;

[2034] means for collecting a user's communication history;

[2035] A means for training a natural language processing model based on the collected hobbies, interests, and communication history;

[2036] a means for generating an optimized conversation using the generated natural language processing model;

[2037] means for analyzing real-time user responses and updating the conversation based on the responses;

[2038] A means for generating conversations between customers and store associates to support interactions in a physical store;

[2039] a means for displaying the generated conversation using an application installed on the smart glasses or head-mounted display;

[2040] A system including:

[2041] (Claim 2)

[2042] 10. The system of claim 1, further comprising means for evaluating the naturalness of the generated speech based on the user's individual speech patterns and preferences.

[2043] (Claim 3)

[2044] 10. The system of claim 1, further comprising means for storing the collected data in a database and using the database to create a conversation generation scenario.

[2045] "Example 2: Combining Emotion Engines"

[2046] (Claim 1)

[2047] a means for collecting user hobbies and interests;

[2048] means for collecting a user's communication history;

[2049] a means for training a natural language processing model based on the collected hobbies, interests, and communication history;

[2050] a means for generating an optimized conversation using the generated natural language processing model;

[2051] A means for analyzing real-time user responses and collecting emotional information;

[2052] A means to update the conversation in real time based on collected emotional information, and

[2053] A system including:

[2054] (Claim 2)

[2055] 10. The system of claim 1, further comprising means for evaluating the naturalness of the generated speech based on the user's individual speech patterns and preferences.

[2056] (Claim 3)

[2057] 10. The system of claim 1, further comprising means for storing the collected data in a database and using the database to create a conversation generation scenario.

[2058] "Application example 2 when combining emotion engines"

[2059] (Claim 1)

[2060] a means for collecting user hobbies and interests;

[2061] means for collecting a user's communication history;

[2062] A means for training a natural language processing model based on the collected hobbies, interests, and communication history;

[2063] a means for generating an optimized conversation using the generated natural language processing model;

[2064] means for analyzing real-time user responses and updating the conversation based on the responses;

[2065] A means for collecting and analyzing emotional information such as facial expression information and tone of voice of the user;

[2066] a means for adjusting the conversation based on the analyzed emotional information;

[2067] A system including:

[2068] (Claim 2)

[2069] 10. The system of claim 1, further comprising means for evaluating the naturalness of the generated speech based on the user's individual speech patterns and preferences.

[2070] (Claim 3)

[2071] 10. The system of claim 1, further comprising means for storing the collected data in a database and using the database to create a conversation generation scenario. [Explanation of symbols]

[2072] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for collecting user hobbies and interests; means for collecting a user's communication history; A means for training a natural language processing model based on the collected hobbies, interests, and communication history; a means for generating an optimized conversation using the generated natural language processing model; means for analyzing real-time user responses and updating the conversation based on the responses; A system including:

2. 10. The system of claim 1, further comprising means for evaluating the naturalness of the generated speech based on the user's individual speech patterns and preferences.

3. 10. The system of claim 1, further comprising means for storing the collected data in a database and using the database to create a conversation generation scenario.

Images