System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving an input from a user; means for analyzing the user's input and generating an appropriate response; and means for converting the generated response into speech and providing it to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there has been a demand for systems that allow users to communicate interactively with popular characters from anime and games. However, previous systems have struggled to naturally analyze user input and provide appropriate responses. Furthermore, they were unable to generate personalized responses based on the user's individual preferences, limiting the user experience. Furthermore, they lacked the ability to freely customize character settings, preventing user satisfaction. [Means for solving the problem]

[0005] The present invention is a system including a means for receiving input from a user, a means for analyzing the user's input and generating an appropriate response, and a means for converting the generated response into voice and providing it to the user (Claim 1). The system further includes a means for saving a user's profile and behavioral history, and a means for analyzing the saved profile and behavioral history and generating a response based on the user's preferences (Claim 2). The system also includes a means for saving character information and providing a character based on the user's selection, and a means for customizing the character's personality and voice settings (Claim 3). This allows the user to naturally interact with a character that matches their preferences, resulting in a more satisfying experience.

[0006] A "user" is a person who interacts with the system.

[0007] "Input" refers to information that a user provides to a system, either text or speech.

[0008] "Parsing" refers to processing user input to understand and make sense of it.

[0009] A "response" is a response generated by the system in response to input from the user.

[0010] "Generation" refers to the process by which the system creates a response.

[0011] "Convert to voice" refers to converting text data into voice data.

[0012] A "profile" is data that includes a user's personal information and settings.

[0013] "Behavioral history" refers to the record of a user's use of the system.

[0014] "Storage" means recording and retaining data.

[0015] "Customization" refers to changing settings and functions according to the user's wishes.

[0016] "Character" refers to a person or creature that appears in an anime or game, and is provided as a subject for interaction with the user in the system.

[0017] A "natural language processing engine" is software that analyzes natural language input from a user and generates an appropriate response.

[0018] A "speech synthesis engine" is software that converts text data into speech that resembles a human voice. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system that allows users to interact naturally with popular characters by analyzing input from the user and generating appropriate responses and translating them into voice. Below, the system program and its processing are explained in natural language, with specific examples.

[0041] System Overview

[0042] User authentication / login

[0043] Device: The user launches the dedicated app and logs in by entering their account information (username and password).

[0044] Server: Receives the user's authentication information and checks it against the database to see if it is correct. If authentication is successful, it starts a session and sends the information back to the terminal. The terminal checks the authentication status and displays the main screen if successful, or an error message if unsuccessful.

[0045] Character Selection and Settings

[0046] Terminal: The main screen displays a list of characters available to the user. The list is retrieved from the server.

[0047] User: Select the desired character and customize their personality, voice pitch, etc. on the detailed settings screen. The customized settings are sent from the device to the server.

[0048] Server: Stores the received configuration data with the user's profile.

[0049] Interaction Generation

[0050] User: Provides text or voice input to the character. Voice input is converted to text using the device's voice recognition function.

[0051] Terminal: Sends user text input to the server.

[0052] Server: Passes the received text to a natural language processing engine to analyze the input. Based on the analysis results, a generative AI generates an appropriate response. The response is then sent back from the server to the device.

[0053] On the device: The returned text response is passed to a speech synthesis engine to generate speech and play it back to the user.

[0054] Specific examples

[0055] Morning greetings and schedule check

[0056] User: Say "Good morning!"

[0057] Device: Converts the user's speech into text and sends it to the server.

[0058] Server: Analyzes the input text and generates a response such as "Good morning! Would you like to check today's schedule?". Then, sends the generated response back to the device.

[0059] Terminal: The returned response is converted into speech by a speech synthesis engine and played back to the user. It says, "Good morning! Would you like to check today's schedule?"

[0060] Adding a task

[0061] User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[0062] Terminal: Sends input text to the server.

[0063] Server: Parse the text and set a reminder for the specified date and time. Generate a confirmation message saying "Your reminder for the meeting has been set for tomorrow at 9 AM."

[0064] Device: Converts a confirmation message into speech and provides it to the user. Says, "I've set a reminder for a meeting tomorrow at 9 AM."

[0065] Learning and Personalization

[0066] Terminal: Sends the user's usage history to the server.

[0067] Server: Stores interaction data and analyzes it to learn user behavior patterns and preferences.

[0068] Server: Generates more personalized responses to users based on the learning results.

[0069] The device converts this personalized response into speech and provides it to the user.

[0070] In this way, the system allows users to naturally interact with the character they select, providing various support such as daily tasks and setting reminders. The interaction is personalized based on the user's preferences and behavioral patterns, providing an experience that makes it seem as if the user and the character are living together.

[0071] The processing flow will be explained below.

[0072] Step 1:

[0073] Device: The user launches the app, the app initializes, and the login screen appears.

[0074] Step 2:

[0075] User: Enter your existing account information and click the login button.

[0076] Step 3:

[0077] Terminal: Encodes the entered authentication information and sends it to the server.

[0078] Step 4:

[0079] Server: Receives the authentication information, checks it against a database, and if authentication is successful, starts the session and sends the information back to the device.

[0080] Step 5:

[0081] Terminal: Checks the authentication status, and displays the main screen if successful, or an error message if unsuccessful.

[0082] Step 6:

[0083] Terminal: On the main screen, sends a request to the server to display a list of characters available to the user.

[0084] Step 7:

[0085] Server: Retrieves available character information from the character database and returns it to the device.

[0086] Step 8:

[0087] Terminal: Displays a list of characters the user can choose from.

[0088] Step 9:

[0089] User: Select the desired character from the list and open the detailed settings screen.

[0090] Step 10:

[0091] User: Customize the character's personality, voice pitch, specific expressions, etc., and press the button to save the settings.

[0092] Step 11:

[0093] Device: Sends customized configuration data to the server.

[0094] Step 12:

[0095] Server: Stores the received configuration data with the user's profile.

[0096] Step 13:

[0097] User: Speaks to the character via text or voice.

[0098] Step 14:

[0099] Device: In the case of voice input, the voice recognition function converts the input into text and sends the text to the server.

[0100] Step 15:

[0101] Server: Passes the received text to a natural language processing engine and analyzes its content.

[0102] Step 16:

[0103] Server: Based on the results of natural language processing, a generative AI generates an appropriate response sentence.

[0104] Step 17:

[0105] Server: Sends the generated response to the terminal.

[0106] Step 18:

[0107] Terminal: The returned response sentence is passed to the speech synthesis engine to generate speech.

[0108] Step 19:

[0109] Terminal: Plays the generated audio and provides responses to the user.

[0110] Step 20:

[0111] User: Set a new task, for example, "Set a reminder for a meeting tomorrow at 9 AM."

[0112] Step 21:

[0113] Terminal: Sends a request to the server for task configuration.

[0114] Step 22:

[0115] Server: Analyzes received tasks and adds reminders to the task list at the set date and time.

[0116] Step 23:

[0117] Server: Generates a confirmation message for adding the task and sends it to the device.

[0118] Step 24:

[0119] Terminal: Converts the confirmation message into speech and provides it to the user.

[0120] Step 25:

[0121] Terminal: Periodically sends the user's interaction history to the server.

[0122] Step 26:

[0123] Server: Stores interaction data and associates it with user profiles.

[0124] Step 27:

[0125] Server: Analyzes the stored data and learns user behavior patterns and preferences.

[0126] Step 28:

[0127] Server: Generates more personalized responses based on the learning results.

[0128] Step 29:

[0129] Terminal: Converts the personalized response into speech and provides it to the user.

[0130] Example 1

[0131] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] Conventional dialogue systems have struggled to achieve natural interactions with users. In particular, they have had challenges in accurately analyzing user input and providing appropriate responses in a natural manner. Furthermore, technology for providing personalized responses based on user preferences and behavioral patterns has also been inadequate. This has led to problems such as a poor user experience and a decline in system utilization.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0134] In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for using a speech recognition engine to convert the user's speech into text, means for analyzing the input using a natural language processing engine, means for generating a response using a generative artificial intelligence model, and means for converting the response into speech using a speech synthesis engine. This enables the user's input to be accurately analyzed and a response to be provided in a natural manner. Furthermore, by storing the user's profile and behavioral history and providing personalized responses based on them, the quality of the user experience can be improved and the system utilization rate can be increased.

[0135] "Means for receiving input from a user" refers to the facility by which a user provides input to the system in text or speech form.

[0136] The "means for analyzing user input and generating an appropriate response" is a function for analyzing input data received from a user and generating an appropriate response based on the content of the input data.

[0137] The "means for converting the generated response into speech and providing it to the user" is a function for converting the generated response in text format into speech format and playing it back to the user.

[0138] A "speech recognition engine that converts user speech into text" is a software component that converts user spoken speech into text format.

[0139] A "natural language processing engine" is a software component that analyzes text input from a user and understands its meaning.

[0140] A "generative artificial intelligence model" is a machine learning model for generating responses based on analyzed text.

[0141] A "speech synthesis engine" is a software component for converting generated text responses into speech form.

[0142] "Means for storing user profiles and behavioral history" refers to a function for storing users' personal information and past interaction data.

[0143] "Means for analyzing stored profiles and behavioral history and generating responses based on user preferences" refers to a function that analyzes stored data to understand the user's preferences and behavioral patterns and generates responses based on them.

[0144] The "means for learning user behavior patterns and preferences based on interaction data" is a function that utilizes the dialogue history with the user to learn the user's behavior patterns and preferences.

[0145] The "means for saving character information and providing a character based on the user's selection" is a function for saving information about each character and providing that character based on the user's selection.

[0146] "Means for customizing character personality and voice settings" is a function that allows the user to adjust settings related to the character's personality and voice.

[0147] The "means for saving customized settings" is a function for saving changes to character settings made by the user.

[0148] The present invention is a system for interacting with a user that utilizes advanced technology to achieve natural interaction by analyzing user input, generating appropriate responses, and providing them to the user in spoken form.

[0149] System Overview

[0150] Hardware and software used

[0151] This system uses the following hardware and software:

[0152] Speech recognition engine: Uses a common speech recognition technology. Here, we use the widely used Google® Speech-to-Text as an example.

[0153] Natural language processing engine: An engine for analyzing the meaning of text data, using advanced natural language processing techniques such as OpenAI® GPT-4®.

[0154] Generative AI model: A machine learning model that generates appropriate responses based on user input.

[0155] Speech synthesis engine: An engine for converting the generated text response into speech, specifically Amazon Polly.

[0156] User-System Interaction

[0157] User authentication / login

[0158] The user launches the dedicated app on their device and enters their account information (username and password) on the login screen. The device sends this information to the server, which then collates it with a database for authentication. If authentication is successful, the server creates a session and returns that information to the device. The device checks the authentication status based on the received session information and displays the main screen.

[0159] Character Selection and Settings

[0160] When the user taps the "Select Character" button on the main screen, the device sends a request for a character list to the server. The server retrieves character information from the database and sends it back to the device. The device displays the received character list on the screen, and the user selects the desired character. The user can further customize the character's personality and voice pitch on the detailed settings screen, and these settings are sent to the server. The server saves the received settings in its database.

[0161] Interaction Generation

[0162] Users can provide text or voice input to the character. For voice input, the device captures the audio and converts it into text using Google Speech-to-Text. This text is sent to a server, which passes it to a natural language processing engine (GPT-4) for analysis. Based on the analysis results, a generative AI model generates an appropriate response, which is then sent back from the server to the device. The device then converts the received response into speech using a speech synthesis engine (Amazon Polly) and plays it back to the user.

[0163] Specific examples

[0164] Morning greetings and schedule check

[0165] When a user says "Good morning!", the device converts the speech to text and sends it to the server. The server generates a response, "Good morning! Would you like to check today's schedule?" and sends it back to the device. The device converts the response to speech and plays it to the user: "Good morning! Would you like to check today's schedule?"

[0166] Adding a task

[0167] When a user types "Set a reminder for a meeting tomorrow at 9 AM," the device sends the text to the server. The server parses the instruction and sets a reminder for the specified date and time. It generates a confirmation message saying "A reminder for a meeting tomorrow at 9 AM" and sends it back to the device. The device converts it into speech and provides it to the user: "A reminder for a meeting tomorrow at 9 AM."

[0168] Learning and Personalization

[0169] Every time a user uses the system, their usage history is sent to the server and stored in a database. The server analyzes this data and learns the user's behavioral patterns and preferences. Based on this learning, the system can generate more personalized responses. For example, if a user frequently adds a specific task, the system will provide information and suggestions related to that task.

[0170] In this way, the system of the present invention uses advanced speech recognition, natural language processing, and generative artificial intelligence models to achieve natural interactions with users and support daily task management and information provision. Examples of prompt sentences include "Good morning!", "Set a reminder for a meeting tomorrow at 9:00 AM," and "What's the weather like today?"

[0171] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0172] Step 1:

[0173] Device: The user launches the dedicated app and enters their account information (username and password) on the login screen. This input data is sent to the server for login authentication.

[0174] Input: Username, Password

[0175] Output: Login request sent to the server

[0176] Step 2:

[0177] Server: Receives the user's account information and checks it against a database. If authentication is successful, the server creates a session and sends the information back to the device. If authentication fails, it sends back an error message.

[0178] Input: Username, Password

[0179] Data processing: database matching, session generation

[0180] Output: Session information or error message

[0181] Step 3:

[0182] Terminal: Checks the authentication status based on the session information received from the server and displays the main screen. If an error message is returned, the error message is displayed.

[0183] Input: Session information or error message

[0184] Output: Main screen or error message display

[0185] Step 4:

[0186] User: Tap the "Character Selection" button on the main screen.

[0187] Input: Tap

[0188] Output: Character selection request

[0189] Step 5:

[0190] Terminal: Sends a character selection request to the server.

[0191] Input: Character selection request

[0192] Output: Character selection request sent to the server

[0193] Step 6:

[0194] Server: Retrieves character information from the database and sends it back to the device.

[0195] Input: Character selection request

[0196] Data processing: Database matching, character information acquisition

[0197] Output: Retrieved character information

[0198] Step 7:

[0199] Terminal: Display the received character information on the screen.

[0200] Input: Character information

[0201] Output: Display the character list

[0202] Step 8:

[0203] User: Select the desired character from the list and customize their personality and voice pitch in the advanced settings screen.

[0204] Input: Character selection, customization settings

[0205] Output: Customization settings information

[0206] Step 9:

[0207] Device: Sends customized configuration information to the server.

[0208] Input: Customization setting information

[0209] Output: Customization settings sent to the server

[0210] Step 10:

[0211] Server: Saves customizations to a database.

[0212] Input: Customization setting information

[0213] Data processing: Database storage

[0214] Output: Save complete message

[0215] Step 11:

[0216] User: Provides text or voice input to the character.

[0217] Input: Text or voice input

[0218] Output: Input data

[0219] Step 12:

[0220] On the device: For voice input, the speech is converted to text using Google Speech-to-Text, and the converted text is sent to the server.

[0221] Input: Voice input

[0222] Data processing: speech recognition, text conversion

[0223] Output: Text data

[0224] Step 13:

[0225] Terminal: Sends text input or converted text to the server.

[0226] Input: Text data

[0227] Output: Text data sent to the server

[0228] Step 14:

[0229] Server: Passes the received text to a natural language processing engine (GPT-4) and analyzes its content.

[0230] Input: Text data

[0231] Data Processing: Natural Language Processing

[0232] Output: Analysis results

[0233] Step 15:

[0234] Server: Based on the analysis results, the generative AI model generates an appropriate response, which is then sent back to the device.

[0235] Input: Analysis results

[0236] Data processing: response generation

[0237] Output: Response data

[0238] Step 16:

[0239] Terminal: Passes the received response data to the speech synthesis engine (Amazon Polly) and converts it into speech.

[0240] Input: Response data

[0241] Data processing: voice synthesis

[0242] Output: Audio data

[0243] Step 17:

[0244] Terminal: Plays the generated audio to the user.

[0245] Input: Audio data

[0246] Output: Audio playback

[0247] Specific operation example

[0248] Examples of morning greetings and schedule confirmation

[0249] 1. User: Says "Good morning!"

[0250] 2. On the device: Capture the user's voice and convert it to text using Google Speech-to-Text.

[0251] 3. Terminal: Send the converted text "Good morning!" to the server.

[0252] 4. Server: Passes the received text "Good morning!" to the natural language processing engine (GPT-4) and instructs it to generate a response.

[0253] 5. Server: Generates a response saying "Good morning! Would you like to check today's schedule?" and sends it back to the device.

[0254] 6. Terminal: The returned response is passed to the speech synthesis engine (Amazon Polly) and converted into speech.

[0255] 7. Terminal: Play the audio to the user: "Good morning! Would you like to check today's schedule?"

[0256] Example of adding a task

[0257] 1. User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[0258] 2. Terminal: Sends the entered text to the server.

[0259] 3. Server: Passes the received text to the natural language processing engine (GPT-4) for analysis.

[0260] 4. Server: Registers the instruction "Set a meeting reminder for tomorrow at 9:00" in the scheduler.

[0261] 5. Server: Generates a confirmation message saying "Your meeting reminder has been set for tomorrow at 9 AM" and sends it back to the device.

[0262] 6. Device: Passes the confirmation message to the speech synthesis engine (Amazon Polly) and converts it into speech.

[0263] 7. Device: The generated speech "A meeting reminder has been set for tomorrow at 9:00" is provided to the user.

[0264] Examples of prompt statements

[0265] "good morning!"

[0266] "Set a meeting reminder for tomorrow at 9 AM"

[0267] "How's the weather today?"

[0268] (Application example 1)

[0269] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0270] Conventional interactive systems have difficulty in providing natural dialogue between users and characters, or in providing information based on the user's preferences. Furthermore, they lack the ability to instantly recommend appropriate information in response to user questions, limiting the user experience. This makes it difficult for users to fully enjoy the dialogue with characters and obtain personalized responses.

[0271] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0272] In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for converting speech input into text, means for analyzing the text input and acquiring and recommending information, and means for saving and reusing information based on user selection, thereby enabling the user to enjoy natural dialogue with a character and receive personalized information recommendations through the interaction.

[0273] A "means for receiving input from a user" is a device or interface that receives input when a user asks a question or gives a command by text or voice.

[0274] The "means for analyzing user input and generating an appropriate response" refers to a device or algorithm that analyzes user input and generates an optimal response for the user based on the content of the input.

[0275] The "means for converting the generated response into speech and providing it to the user" refers to a device or software for converting the generated text response into speech and playing the speech to the user.

[0276] A "voice-to-text converter" is a device or software that converts a user's speech into text in a machine-parseable format.

[0277] The "means for analyzing text input and retrieving and recommending information" refers to a device or algorithm that analyzes the content input as text, retrieves appropriate information, and recommends it to the user.

[0278] The "means for saving and reusing information based on user selection" refers to a device or software that saves information selected or set by the user and reuses it in a later interaction.

[0279] "Means for storing user profiles and behavioral histories" refers to devices or software that store users' personal information and behavioral histories in a database.

[0280] "Means for analyzing stored profile and behavioral history and generating responses based on user preferences" refers to devices or algorithms that analyze stored data and generate responses tailored to the user's preferences.

[0281] The "means for storing character information and providing a character based on a user's selection" refers to a device or software that stores information about a character and provides the character selected by the user on the system.

[0282] "Means for customizing the character's personality and voice settings" refers to devices or software that allow the user to customize the character's personality, voice pitch, etc.

[0283] The "means for personalizing information provided by a character based on user preferences" refers to a device or algorithm that customizes information provided by a character based on user preferences.

[0284] System Overview

[0285] The present invention is a system that includes a set of functions such as means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for converting speech input into text, means for analyzing the text input and retrieving and recommending information, and means for storing and reusing information based on user selection.

[0286] Hardware and software used

[0287] Hardware: Smartphone

[0288] software:

[0289] Speech recognition engine: Google Cloud Speech-to-Text

[0290] Natural Language Processing Engine: OpenAI GPT-4

[0291] Speech synthesis engine: Amazon Polly

[0292] Database: Firebase

[0293] Interface: Flutter (registered trademark)

[0294] Operating procedure

[0295] User authentication / login

[0296] The user launches a dedicated app on their smartphone and logs in by entering their account information (username and password). The server receives the user's authentication information and verifies it against a database to ensure it is correct. If authentication is successful, the server starts a session and returns the information to the device. The device then checks the authentication status, and displays the main screen if successful, or an error message if login is unsuccessful.

[0297] Character Selection and Settings

[0298] The user can view a list of available characters on the main screen and select the desired character. It is also possible to customize the character's personality, voice pitch, etc. The customized settings are sent from the device to the server, which then stores the received settings data along with the user's profile.

[0299] Interaction Generation

[0300] The user provides text or voice input to the character. For voice input, the device converts the speech to text using Google Cloud Speech-to-Text. The converted text input is sent to the server, which passes it to a natural language processing engine (GPT-4) for analysis. Based on the analysis results, the generative AI (GPT-4) generates an appropriate response, which is sent back from the server to the device. The device then converts the returned text response into speech using Amazon Polly and plays it back to the user.

[0301] Specific examples

[0302] Movie Recommendations

[0303] A user speaks to their smartphone, saying, "Tell me some recommended action movies." The device converts the speech into text using Google Cloud Speech-to-Text and sends it to the server. The server analyzes the text, obtains the appropriate movie information, and generates a response. For example, GPT-4 generates a response such as, "The recommended action movie is 'Action Movie A.'" The generated response is converted into audio using Amazon Polly and played back to the user.

[0304] Prompt example

[0305] A specific example of a prompt is as follows:

[0306] User Question: What are some good action movies? Movie Type: Action. Please recommend both new and classic movies.

[0307] In this way, a system can be constructed that allows natural dialogue via characters and provides appropriate information in response to specific user questions.

[0308] As described above, the system of the present invention can provide personalized information recommendations while allowing the user to naturally interact with a character selected by the user, thereby improving the user experience.

[0309] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0310] Step 1:

[0311] The user launches a dedicated app on their smartphone and logs in by entering their account information (username and password). The input data is sent from the device to the server. The server receives this input, checks it against a database to see if it is correct, and returns the results. The server outputs session information if authentication is successful, or an error message if it fails.

[0312] Step 2:

[0313] If login is successful, the terminal displays the main screen to the user. A list of characters available to the user is displayed on the screen, waiting for a selection. To generate the list, the terminal obtains character information from the server and provides it to the user. Character information consists of the character's name, appearance, personality, etc.

[0314] Step 3:

[0315] The user selects the desired character and customizes the character's personality, voice pitch, etc. on the detailed settings screen. This setting data is sent from the device to the server, which saves it along with the user's profile. The server outputs a message confirming that the save was successful.

[0316] Step 4:

[0317] The user begins a dialogue with the selected character. The user speaks, "Tell me some good action movies." The voice input is converted to text by the device's Google Cloud Speech-to-Text engine. The converted text is then sent to the server.

[0318] Step 5:

[0319] The server passes the received text to a natural language processing engine (GPT-4). The server processes the data to generate a prompt, formatting it like this: "User question: What action movies do you recommend? Movie type: Action. Please recommend both new and classic movies." GPT-4 then parses this input and generates an appropriate response. The server's output is the generated text response.

[0320] Step 6:

[0321] The server returns the generated response text to the device. The device converts the response text into speech using Amazon Polly. Here, the input is the text response from GPT-4, and the output is the audio data. The device plays this audio and provides it to the user as "The recommended action movie is 'Action Movie A'."

[0322] Step 7:

[0323] When a user selects a particular movie, the selection is sent from the device to the server, which stores this information as a user history and uses it for future recommendations. The stored information is also used as feedback data to personalize future interactions.

[0324] By implementing each of the above steps in order, users will be able to receive personalized movie recommendations while enjoying natural interactions with characters.

[0325] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0326] This invention is a system that allows users to interact naturally with popular characters. It analyzes user input, generates appropriate responses, and translates them into voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized responses become possible. Below, the system's program and its processing are explained in natural language, with specific examples.

[0327] System Overview

[0328] User authentication / login

[0329] Device: The user launches the dedicated app and logs in by entering their account information (username and password).

[0330] Server: Receives the user's authentication information and checks it against the database to see if it is correct. If authentication is successful, it starts a session and sends the information back to the terminal. The terminal checks the authentication status and displays the main screen if successful, or an error message if unsuccessful.

[0331] Character Selection and Settings

[0332] Terminal: Sends a request to the server to display a list of characters available to the user on the main screen.

[0333] Server: Retrieves available character information from the character database and returns it to the device.

[0334] User: Select the desired character from the list and customize their personality, voice pitch, etc. on the detailed settings screen. The customized settings are sent from the device to the server.

[0335] Server: Stores the received configuration data with the user's profile.

[0336] Interaction Generation and Emotion Recognition

[0337] User: Enter text or voice input to the character. Voice input is converted to text using the device's voice recognition function.

[0338] Terminal: Sends user text input to the server.

[0339] Server: Passes the received text to a natural language processing engine for content analysis. Next, an emotion engine is used to recognize emotions from the user's input. Based on this emotion data, a generative AI generates an appropriate response. The response is then sent back from the server to the device.

[0340] On the device: The returned text response is passed to a speech synthesis engine to generate speech and play it back to the user.

[0341] Specific examples

[0342] Morning greetings and schedule check

[0343] User: Say "Good morning."

[0344] Device: Converts the user's speech into text and sends it to the server.

[0345] Server: Analyzes the input text and uses the emotion engine to recognize that the user is in a refreshing mood. It generates a response such as "Good morning! What a lovely morning. Would you like to check today's schedule?" and sends it to the device.

[0346] Terminal: The returned response is converted into speech using a speech synthesis engine and played back to the user. It says, "Good morning! What a beautiful morning. Would you like to check today's schedule?"

[0347] Adding a task

[0348] User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[0349] Terminal: Sends input text to the server.

[0350] Server: Analyzes the text and sets a reminder for the specified date and time. The emotion engine recognizes that the user is a little impatient and generates a confirmation message saying, "I've set a reminder for your meeting tomorrow at 9am. Is there anything else I can help you with?"

[0351] Device: Converts a confirmation message to speech and plays it to the user. Says, "I've set a reminder for your meeting tomorrow at 9:00. Is there anything else I can help you with?"

[0352] Learning and Personalization

[0353] Terminal: Periodically sends the user's interaction history to the server.

[0354] Server: Stores interaction data and emotion data, scans them, and learns user behavior patterns and emotional changes.

[0355] Server: Based on the learning results, the model is updated to generate personalized responses for the next interaction.

[0356] Terminal: Converts the personalized response into speech and provides it to the user.

[0357] In this way, the present invention generates responses based on user input and emotions, providing a natural interaction experience for the user. By responding to the user's preferences and emotions, the interaction becomes more satisfying and the relationship between the user and the character becomes deeper.

[0358] The processing flow will be explained below.

[0359] Step 1:

[0360] Device: The user launches the app, the app initializes, and the login screen appears.

[0361] Step 2:

[0362] User: Enter your existing account information and click the login button.

[0363] Step 3:

[0364] Terminal: Encodes the entered authentication information and sends it to the server.

[0365] Step 4:

[0366] Server: Receives the authentication information, checks it against a database, and if authentication is successful, starts the session and sends the information back to the device.

[0367] Step 5:

[0368] Terminal: Checks the authentication status, and displays the main screen if successful, or an error message if unsuccessful.

[0369] Step 6:

[0370] Terminal: On the main screen, sends a request to the server to display a list of characters available to the user.

[0371] Step 7:

[0372] Server: Retrieves available character information from the character database and returns it to the device.

[0373] Step 8:

[0374] Terminal: Displays a list of characters the user can choose from.

[0375] Step 9:

[0376] User: Select the desired character from the list and open the detailed settings screen.

[0377] Step 10:

[0378] User: Customize the character's personality, voice pitch, specific expressions, etc., and press the button to save the settings.

[0379] Step 11:

[0380] Device: Sends customized configuration data to the server.

[0381] Step 12:

[0382] Server: Stores the received configuration data with the user's profile.

[0383] Step 13:

[0384] User: Speaks to the character via text or voice.

[0385] Step 14:

[0386] Device: In the case of voice input, the voice recognition function converts the input into text and sends the text to the server.

[0387] Step 15:

[0388] Server: Passes the received text to a natural language processing engine and analyzes its content.

[0389] Step 16:

[0390] Server: Sends the parsed text to the emotion engine to recognize the user's emotion.

[0391] Step 17:

[0392] Server: Using the user's emotional data recognized by the emotion engine, the generative AI generates personalized responses.

[0393] Step 18:

[0394] Server: Sends the generated response to the terminal.

[0395] Step 19:

[0396] Terminal: The returned response sentence is passed to the speech synthesis engine to generate speech.

[0397] Step 20:

[0398] Terminal: Plays the generated audio and provides responses to the user.

[0399] Step 21:

[0400] User: Set a new task, for example, "Set a reminder for a meeting tomorrow at 9 AM."

[0401] Step 22:

[0402] Terminal: Sends a request to the server for task configuration.

[0403] Step 23:

[0404] Server: Analyzes received tasks and adds reminders to the task list at the set date and time.

[0405] Step 24:

[0406] Server: Generates a confirmation message for adding the task and sends it to the device.

[0407] Step 25:

[0408] Terminal: Converts the confirmation message into speech and provides it to the user.

[0409] Step 26:

[0410] Terminal: Periodically transmits the user's interaction history and emotion data to the server.

[0411] Step 27:

[0412] Server: Stores interaction and emotion data and associates it with user profiles.

[0413] Step 28:

[0414] Server: Analyzes the stored data and learns changes in user behavior patterns and emotions.

[0415] Step 29:

[0416] Server: Based on the learning results, generates personalized responses for the next interaction and updates the model.

[0417] Step 30:

[0418] Terminal: Converts the personalized response into speech and provides it to the user.

[0419] Example 2

[0420] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0421] Conventional dialogue systems simply respond to user input, making it difficult to recognize emotions in real time or generate personalized responses. Furthermore, they lack the ability to use the user's interaction history to improve future responses, resulting in low user satisfaction. Furthermore, limited options for customizing character personalities and settings make it difficult to provide a compelling experience for users.

[0422] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0423] In this invention, the server includes means for receiving voice or text input from a user, means for converting the user's voice input into text, natural language processing means for analyzing the user's input and generating an appropriate response, means for recognizing the user's emotions, and means for converting the generated response into voice and providing it to the user, thereby enabling recognition of the user's emotions in real time and generating personalized responses.

[0424] "Means for receiving voice or text input from a user" refers to a device that has an interface through which a user can provide input in the form of voice or text.

[0425] "Means for converting user voice input into text" refers to a device that uses voice recognition technology to convert speech into text form.

[0426] "Natural language processing means for analyzing user input and generating an appropriate response" refers to a device that uses natural language processing technology to analyze user input, understand the context and intent, and generate an appropriate response.

[0427] "Means for recognizing user emotions" refers to a device that uses analytical technology to extract and identify emotions from a user's voice or text.

[0428] The "means for converting the generated response into speech and providing it to the user" refers to a device that uses speech synthesis technology to convert a text response into speech and provide it to the user.

[0429] "Means for storing user profiles and behavioral history" refers to a device equipped with a database that stores users' personal information and past interaction data.

[0430] "Means for analyzing stored profiles and behavioral history to generate responses based on user preferences" refers to a device that uses technology to analyze stored data and generate individually optimized responses based on a user's preferences and past behavioral patterns.

[0431] "Means for storing character information and providing a character based on a user's selection" refers to a device that has a database and an interface for storing information on multiple characters and providing data on a character selected by a user.

[0432] "Means for customizing character personality and voice settings" refers to a device that uses an interface that allows a user to adjust or change a character's characteristics or voice features, and technology that stores those settings.

[0433] The present invention is a system for realizing natural interaction between a user and an interactive character, and is implemented by combining the following hardware and software.

[0434] System Overview

[0435] User authentication / login

[0436] Device:

[0437] The user launches a dedicated app and logs in by entering their username and password. When the user presses the "Login" button, the device encrypts the input information and sends it to the server. The hardware used is a typical computer device such as a smartphone, tablet, or PC. The software uses an encryption library (e.g., OpenSSL).

[0438] server:

[0439] The server compares the received authentication information with a database (e.g., MySQL (registered trademark)), and if authentication is successful, starts a session and returns the information to the terminal. The server software used may be a web server (e.g., Apache (registered trademark), Nginx) or a database server.

[0440] Device:

[0441] The device checks the received session token and displays the main screen if authentication is successful, or an error message if authentication is unsuccessful.The software used is a mobile app development framework (e.g., React Native, Flutter).

[0442] Character Selection and Settings

[0443] Device:

[0444] Send a request to the server to display the list of characters available to the user on the main screen, using an HTTP request.

[0445] server:

[0446] The server retrieves available character information from a character database (e.g., MongoDB) and returns it to the device. The returned data is in JSON format.

[0447] Device:

[0448] The device displays the received character information on the main screen. The user selects a character on the screen and customizes their personality, voice pitch, etc. on the detailed settings screen. The customized settings are then sent back to the server in JSON format.

[0449] server:

[0450] The server stores the received configuration data in a database together with the user's profile.

[0451] Interaction Generation and Emotion Recognition

[0452] User:

[0453] The user can input text or voice to the character. Voice input is converted to text using the device's voice recognition function (e.g., Google Speech-to-Text).

[0454] Device:

[0455] The device sends the user's text input to the server, using HTTP or WebSockets.

[0456] server:

[0457] The server passes the received text to a natural language processing engine (e.g., spaCy, Hugging Face Transformers) for content analysis. The server also uses an emotion recognition engine (e.g., a BERT-based model) to recognize the user's emotions. Based on this emotion data, the server generates an appropriate response using a generative AI model (e.g., OpenAI GPT-4) and sends it back to the device.

[0458] Device:

[0459] The device passes the returned text response to a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio and play it back to the user.

[0460] Specific examples

[0461] Morning greetings and schedule check

[0462] User:

[0463] Say "Good morning."

[0464] Device:

[0465] The user's speech is converted into text and sent to the server.

[0466] server:

[0467] The input text is analyzed using a natural language processing engine, and the emotion engine is used to recognize that the user is in a refreshing mood. The response "Good morning! What a lovely morning. Would you like to check today's schedule?" is generated and sent to the device.

[0468] Device:

[0469] The returned response is converted into speech by a speech synthesis engine and played back to the user.

[0470] Example prompts to input to a generative AI model:

[0471] "A user says good morning. He seems very refreshed. How would you respond?"

[0472] Adding a task

[0473] User:

[0474] Type "Set a meeting reminder for tomorrow at 9 AM."

[0475] Device:

[0476] Sends the input text to the server.

[0477] server:

[0478] It analyzes the text and sets a reminder for the specified date and time. The emotion engine recognizes that the user is a little impatient and generates a confirmation message saying, "I've set a reminder for your meeting tomorrow at 9:00. Is there anything else I can help you with?"

[0479] Device:

[0480] A confirmation message is converted into audio and played to the user.

[0481] Example prompts to input to a generative AI model:

[0482] "A user types that they want to set a reminder for a meeting tomorrow at 9 AM. They seem a little impatient. How would you respond?"

[0483] Learning and Personalization

[0484] Device:

[0485] The user's interaction history is periodically sent to the server.

[0486] server:

[0487] It stores interaction and emotion data, analyzes them, and learns user behavior patterns and emotional changes. Based on the learning results, it updates the model to generate personalized responses for the next interaction.

[0488] Device:

[0489] The personalized response is converted into speech and provided to the user.

[0490] In this way, the present invention generates responses based on user input and emotions, providing a natural interaction experience for the user. By responding to the user's preferences and emotions, the interaction becomes more satisfying and the relationship between the user and the character becomes deeper.

[0491] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0492] An explanation of the processing flow of this system's program, broken down into processing steps

[0493] Step 1:

[0494] User Login

[0495] Input: The user launches the dedicated app and enters their username and password.

[0496] Specific operation: When the user presses the "Login" button, the terminal encrypts the username and password and sends them to the server.

[0497] Output: Encrypted user credentials are sent to the server.

[0498] Step 2:

[0499] Authentication verification

[0500] Input: Encrypted user credentials

[0501] What happens: The server checks the received credentials against its database. It performs a database query to verify that the username and password match.

[0502] Output: Generates a result indicating whether authentication was successful or not, and sends it back to the terminal along with the session information.

[0503] Step 3:

[0504] Viewing the login results

[0505] Input: Authentication result and session information

[0506] Specific operation: The device checks the received authentication result, and displays the main screen if successful, or an error message if unsuccessful.

[0507] Output: Display of the main screen or error message.

[0508] Step 4:

[0509] Character Information Request

[0510] Input: Request required to display the main screen

[0511] Specific operation: The device sends a request to the server to obtain available character information.

[0512] Output: Sends a request to the server to get character data.

[0513] Step 5:

[0514] Acquiring and returning character information

[0515] Input: Character data acquisition request

[0516] Specific operation: The server retrieves character information from the character database and returns it to the device in JSON format.

[0517] Output: Returns character information to the terminal.

[0518] Step 6:

[0519] Displaying the character list

[0520] Input: Character information in JSON format

[0521] Specific operation: The device analyzes character information and displays it in a list on the main screen.

[0522] Output: Display the character list on the main screen.

[0523] Step 7:

[0524] Character selection and detailed settings

[0525] Input: User-selected character and customization settings

[0526] Specific operation: The user selects a character and customizes the character's personality, voice pitch, etc. in the detailed settings screen. These settings are sent from the device to the server.

[0527] Output: Sends customization information to the server.

[0528] Step 8:

[0529] Saving the configuration information

[0530] Input: Customization setting information

[0531] Specific operation: The server stores the received customization setting information in the database.

[0532] Output: Save customized settings information to a database.

[0533] Step 9:

[0534] User Input and Conversion

[0535] Input: User voice or text input

[0536] Specific operation: The user inputs to the character by voice or text. In the case of voice input, the device converts the voice into text in real time.

[0537] Output: The converted text data is sent to the server.

[0538] Step 10:

[0539] Natural Language Processing and Emotion Recognition

[0540] Input: Text data

[0541] Specific operation: The server passes the text data to a natural language processing engine for analysis, and recognizes emotions using an emotion engine.

[0542] Output: Analysis results and emotion data.

[0543] Step 11:

[0544] Generate a response

[0545] Input: Analysis results and emotion data

[0546] Specific operation: The server uses a generative AI model (e.g., OpenAI GPT-4) to generate an appropriate response.

[0547] Output: Sends the generated response to the terminal.

[0548] Step 12:

[0549] Audio generation and playback

[0550] Input: Generated response

[0551] Specific operation: The device passes the response sentence to a speech synthesis engine to generate speech and play it back to the user.

[0552] Output: Playback of the generated audio.

[0553] Example prompt

[0554] 1. Morning greeting and schedule prompt:

[0555] "A user says good morning. He seems very refreshed. How would you respond?"

[0556] 2. Prompt for adding a task:

[0557] "A user types that they want to set a reminder for a meeting tomorrow at 9 AM. They seem a little impatient. How would you respond?"

[0558] (Application example 2)

[0559] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0560] Conventional interaction systems have fixed responses to user inputs and have not been able to realize natural dialogue that takes into account the user's emotions and gaze. Furthermore, in the work environment, there is a lack of efficient and safe work support, which has led to the issue of workers being unable to quickly obtain the information they need.

[0561] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into voice and providing it to the user, means for recognizing the user's emotions, and means for tracking the user's gaze. This enables natural dialogue that takes the user's emotions and gaze into consideration. The server also includes means for saving a user's profile and behavioral history, means for analyzing the saved profile and behavioral history and generating a response based on the user's preferences, means for visually providing the generated response, and means for displaying information via a visual output device worn by the user. This enables workers to efficiently and safely obtain necessary information in the work environment and support them. The server also includes means for saving character information and providing a character based on the user's selection, means for customizing the character's personality and voice settings, and means for creating appropriate prompt sentences based on a generative AI model. This allows for more personalized responses to the user and an improved dialogue experience.

[0562] "Means for receiving input from a user" refers to hardware and software for capturing and transmitting information entered by a user through speech or text to the system.

[0563] "Means for analyzing user input and generating an appropriate response" refers to a technology that analyzes received user input information using a natural language processing engine or the like and generates a response based on that information.

[0564] The "means for converting the generated response into speech and providing it to the user" is a technology that uses a speech synthesis engine to convert the generated text response into speech and plays it aloud to the user through a speaker or the like.

[0565] "Means for recognizing user emotions" refers to technology that analyzes emotions from the user's voice, text, facial expressions, etc., and determines the user's current feelings and state.

[0566] "Means for tracking the user's gaze" refers to technology that uses a camera or sensor to track the direction or object the user is looking at and records it as gaze information.

[0567] "Means for storing a user's profile and behavior history" refers to technology for storing a user's personal information, past operation history, behavior patterns, etc. in a storage device.

[0568] "Means for analyzing stored profiles and behavioral history and generating responses based on the user's preferences" refers to technology that analyzes stored data and generates personalized responses tailored to the user's tendencies and preferences.

[0569] The "means for visually presenting the generated response" is a technique for displaying the generated response on a display device as visual information such as text, images, or videos.

[0570] "Means for displaying information via a visual output device worn by a user" refers to technology for displaying information through a device worn by a user, such as smart glasses or a head-mounted display.

[0571] "Means for saving character information and providing characters based on user selection" refers to technology that saves character attributes and settings in a database and provides the character on screen or via audio based on the user's selection.

[0572] "Means for customizing character personality and voice settings" refers to technology that allows a user to change the personality, voice pitch, tone, etc. of a character selected by the user to suit the user's preferences.

[0573] "Means for creating appropriate prompt sentences based on a generative AI model" refers to a technology that uses an AI model to generate optimal response sentences based on input from a user.

[0574] The present invention provides a system that supports factory work by enabling natural interactions with input from users. This system allows characters to interact with the user through smart glasses worn by the user, improving work efficiency and safety. Detailed embodiments for implementing the present invention are described below.

[0575] System Overview

[0576] The system consists of the following main components:

[0577] 1. A means of receiving input from the user

[0578] Hardware: Smart glasses (e.g., Google Glass®)

[0579] Process: Capture the user's voice and gaze data and send it to the system.

[0580] 2. A means of analyzing user input and generating an appropriate response

[0581] Software: Speech recognition software (e.g., Google Cloud Speech-to-Text), natural language processing engines (e.g., OpenAI's GPT-4)

[0582] Process: Convert the captured audio into text, analyze the text and generate an appropriate response.

[0583] 3. A means of converting the generated response into speech and providing it to the user

[0584] Software: Speech synthesis engine (e.g. Amazon Polly)

[0585] Process: The generated text is converted into speech and provided to the user through the smart glasses speaker.

[0586] 4. How to Recognize User Emotions

[0587] Software: Emotion recognition engine (e.g., Microsoft® Azure® Emotion API)

[0588] Process: Analyze the user's emotions from captured voice and gaze data and provide feedback to the system.

[0589] 5. A way to track the user's gaze

[0590] Hardware: Built-in camera in smart glasses

[0591] Process: Track the user's gaze in real time and analyze the gaze data.

[0592] 6. Means of storing user profiles and activity histories

[0593] Hardware: Cloud-based database

[0594] Process: Stores user personal information, past operation history, behavioral patterns, etc.

[0595] 7. A means of analyzing stored profiles and behavioral history to generate responses based on user preferences.

[0596] Software: Data analysis engine

[0597] Process: Analyze the stored data and generate a personalized response based on the user's preferences.

[0598] 8. A visual means of providing the generated response

[0599] Hardware: Smart glasses display

[0600] Process: The generated response is visually displayed and presented to the user.

[0601] 9. Means for storing character information and providing characters based on user selection

[0602] Hardware: Cloud-based database

[0603] Process: Stores character attributes and settings and provides characters based on user selections.

[0604] 10. A way to customize character personalities and voice settings

[0605] Software: Character Setting Engine

[0606] Process: Modify the personality, voice pitch, tone, etc. of the user-selected character to suit the user's preferences.

[0607] 11. A means of creating appropriate prompts based on generative AI models

[0608] Software: Generative AI models (e.g., OpenAI's GPT-4)

[0609] Process: Use an AI model to generate optimal responses based on user input.

[0610] Specific examples

[0611] Check the checklist before starting work

[0612] 1. User: Talk to the character and say, "Check today's work."

[0613] 2. Smart glasses: Capture the user's voice and send it to voice recognition software.

[0614] 3. Server: Converts the input speech into text and passes it to the natural language processing engine.

[0615] 4. Natural language processing engine: Analyzes the text and generates a corresponding checklist.

[0616] 5. Emotion Recognition Engine: Recognizes when the user is focused.

[0617] 6. Generative AI: Generates a response such as, "Please be mindful of safety as you work today. Your first task is maintenance on Machine A. Would you like to review the checklist?"

[0618] 7. Speech synthesis engine: Converts the response into speech and provides it to the user through the smart glasses' speaker.

[0619] 8. Smart glasses: Visually display the generated text on a display.

[0620] Prompt Sentence Examples

[0621] Character Name: Factory character "Mechatrobo"

[0622] Input sentence: Tell the character "Tell me how to operate machine B"

[0623] Emotions: A little anxious

[0624] Response: Mecha-Robo: "I'll teach you how to operate Machine B. First, pull down the lever on the left. Next, press the button in the middle. If you have any trouble, please stay calm and let's go through each step one by one."

[0625] In this way, the system of the present invention improves work efficiency and safety by analyzing the user's voice and emotions and providing visual and audio responses based on those analyses. By personalizing responses based on the user's profile and behavioral history, the system provides higher satisfaction and enhances its utility in the work environment.

[0626] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0627] Step 1:

[0628] User: The user puts on the smart glasses and says, "See what I did today."

[0629] Input: User's voice

[0630] Output: Capture of audio data

[0631] How it works: The smart glasses capture the user's voice with a built-in microphone.

[0632] Step 2:

[0633] Device: The smart glasses capture and transmit the audio to the voice recognition software.

[0634] Input: Audio data

[0635] Output: Text data

[0636] How it works: Uses a speech recognition engine (e.g. Google Cloud Speech-to-Text) to convert input voice data into text.

[0637] Step 3:

[0638] Terminal: Sends text data to the server.

[0639] Input: Text data

[0640] Output: Sends text data to the server

[0641] How it works: Smart glasses send text data over the network to a server.

[0642] Step 4:

[0643] Server: The server analyzes the received text data.

[0644] Input: Text data

[0645] Output: Analysis results

[0646] How it works: It uses a natural language processing engine (e.g., OpenAI's GPT-4) to analyze text data and understand user intent.

[0647] Step 5:

[0648] Server: The server recognizes the user's emotions using an emotion recognition engine.

[0649] Input: Text data

[0650] Output: Emotion data

[0651] How it works: It uses an emotion recognition engine (e.g. Microsoft Azure's Emotion API) to analyze the user's emotions from the input text data.

[0652] Step 6:

[0653] Server: The server generates an appropriate response using a generative AI model based on the analysis results and emotion data.

[0654] Input: Text data, emotion data

[0655] Output: Response text

[0656] How it works: Uses a generative AI model (e.g., OpenAI's GPT-4) to generate a response text based on the analysis results and sentiment data.

[0657] Examples:

[0658] Character name: Factory character "Work Robot"

[0659] Input: "Check today's work"

[0660] Emotion: Focused

[0661] Response: Worker robot: "Please be mindful of safety as you work today. The first task is maintenance on Machine A. Would you like to review the checklist?"

[0662] "

[0663] Step 7:

[0664] Server: Pass the generated response text to the speech synthesis engine.

[0665] Input: Response text

[0666] Output: Audio data

[0667] What it does: Converts response text into speech using a text-to-speech engine (e.g., Amazon Polly).

[0668] Step 8:

[0669] Terminal: Sends audio data to smart glasses.

[0670] Input: Audio data

[0671] Output: Receives audio data

[0672] How it works: Audio data is sent over the network to smart glasses.

[0673] Step 9:

[0674] Terminal: The smart glasses play the audio data and provide it to the user.

[0675] Input: Audio data

[0676] Output: Play audio

[0677] What it does: Plays audio using the smart glasses' speakers and provides it to the user.

[0678] Step 10:

[0679] Terminal: Visually displays the generated response text.

[0680] Input: Response text

[0681] Output: Visual information

[0682] What it does: Visually display the response text on the smart glasses display to provide information to the user.

[0683] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0684] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0685] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0686] [Second embodiment]

[0687] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0688] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0689] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0690] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0691] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0692] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0693] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0694] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0695] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0696] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0697] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0698] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0699] This invention is a system that allows users to interact naturally with popular characters by analyzing input from the user and generating appropriate responses and translating them into voice. Below, the system program and its processing are explained in natural language, with specific examples.

[0700] System Overview

[0701] User authentication / login

[0702] Device: The user launches the dedicated app and logs in by entering their account information (username and password).

[0703] Server: Receives the user's authentication information and checks it against the database to see if it is correct. If authentication is successful, it starts a session and sends the information back to the terminal. The terminal checks the authentication status and displays the main screen if successful, or an error message if unsuccessful.

[0704] Character Selection and Settings

[0705] Terminal: The main screen displays a list of characters available to the user. The list is retrieved from the server.

[0706] User: Select the desired character and customize their personality, voice pitch, etc. on the detailed settings screen. The customized settings are sent from the device to the server.

[0707] Server: Stores the received configuration data with the user's profile.

[0708] Interaction Generation

[0709] User: Provides text or voice input to the character. Voice input is converted to text using the device's voice recognition function.

[0710] Terminal: Sends user text input to the server.

[0711] Server: Passes the received text to a natural language processing engine to analyze the input. Based on the analysis results, a generative AI generates an appropriate response. The response is then sent back from the server to the device.

[0712] On the device: The returned text response is passed to a speech synthesis engine to generate speech and play it back to the user.

[0713] Specific examples

[0714] Morning greetings and schedule check

[0715] User: Say "Good morning!"

[0716] Device: Converts the user's speech into text and sends it to the server.

[0717] Server: Analyzes the input text and generates a response such as "Good morning! Would you like to check today's schedule?". Then, sends the generated response back to the device.

[0718] Terminal: The returned response is converted into speech by a speech synthesis engine and played back to the user. It says, "Good morning! Would you like to check today's schedule?"

[0719] Adding a task

[0720] User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[0721] Terminal: Sends input text to the server.

[0722] Server: Parse the text and set a reminder for the specified date and time. Generate a confirmation message saying "Your reminder for the meeting has been set for tomorrow at 9 AM."

[0723] Device: Converts a confirmation message into speech and provides it to the user. Says, "I've set a reminder for a meeting tomorrow at 9 AM."

[0724] Learning and Personalization

[0725] Terminal: Sends the user's usage history to the server.

[0726] Server: Stores interaction data and analyzes it to learn user behavior patterns and preferences.

[0727] Server: Generates more personalized responses to users based on the learning results.

[0728] The device converts this personalized response into speech and provides it to the user.

[0729] In this way, the system allows users to naturally interact with the character they select, providing various support such as daily tasks and setting reminders. The interaction is personalized based on the user's preferences and behavioral patterns, providing an experience that makes it seem as if the user and the character are living together.

[0730] The processing flow will be explained below.

[0731] Step 1:

[0732] Device: The user launches the app, the app initializes, and the login screen appears.

[0733] Step 2:

[0734] User: Enter your existing account information and click the login button.

[0735] Step 3:

[0736] Terminal: Encodes the entered authentication information and sends it to the server.

[0737] Step 4:

[0738] Server: Receives the authentication information, checks it against a database, and if authentication is successful, starts the session and sends the information back to the device.

[0739] Step 5:

[0740] Terminal: Checks the authentication status, and displays the main screen if successful, or an error message if unsuccessful.

[0741] Step 6:

[0742] Terminal: On the main screen, sends a request to the server to display a list of characters available to the user.

[0743] Step 7:

[0744] Server: Retrieves available character information from the character database and returns it to the device.

[0745] Step 8:

[0746] Terminal: Displays a list of characters the user can choose from.

[0747] Step 9:

[0748] User: Select the desired character from the list and open the detailed settings screen.

[0749] Step 10:

[0750] User: Customize the character's personality, voice pitch, specific expressions, etc., and press the button to save the settings.

[0751] Step 11:

[0752] Device: Sends customized configuration data to the server.

[0753] Step 12:

[0754] Server: Stores the received configuration data with the user's profile.

[0755] Step 13:

[0756] User: Speaks to the character via text or voice.

[0757] Step 14:

[0758] Device: In the case of voice input, the voice recognition function converts the input into text and sends the text to the server.

[0759] Step 15:

[0760] Server: Passes the received text to a natural language processing engine and analyzes its content.

[0761] Step 16:

[0762] Server: Based on the results of natural language processing, a generative AI generates an appropriate response sentence.

[0763] Step 17:

[0764] Server: Sends the generated response to the terminal.

[0765] Step 18:

[0766] Terminal: The returned response sentence is passed to the speech synthesis engine to generate speech.

[0767] Step 19:

[0768] Terminal: Plays the generated audio and provides responses to the user.

[0769] Step 20:

[0770] User: Set a new task, for example, "Set a reminder for a meeting tomorrow at 9 AM."

[0771] Step 21:

[0772] Terminal: Sends a request to the server for task configuration.

[0773] Step 22:

[0774] Server: Analyzes received tasks and adds reminders to the task list at the set date and time.

[0775] Step 23:

[0776] Server: Generates a confirmation message for adding the task and sends it to the device.

[0777] Step 24:

[0778] Terminal: Converts the confirmation message into speech and provides it to the user.

[0779] Step 25:

[0780] Terminal: Periodically sends the user's interaction history to the server.

[0781] Step 26:

[0782] Server: Stores interaction data and associates it with user profiles.

[0783] Step 27:

[0784] Server: Analyzes the stored data and learns user behavior patterns and preferences.

[0785] Step 28:

[0786] Server: Generates more personalized responses based on the learning results.

[0787] Step 29:

[0788] Terminal: Converts the personalized response into speech and provides it to the user.

[0789] Example 1

[0790] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0791] Conventional dialogue systems have struggled to achieve natural interactions with users. In particular, they have had challenges in accurately analyzing user input and providing appropriate responses in a natural manner. Furthermore, technology for providing personalized responses based on user preferences and behavioral patterns has also been inadequate. This has led to problems such as a poor user experience and a decline in system utilization.

[0792] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0793] In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for using a speech recognition engine to convert the user's speech into text, means for analyzing the input using a natural language processing engine, means for generating a response using a generative artificial intelligence model, and means for converting the response into speech using a speech synthesis engine. This enables the user's input to be accurately analyzed and a response to be provided in a natural manner. Furthermore, by storing the user's profile and behavioral history and providing personalized responses based on them, the quality of the user experience can be improved and the system utilization rate can be increased.

[0794] "Means for receiving input from a user" refers to the facility by which a user provides input to the system in text or speech form.

[0795] The "means for analyzing user input and generating an appropriate response" is a function for analyzing input data received from a user and generating an appropriate response based on the content of the input data.

[0796] The "means for converting the generated response into speech and providing it to the user" is a function for converting the generated response in text format into speech format and playing it back to the user.

[0797] A "speech recognition engine that converts user speech into text" is a software component that converts user spoken speech into text format.

[0798] A "natural language processing engine" is a software component that analyzes text input from a user and understands its meaning.

[0799] A "generative artificial intelligence model" is a machine learning model for generating responses based on analyzed text.

[0800] A "speech synthesis engine" is a software component for converting generated text responses into speech form.

[0801] "Means for storing user profiles and behavioral history" refers to a function for storing users' personal information and past interaction data.

[0802] "Means for analyzing stored profiles and behavioral history and generating responses based on user preferences" refers to a function that analyzes stored data to understand the user's preferences and behavioral patterns and generates responses based on them.

[0803] The "means for learning user behavior patterns and preferences based on interaction data" is a function that utilizes the dialogue history with the user to learn the user's behavior patterns and preferences.

[0804] The "means for saving character information and providing a character based on the user's selection" is a function for saving information about each character and providing that character based on the user's selection.

[0805] "Means for customizing character personality and voice settings" is a function that allows the user to adjust settings related to the character's personality and voice.

[0806] The "means for saving customized settings" is a function for saving changes to character settings made by the user.

[0807] The present invention is a system for interacting with a user that utilizes advanced technology to achieve natural interaction by analyzing user input, generating appropriate responses, and providing them to the user in spoken form.

[0808] System Overview

[0809] Hardware and software used

[0810] This system uses the following hardware and software:

[0811] Speech recognition engine: Use a common speech recognition technology. Here, we will use the widely used Google Speech-to-Text as an example.

[0812] Natural Language Processing Engine: An engine for analyzing the meaning of text data, using advanced natural language processing techniques such as OpenAI GPT-4.

[0813] Generative AI model: A machine learning model that generates appropriate responses based on user input.

[0814] Speech synthesis engine: An engine for converting the generated text response into speech, specifically Amazon Polly.

[0815] User-System Interaction

[0816] User authentication / login

[0817] The user launches the dedicated app on their device and enters their account information (username and password) on the login screen. The device sends this information to the server, which then collates it with a database for authentication. If authentication is successful, the server creates a session and returns that information to the device. The device checks the authentication status based on the received session information and displays the main screen.

[0818] Character Selection and Settings

[0819] When the user taps the "Select Character" button on the main screen, the device sends a request for a character list to the server. The server retrieves character information from the database and sends it back to the device. The device displays the received character list on the screen, and the user selects the desired character. The user can further customize the character's personality and voice pitch on the detailed settings screen, and these settings are sent to the server. The server saves the received settings in its database.

[0820] Interaction Generation

[0821] Users can provide text or voice input to the character. For voice input, the device captures the audio and converts it into text using Google Speech-to-Text. This text is sent to a server, which passes it to a natural language processing engine (GPT-4) for analysis. Based on the analysis results, a generative AI model generates an appropriate response, which is then sent back from the server to the device. The device then converts the received response into speech using a speech synthesis engine (Amazon Polly) and plays it back to the user.

[0822] Specific examples

[0823] Morning greetings and schedule check

[0824] When a user says "Good morning!", the device converts the speech to text and sends it to the server. The server generates a response, "Good morning! Would you like to check today's schedule?" and sends it back to the device. The device converts the response to speech and plays it to the user: "Good morning! Would you like to check today's schedule?"

[0825] Adding a task

[0826] When a user types "Set a reminder for a meeting tomorrow at 9 AM," the device sends the text to the server. The server parses the instruction and sets a reminder for the specified date and time. It generates a confirmation message saying "A reminder for a meeting tomorrow at 9 AM" and sends it back to the device. The device converts it into speech and provides it to the user: "A reminder for a meeting tomorrow at 9 AM."

[0827] Learning and Personalization

[0828] Every time a user uses the system, their usage history is sent to the server and stored in a database. The server analyzes this data and learns the user's behavioral patterns and preferences. Based on this learning, the system can generate more personalized responses. For example, if a user frequently adds a specific task, the system will provide information and suggestions related to that task.

[0829] In this way, the system of the present invention uses advanced speech recognition, natural language processing, and generative artificial intelligence models to achieve natural interactions with users and support daily task management and information provision. Examples of prompt sentences include "Good morning!", "Set a reminder for a meeting tomorrow at 9:00 AM," and "What's the weather like today?"

[0830] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0831] Step 1:

[0832] Device: The user launches the dedicated app and enters their account information (username and password) on the login screen. This input data is sent to the server for login authentication.

[0833] Input: Username, Password

[0834] Output: Login request sent to the server

[0835] Step 2:

[0836] Server: Receives the user's account information and checks it against a database. If authentication is successful, the server creates a session and sends the information back to the device. If authentication fails, it sends back an error message.

[0837] Input: Username, Password

[0838] Data processing: database matching, session generation

[0839] Output: Session information or error message

[0840] Step 3:

[0841] Terminal: Checks the authentication status based on the session information received from the server and displays the main screen. If an error message is returned, the error message is displayed.

[0842] Input: Session information or error message

[0843] Output: Main screen or error message display

[0844] Step 4:

[0845] User: Tap the "Character Selection" button on the main screen.

[0846] Input: Tap

[0847] Output: Character selection request

[0848] Step 5:

[0849] Terminal: Sends a character selection request to the server.

[0850] Input: Character selection request

[0851] Output: Character selection request sent to the server

[0852] Step 6:

[0853] Server: Retrieves character information from the database and sends it back to the device.

[0854] Input: Character selection request

[0855] Data processing: Database matching, character information acquisition

[0856] Output: Retrieved character information

[0857] Step 7:

[0858] Terminal: Display the received character information on the screen.

[0859] Input: Character information

[0860] Output: Display the character list

[0861] Step 8:

[0862] User: Select the desired character from the list and customize their personality and voice pitch in the advanced settings screen.

[0863] Input: Character selection, customization settings

[0864] Output: Customization settings information

[0865] Step 9:

[0866] Device: Sends customized configuration information to the server.

[0867] Input: Customization setting information

[0868] Output: Customization settings sent to the server

[0869] Step 10:

[0870] Server: Saves customizations to a database.

[0871] Input: Customization setting information

[0872] Data processing: Database storage

[0873] Output: Save complete message

[0874] Step 11:

[0875] User: Provides text or voice input to the character.

[0876] Input: Text or voice input

[0877] Output: Input data

[0878] Step 12:

[0879] On the device: For voice input, the speech is converted to text using Google Speech-to-Text, and the converted text is sent to the server.

[0880] Input: Voice input

[0881] Data processing: speech recognition, text conversion

[0882] Output: Text data

[0883] Step 13:

[0884] Terminal: Sends text input or converted text to the server.

[0885] Input: Text data

[0886] Output: Text data sent to the server

[0887] Step 14:

[0888] Server: Passes the received text to a natural language processing engine (GPT-4) and analyzes its content.

[0889] Input: Text data

[0890] Data Processing: Natural Language Processing

[0891] Output: Analysis results

[0892] Step 15:

[0893] Server: Based on the analysis results, the generative AI model generates an appropriate response, which is then sent back to the device.

[0894] Input: Analysis results

[0895] Data processing: response generation

[0896] Output: Response data

[0897] Step 16:

[0898] Terminal: Passes the received response data to the speech synthesis engine (Amazon Polly) and converts it into speech.

[0899] Input: Response data

[0900] Data processing: voice synthesis

[0901] Output: Audio data

[0902] Step 17:

[0903] Terminal: Plays the generated audio to the user.

[0904] Input: Audio data

[0905] Output: Audio playback

[0906] Specific operation example

[0907] Examples of morning greetings and schedule confirmation

[0908] 1. User: Says "Good morning!"

[0909] 2. On the device: Capture the user's voice and convert it to text using Google Speech-to-Text.

[0910] 3. Terminal: Send the converted text "Good morning!" to the server.

[0911] 4. Server: Passes the received text "Good morning!" to the natural language processing engine (GPT-4) and instructs it to generate a response.

[0912] 5. Server: Generates a response saying "Good morning! Would you like to check today's schedule?" and sends it back to the device.

[0913] 6. Terminal: The returned response is passed to the speech synthesis engine (Amazon Polly) and converted into speech.

[0914] 7. Terminal: Play the audio to the user: "Good morning! Would you like to check today's schedule?"

[0915] Example of adding a task

[0916] 1. User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[0917] 2. Terminal: Sends the entered text to the server.

[0918] 3. Server: Passes the received text to the natural language processing engine (GPT-4) for analysis.

[0919] 4. Server: Registers the instruction "Set a meeting reminder for tomorrow at 9:00" in the scheduler.

[0920] 5. Server: Generates a confirmation message saying "Your meeting reminder has been set for tomorrow at 9 AM" and sends it back to the device.

[0921] 6. Device: Passes the confirmation message to the speech synthesis engine (Amazon Polly) and converts it into speech.

[0922] 7. Device: The generated speech "A meeting reminder has been set for tomorrow at 9:00" is provided to the user.

[0923] Examples of prompt statements

[0924] "good morning!"

[0925] "Set a meeting reminder for tomorrow at 9 AM"

[0926] "How's the weather today?"

[0927] (Application example 1)

[0928] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0929] Conventional interactive systems have difficulty in providing natural dialogue between users and characters, or in providing information based on the user's preferences. Furthermore, they lack the ability to instantly recommend appropriate information in response to user questions, limiting the user experience. This makes it difficult for users to fully enjoy the dialogue with characters and obtain personalized responses.

[0930] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0931] In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for converting speech input into text, means for analyzing the text input and acquiring and recommending information, and means for saving and reusing information based on user selection, thereby enabling the user to enjoy natural dialogue with a character and receive personalized information recommendations through the interaction.

[0932] A "means for receiving input from a user" is a device or interface that receives input when a user asks a question or gives a command by text or voice.

[0933] The "means for analyzing user input and generating an appropriate response" refers to a device or algorithm that analyzes user input and generates an optimal response for the user based on the content of the input.

[0934] The "means for converting the generated response into speech and providing it to the user" refers to a device or software for converting the generated text response into speech and playing the speech to the user.

[0935] A "voice-to-text converter" is a device or software that converts a user's speech into text in a machine-parseable format.

[0936] The "means for analyzing text input and retrieving and recommending information" refers to a device or algorithm that analyzes the content input as text, retrieves appropriate information, and recommends it to the user.

[0937] The "means for saving and reusing information based on user selection" refers to a device or software that saves information selected or set by the user and reuses it in a later interaction.

[0938] "Means for storing user profiles and behavioral histories" refers to devices or software that store users' personal information and behavioral histories in a database.

[0939] "Means for analyzing stored profile and behavioral history and generating responses based on user preferences" refers to devices or algorithms that analyze stored data and generate responses tailored to the user's preferences.

[0940] The "means for storing character information and providing a character based on a user's selection" refers to a device or software that stores information about a character and provides the character selected by the user on the system.

[0941] "Means for customizing the character's personality and voice settings" refers to devices or software that allow the user to customize the character's personality, voice pitch, etc.

[0942] The "means for personalizing information provided by a character based on user preferences" refers to a device or algorithm that customizes information provided by a character based on user preferences.

[0943] System Overview

[0944] The present invention is a system that includes a set of functions such as means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for converting speech input into text, means for analyzing the text input and retrieving and recommending information, and means for storing and reusing information based on user selection.

[0945] Hardware and software used

[0946] Hardware: Smartphone

[0947] software:

[0948] Speech recognition engine: Google Cloud Speech-to-Text

[0949] Natural Language Processing Engine: OpenAI GPT-4

[0950] Speech synthesis engine: Amazon Polly

[0951] Database: Firebase

[0952] Interface: Flutter

[0953] Operating procedure

[0954] User authentication / login

[0955] The user launches a dedicated app on their smartphone and logs in by entering their account information (username and password). The server receives the user's authentication information and verifies it against a database to ensure it is correct. If authentication is successful, the server starts a session and returns the information to the device. The device then checks the authentication status, and displays the main screen if successful, or an error message if login is unsuccessful.

[0956] Character Selection and Settings

[0957] The user can view a list of available characters on the main screen and select the desired character. It is also possible to customize the character's personality, voice pitch, etc. The customized settings are sent from the device to the server, which then stores the received settings data along with the user's profile.

[0958] Interaction Generation

[0959] The user provides text or voice input to the character. For voice input, the device converts the speech to text using Google Cloud Speech-to-Text. The converted text input is sent to the server, which passes it to a natural language processing engine (GPT-4) for analysis. Based on the analysis results, the generative AI (GPT-4) generates an appropriate response, which is sent back from the server to the device. The device then converts the returned text response into speech using Amazon Polly and plays it back to the user.

[0960] Specific examples

[0961] Movie Recommendations

[0962] A user speaks to their smartphone, saying, "Tell me some recommended action movies." The device converts the speech into text using Google Cloud Speech-to-Text and sends it to the server. The server analyzes the text, obtains the appropriate movie information, and generates a response. For example, GPT-4 generates a response such as, "The recommended action movie is 'Action Movie A.'" The generated response is converted into audio using Amazon Polly and played back to the user.

[0963] Prompt example

[0964] A specific example of a prompt is as follows:

[0965] User Question: What are some good action movies? Movie Type: Action. Please recommend both new and classic movies.

[0966] In this way, a system can be constructed that allows natural dialogue via characters and provides appropriate information in response to specific user questions.

[0967] As described above, the system of the present invention can provide personalized information recommendations while allowing the user to naturally interact with a character selected by the user, thereby improving the user experience.

[0968] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0969] Step 1:

[0970] The user launches a dedicated app on their smartphone and logs in by entering their account information (username and password). The input data is sent from the device to the server. The server receives this input, checks it against a database to see if it is correct, and returns the results. The server outputs session information if authentication is successful, or an error message if it fails.

[0971] Step 2:

[0972] If login is successful, the terminal displays the main screen to the user. A list of characters available to the user is displayed on the screen, waiting for a selection. To generate the list, the terminal obtains character information from the server and provides it to the user. Character information consists of the character's name, appearance, personality, etc.

[0973] Step 3:

[0974] The user selects the desired character and customizes the character's personality, voice pitch, etc. on the detailed settings screen. This setting data is sent from the device to the server, which saves it along with the user's profile. The server outputs a message confirming that the save was successful.

[0975] Step 4:

[0976] The user begins a dialogue with the selected character. The user speaks, "Tell me some good action movies." The voice input is converted to text by the device's Google Cloud Speech-to-Text engine. The converted text is then sent to the server.

[0977] Step 5:

[0978] The server passes the received text to a natural language processing engine (GPT-4). The server processes the data to generate a prompt, formatting it like this: "User question: What action movies do you recommend? Movie type: Action. Please recommend both new and classic movies." GPT-4 then parses this input and generates an appropriate response. The server's output is the generated text response.

[0979] Step 6:

[0980] The server returns the generated response text to the device. The device converts the response text into speech using Amazon Polly. Here, the input is the text response from GPT-4, and the output is the audio data. The device plays this audio and provides it to the user as "The recommended action movie is 'Action Movie A'."

[0981] Step 7:

[0982] When a user selects a particular movie, the selection is sent from the device to the server, which stores this information as a user history and uses it for future recommendations. The stored information is also used as feedback data to personalize future interactions.

[0983] By implementing each of the above steps in order, users will be able to receive personalized movie recommendations while enjoying natural interactions with characters.

[0984] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0985] This invention is a system that allows users to interact naturally with popular characters. It analyzes user input, generates appropriate responses, and translates them into voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized responses become possible. Below, the system's program and its processing are explained in natural language, with specific examples.

[0986] System Overview

[0987] User authentication / login

[0988] Device: The user launches the dedicated app and logs in by entering their account information (username and password).

[0989] Server: Receives the user's authentication information and checks it against the database to see if it is correct. If authentication is successful, it starts a session and sends the information back to the terminal. The terminal checks the authentication status and displays the main screen if successful, or an error message if unsuccessful.

[0990] Character Selection and Settings

[0991] Terminal: Sends a request to the server to display a list of characters available to the user on the main screen.

[0992] Server: Retrieves available character information from the character database and returns it to the device.

[0993] User: Select the desired character from the list and customize their personality, voice pitch, etc. on the detailed settings screen. The customized settings are sent from the device to the server.

[0994] Server: Stores the received configuration data with the user's profile.

[0995] Interaction Generation and Emotion Recognition

[0996] User: Enter text or voice input to the character. Voice input is converted to text using the device's voice recognition function.

[0997] Terminal: Sends user text input to the server.

[0998] Server: Passes the received text to a natural language processing engine for content analysis. Next, an emotion engine is used to recognize emotions from the user's input. Based on this emotion data, a generative AI generates an appropriate response. The response is then sent back from the server to the device.

[0999] On the device: The returned text response is passed to a speech synthesis engine to generate speech and play it back to the user.

[1000] Specific examples

[1001] Morning greetings and schedule check

[1002] User: Say "Good morning."

[1003] Device: Converts the user's speech into text and sends it to the server.

[1004] Server: Analyzes the input text and uses the emotion engine to recognize that the user is in a refreshing mood. It generates a response such as "Good morning! What a lovely morning. Would you like to check today's schedule?" and sends it to the device.

[1005] Terminal: The returned response is converted into speech using a speech synthesis engine and played back to the user. It says, "Good morning! What a beautiful morning. Would you like to check today's schedule?"

[1006] Adding a task

[1007] User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[1008] Terminal: Sends input text to the server.

[1009] Server: Analyzes the text and sets a reminder for the specified date and time. The emotion engine recognizes that the user is a little impatient and generates a confirmation message saying, "I've set a reminder for your meeting tomorrow at 9am. Is there anything else I can help you with?"

[1010] Device: Converts a confirmation message to speech and plays it to the user. Says, "I've set a reminder for your meeting tomorrow at 9:00. Is there anything else I can help you with?"

[1011] Learning and Personalization

[1012] Terminal: Periodically sends the user's interaction history to the server.

[1013] Server: Stores interaction data and emotion data, scans them, and learns user behavior patterns and emotional changes.

[1014] Server: Based on the learning results, the model is updated to generate personalized responses for the next interaction.

[1015] Terminal: Converts the personalized response into speech and provides it to the user.

[1016] In this way, the present invention generates responses based on user input and emotions, providing a natural interaction experience for the user. By responding to the user's preferences and emotions, the interaction becomes more satisfying and the relationship between the user and the character becomes deeper.

[1017] The processing flow will be explained below.

[1018] Step 1:

[1019] Device: The user launches the app, the app initializes, and the login screen appears.

[1020] Step 2:

[1021] User: Enter your existing account information and click the login button.

[1022] Step 3:

[1023] Terminal: Encodes the entered authentication information and sends it to the server.

[1024] Step 4:

[1025] Server: Receives the authentication information, checks it against a database, and if authentication is successful, starts the session and sends the information back to the device.

[1026] Step 5:

[1027] Terminal: Checks the authentication status, and displays the main screen if successful, or an error message if unsuccessful.

[1028] Step 6:

[1029] Terminal: On the main screen, sends a request to the server to display a list of characters available to the user.

[1030] Step 7:

[1031] Server: Retrieves available character information from the character database and returns it to the device.

[1032] Step 8:

[1033] Terminal: Displays a list of characters the user can choose from.

[1034] Step 9:

[1035] User: Select the desired character from the list and open the detailed settings screen.

[1036] Step 10:

[1037] User: Customize the character's personality, voice pitch, specific expressions, etc., and press the button to save the settings.

[1038] Step 11:

[1039] Device: Sends customized configuration data to the server.

[1040] Step 12:

[1041] Server: Stores the received configuration data with the user's profile.

[1042] Step 13:

[1043] User: Speaks to the character via text or voice.

[1044] Step 14:

[1045] Device: In the case of voice input, the voice recognition function converts the input into text and sends the text to the server.

[1046] Step 15:

[1047] Server: Passes the received text to a natural language processing engine and analyzes its content.

[1048] Step 16:

[1049] Server: Sends the parsed text to the emotion engine to recognize the user's emotion.

[1050] Step 17:

[1051] Server: Using the user's emotional data recognized by the emotion engine, the generative AI generates personalized responses.

[1052] Step 18:

[1053] Server: Sends the generated response to the terminal.

[1054] Step 19:

[1055] Terminal: The returned response sentence is passed to the speech synthesis engine to generate speech.

[1056] Step 20:

[1057] Terminal: Plays the generated audio and provides responses to the user.

[1058] Step 21:

[1059] User: Set a new task, for example, "Set a reminder for a meeting tomorrow at 9 AM."

[1060] Step 22:

[1061] Terminal: Sends a request to the server for task configuration.

[1062] Step 23:

[1063] Server: Analyzes received tasks and adds reminders to the task list at the set date and time.

[1064] Step 24:

[1065] Server: Generates a confirmation message for adding the task and sends it to the device.

[1066] Step 25:

[1067] Terminal: Converts the confirmation message into speech and provides it to the user.

[1068] Step 26:

[1069] Terminal: Periodically transmits the user's interaction history and emotion data to the server.

[1070] Step 27:

[1071] Server: Stores interaction and emotion data and associates it with user profiles.

[1072] Step 28:

[1073] Server: Analyzes the stored data and learns changes in user behavior patterns and emotions.

[1074] Step 29:

[1075] Server: Based on the learning results, generates personalized responses for the next interaction and updates the model.

[1076] Step 30:

[1077] Terminal: Converts the personalized response into speech and provides it to the user.

[1078] Example 2

[1079] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1080] Conventional dialogue systems simply respond to user input, making it difficult to recognize emotions in real time or generate personalized responses. Furthermore, they lack the ability to use the user's interaction history to improve future responses, resulting in low user satisfaction. Furthermore, limited options for customizing character personalities and settings make it difficult to provide a compelling experience for users.

[1081] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1082] In this invention, the server includes means for receiving voice or text input from a user, means for converting the user's voice input into text, natural language processing means for analyzing the user's input and generating an appropriate response, means for recognizing the user's emotions, and means for converting the generated response into voice and providing it to the user, thereby enabling recognition of the user's emotions in real time and generating personalized responses.

[1083] "Means for receiving voice or text input from a user" refers to a device that has an interface through which a user can provide input in the form of voice or text.

[1084] "Means for converting user voice input into text" refers to a device that uses voice recognition technology to convert speech into text form.

[1085] "Natural language processing means for analyzing user input and generating an appropriate response" refers to a device that uses natural language processing technology to analyze user input, understand the context and intent, and generate an appropriate response.

[1086] "Means for recognizing user emotions" refers to a device that uses analytical technology to extract and identify emotions from a user's voice or text.

[1087] The "means for converting the generated response into speech and providing it to the user" refers to a device that uses speech synthesis technology to convert a text response into speech and provide it to the user.

[1088] "Means for storing user profiles and behavioral history" refers to a device equipped with a database that stores users' personal information and past interaction data.

[1089] "Means for analyzing stored profiles and behavioral history to generate responses based on user preferences" refers to a device that uses technology to analyze stored data and generate individually optimized responses based on a user's preferences and past behavioral patterns.

[1090] "Means for storing character information and providing a character based on a user's selection" refers to a device that has a database and an interface for storing information on multiple characters and providing data on a character selected by a user.

[1091] "Means for customizing character personality and voice settings" refers to a device that uses an interface that allows a user to adjust or change a character's characteristics or voice features, and technology that stores those settings.

[1092] The present invention is a system for realizing natural interaction between a user and an interactive character, and is implemented by combining the following hardware and software.

[1093] System Overview

[1094] User authentication / login

[1095] Device:

[1096] The user launches a dedicated app and logs in by entering their username and password. When the user presses the "Login" button, the device encrypts the input information and sends it to the server. The hardware used is a typical computer device such as a smartphone, tablet, or PC. The software uses an encryption library (e.g., OpenSSL).

[1097] server:

[1098] The server compares the received authentication information with a database (e.g., MySQL), and if authentication is successful, it starts a session and returns the information to the terminal. The server software used can be a web server (e.g., Apache, Nginx) or a database server.

[1099] Device:

[1100] The device checks the received session token and displays the main screen if authentication is successful, or an error message if authentication is unsuccessful.The software used is a mobile app development framework (e.g., React Native, Flutter).

[1101] Character Selection and Settings

[1102] Device:

[1103] Send a request to the server to display the list of characters available to the user on the main screen, using an HTTP request.

[1104] server:

[1105] The server retrieves available character information from a character database (e.g., MongoDB) and returns it to the device. The returned data is in JSON format.

[1106] Device:

[1107] The device displays the received character information on the main screen. The user selects a character on the screen and customizes their personality, voice pitch, etc. on the detailed settings screen. The customized settings are then sent back to the server in JSON format.

[1108] server:

[1109] The server stores the received configuration data in a database together with the user's profile.

[1110] Interaction Generation and Emotion Recognition

[1111] User:

[1112] The user can input text or voice to the character. Voice input is converted to text using the device's voice recognition function (e.g., Google Speech-to-Text).

[1113] Device:

[1114] The device sends the user's text input to the server, using HTTP or WebSockets.

[1115] server:

[1116] The server passes the received text to a natural language processing engine (e.g., spaCy, Hugging Face Transformers) for content analysis. The server also uses an emotion recognition engine (e.g., a BERT-based model) to recognize the user's emotions. Based on this emotion data, the server generates an appropriate response using a generative AI model (e.g., OpenAI GPT-4) and sends it back to the device.

[1117] Device:

[1118] The device passes the returned text response to a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio and play it back to the user.

[1119] Specific examples

[1120] Morning greetings and schedule check

[1121] User:

[1122] Say "Good morning."

[1123] Device:

[1124] The user's speech is converted into text and sent to the server.

[1125] server:

[1126] The input text is analyzed using a natural language processing engine, and the emotion engine is used to recognize that the user is in a refreshing mood. The response "Good morning! What a lovely morning. Would you like to check today's schedule?" is generated and sent to the device.

[1127] Device:

[1128] The returned response is converted into speech by a speech synthesis engine and played back to the user.

[1129] Example prompts to input to a generative AI model:

[1130] "A user says good morning. He seems very refreshed. How would you respond?"

[1131] Adding a task

[1132] User:

[1133] Type "Set a meeting reminder for tomorrow at 9 AM."

[1134] Device:

[1135] Sends the input text to the server.

[1136] server:

[1137] It analyzes the text and sets a reminder for the specified date and time. The emotion engine recognizes that the user is a little impatient and generates a confirmation message saying, "I've set a reminder for your meeting tomorrow at 9:00. Is there anything else I can help you with?"

[1138] Device:

[1139] A confirmation message is converted into audio and played to the user.

[1140] Example prompts to input to a generative AI model:

[1141] "A user types that they want to set a reminder for a meeting tomorrow at 9 AM. They seem a little impatient. How would you respond?"

[1142] Learning and Personalization

[1143] Device:

[1144] The user's interaction history is periodically sent to the server.

[1145] server:

[1146] It stores interaction and emotion data, analyzes them, and learns user behavior patterns and emotional changes. Based on the learning results, it updates the model to generate personalized responses for the next interaction.

[1147] Device:

[1148] The personalized response is converted into speech and provided to the user.

[1149] In this way, the present invention generates responses based on user input and emotions, providing a natural interaction experience for the user. By responding to the user's preferences and emotions, the interaction becomes more satisfying and the relationship between the user and the character becomes deeper.

[1150] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1151] An explanation of the processing flow of this system's program, broken down into processing steps

[1152] Step 1:

[1153] User Login

[1154] Input: The user launches the dedicated app and enters their username and password.

[1155] Specific operation: When the user presses the "Login" button, the terminal encrypts the username and password and sends them to the server.

[1156] Output: Encrypted user credentials are sent to the server.

[1157] Step 2:

[1158] Authentication verification

[1159] Input: Encrypted user credentials

[1160] What happens: The server checks the received credentials against its database. It performs a database query to verify that the username and password match.

[1161] Output: Generates a result indicating whether authentication was successful or not, and sends it back to the terminal along with the session information.

[1162] Step 3:

[1163] Viewing the login results

[1164] Input: Authentication result and session information

[1165] Specific operation: The device checks the received authentication result, and displays the main screen if successful, or an error message if unsuccessful.

[1166] Output: Display of the main screen or error message.

[1167] Step 4:

[1168] Character Information Request

[1169] Input: Request required to display the main screen

[1170] Specific operation: The device sends a request to the server to obtain available character information.

[1171] Output: Sends a request to the server to get character data.

[1172] Step 5:

[1173] Acquiring and returning character information

[1174] Input: Character data acquisition request

[1175] Specific operation: The server retrieves character information from the character database and returns it to the device in JSON format.

[1176] Output: Returns character information to the terminal.

[1177] Step 6:

[1178] Displaying the character list

[1179] Input: Character information in JSON format

[1180] Specific operation: The device analyzes character information and displays it in a list on the main screen.

[1181] Output: Display the character list on the main screen.

[1182] Step 7:

[1183] Character selection and detailed settings

[1184] Input: User-selected character and customization settings

[1185] Specific operation: The user selects a character and customizes the character's personality, voice pitch, etc. in the detailed settings screen. These settings are sent from the device to the server.

[1186] Output: Sends customization information to the server.

[1187] Step 8:

[1188] Saving the configuration information

[1189] Input: Customization setting information

[1190] Specific operation: The server stores the received customization setting information in the database.

[1191] Output: Save customized settings information to a database.

[1192] Step 9:

[1193] User Input and Conversion

[1194] Input: User voice or text input

[1195] Specific operation: The user inputs to the character by voice or text. In the case of voice input, the device converts the voice into text in real time.

[1196] Output: The converted text data is sent to the server.

[1197] Step 10:

[1198] Natural Language Processing and Emotion Recognition

[1199] Input: Text data

[1200] Specific operation: The server passes the text data to a natural language processing engine for analysis, and recognizes emotions using an emotion engine.

[1201] Output: Analysis results and emotion data.

[1202] Step 11:

[1203] Generate a response

[1204] Input: Analysis results and emotion data

[1205] Specific operation: The server uses a generative AI model (e.g., OpenAI GPT-4) to generate an appropriate response.

[1206] Output: Sends the generated response to the terminal.

[1207] Step 12:

[1208] Audio generation and playback

[1209] Input: Generated response

[1210] Specific operation: The device passes the response sentence to a speech synthesis engine to generate speech and play it back to the user.

[1211] Output: Playback of the generated audio.

[1212] Example prompt

[1213] 1. Morning greeting and schedule prompt:

[1214] "A user says good morning. He seems very refreshed. How would you respond?"

[1215] 2. Prompt for adding a task:

[1216] "A user types that they want to set a reminder for a meeting tomorrow at 9 AM. They seem a little impatient. How would you respond?"

[1217] (Application example 2)

[1218] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1219] Conventional interaction systems have fixed responses to user inputs and have not been able to realize natural dialogue that takes into account the user's emotions and gaze. Furthermore, in the work environment, there is a lack of efficient and safe work support, which has led to the issue of workers being unable to quickly obtain the information they need.

[1220] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into voice and providing it to the user, means for recognizing the user's emotions, and means for tracking the user's gaze. This enables natural dialogue that takes the user's emotions and gaze into consideration. The server also includes means for saving a user's profile and behavioral history, means for analyzing the saved profile and behavioral history and generating a response based on the user's preferences, means for visually providing the generated response, and means for displaying information via a visual output device worn by the user. This enables workers to efficiently and safely obtain necessary information in the work environment and support them. The server also includes means for saving character information and providing a character based on the user's selection, means for customizing the character's personality and voice settings, and means for creating appropriate prompt sentences based on a generative AI model. This allows for more personalized responses to the user and an improved dialogue experience.

[1221] "Means for receiving input from a user" refers to hardware and software for capturing and transmitting information entered by a user through speech or text to the system.

[1222] "Means for analyzing user input and generating an appropriate response" refers to a technology that analyzes received user input information using a natural language processing engine or the like and generates a response based on that information.

[1223] The "means for converting the generated response into speech and providing it to the user" is a technology that uses a speech synthesis engine to convert the generated text response into speech and plays it aloud to the user through a speaker or the like.

[1224] "Means for recognizing user emotions" refers to technology that analyzes emotions from the user's voice, text, facial expressions, etc., and determines the user's current feelings and state.

[1225] "Means for tracking the user's gaze" refers to technology that uses a camera or sensor to track the direction or object the user is looking at and records it as gaze information.

[1226] "Means for storing a user's profile and behavior history" refers to technology for storing a user's personal information, past operation history, behavior patterns, etc. in a storage device.

[1227] "Means for analyzing stored profiles and behavioral history and generating responses based on the user's preferences" refers to technology that analyzes stored data and generates personalized responses tailored to the user's tendencies and preferences.

[1228] The "means for visually presenting the generated response" is a technique for displaying the generated response on a display device as visual information such as text, images, or videos.

[1229] "Means for displaying information via a visual output device worn by a user" refers to technology for displaying information through a device worn by a user, such as smart glasses or a head-mounted display.

[1230] "Means for saving character information and providing characters based on user selection" refers to technology that saves character attributes and settings in a database and provides the character on screen or via audio based on the user's selection.

[1231] "Means for customizing character personality and voice settings" refers to technology that allows a user to change the personality, voice pitch, tone, etc. of a character selected by the user to suit the user's preferences.

[1232] "Means for creating appropriate prompt sentences based on a generative AI model" refers to a technology that uses an AI model to generate optimal response sentences based on input from a user.

[1233] The present invention provides a system that supports factory work by enabling natural interactions with input from users. This system allows characters to interact with the user through smart glasses worn by the user, improving work efficiency and safety. Detailed embodiments for implementing the present invention are described below.

[1234] System Overview

[1235] The system consists of the following main components:

[1236] 1. A means of receiving input from the user

[1237] Hardware: Smart glasses (e.g., Google Glass)

[1238] Process: Capture the user's voice and gaze data and send it to the system.

[1239] 2. A means of analyzing user input and generating an appropriate response

[1240] Software: Speech recognition software (e.g., Google Cloud Speech-to-Text), natural language processing engines (e.g., OpenAI's GPT-4)

[1241] Process: Convert the captured audio into text, analyze the text and generate an appropriate response.

[1242] 3. A means of converting the generated response into speech and providing it to the user

[1243] Software: Speech synthesis engine (e.g. Amazon Polly)

[1244] Process: The generated text is converted into speech and provided to the user through the smart glasses speaker.

[1245] 4. How to Recognize User Emotions

[1246] Software: Emotion recognition engine (e.g., Microsoft Azure Emotion API)

[1247] Process: Analyze the user's emotions from captured voice and gaze data and provide feedback to the system.

[1248] 5. A way to track the user's gaze

[1249] Hardware: Built-in camera in smart glasses

[1250] Process: Track the user's gaze in real time and analyze the gaze data.

[1251] 6. Means of storing user profiles and activity histories

[1252] Hardware: Cloud-based database

[1253] Process: Stores user personal information, past operation history, behavioral patterns, etc.

[1254] 7. A means of analyzing stored profiles and behavioral history to generate responses based on user preferences.

[1255] Software: Data analysis engine

[1256] Process: Analyze the stored data and generate a personalized response based on the user's preferences.

[1257] 8. A visual means of providing the generated response

[1258] Hardware: Smart glasses display

[1259] Process: The generated response is visually displayed and presented to the user.

[1260] 9. Means for storing character information and providing characters based on user selection

[1261] Hardware: Cloud-based database

[1262] Process: Stores character attributes and settings and provides characters based on user selections.

[1263] 10. A way to customize character personalities and voice settings

[1264] Software: Character Setting Engine

[1265] Process: Modify the personality, voice pitch, tone, etc. of the user-selected character to suit the user's preferences.

[1266] 11. A means of creating appropriate prompts based on generative AI models

[1267] Software: Generative AI models (e.g., OpenAI's GPT-4)

[1268] Process: Use an AI model to generate optimal responses based on user input.

[1269] Specific examples

[1270] Check the checklist before starting work

[1271] 1. User: Talk to the character and say, "Check today's work."

[1272] 2. Smart glasses: Capture the user's voice and send it to voice recognition software.

[1273] 3. Server: Converts the input speech into text and passes it to the natural language processing engine.

[1274] 4. Natural language processing engine: Analyzes the text and generates a corresponding checklist.

[1275] 5. Emotion Recognition Engine: Recognizes when the user is focused.

[1276] 6. Generative AI: Generates a response such as, "Please be mindful of safety as you work today. Your first task is maintenance on Machine A. Would you like to review the checklist?"

[1277] 7. Speech synthesis engine: Converts the response into speech and provides it to the user through the smart glasses' speaker.

[1278] 8. Smart glasses: Visually display the generated text on a display.

[1279] Prompt Sentence Examples

[1280] Character Name: Factory character "Mechatrobo"

[1281] Input sentence: Tell the character "Tell me how to operate machine B"

[1282] Emotions: A little anxious

[1283] Response: Mecha-Robo: "I'll teach you how to operate Machine B. First, pull down the lever on the left. Next, press the button in the middle. If you have any trouble, please stay calm and let's go through each step one by one."

[1284] In this way, the system of the present invention improves work efficiency and safety by analyzing the user's voice and emotions and providing visual and audio responses based on those analyses. By personalizing responses based on the user's profile and behavioral history, the system provides higher satisfaction and enhances its utility in the work environment.

[1285] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1286] Step 1:

[1287] User: The user puts on the smart glasses and says, "See what I did today."

[1288] Input: User's voice

[1289] Output: Capture of audio data

[1290] How it works: The smart glasses capture the user's voice with a built-in microphone.

[1291] Step 2:

[1292] Device: The smart glasses capture and transmit the audio to the voice recognition software.

[1293] Input: Audio data

[1294] Output: Text data

[1295] How it works: Uses a speech recognition engine (e.g. Google Cloud Speech-to-Text) to convert input voice data into text.

[1296] Step 3:

[1297] Terminal: Sends text data to the server.

[1298] Input: Text data

[1299] Output: Sends text data to the server

[1300] How it works: Smart glasses send text data over the network to a server.

[1301] Step 4:

[1302] Server: The server analyzes the received text data.

[1303] Input: Text data

[1304] Output: Analysis results

[1305] How it works: It uses a natural language processing engine (e.g., OpenAI's GPT-4) to analyze text data and understand user intent.

[1306] Step 5:

[1307] Server: The server recognizes the user's emotions using an emotion recognition engine.

[1308] Input: Text data

[1309] Output: Emotion data

[1310] How it works: It uses an emotion recognition engine (e.g. Microsoft Azure's Emotion API) to analyze the user's emotions from the input text data.

[1311] Step 6:

[1312] Server: The server generates an appropriate response using a generative AI model based on the analysis results and emotion data.

[1313] Input: Text data, emotion data

[1314] Output: Response text

[1315] How it works: Uses a generative AI model (e.g., OpenAI's GPT-4) to generate a response text based on the analysis results and sentiment data.

[1316] Examples:

[1317] Character name: Factory character "Work Robot"

[1318] Input: "Check today's work"

[1319] Emotion: Focused

[1320] Response: Worker robot: "Please be mindful of safety as you work today. The first task is maintenance on Machine A. Would you like to review the checklist?"

[1321] "

[1322] Step 7:

[1323] Server: Pass the generated response text to the speech synthesis engine.

[1324] Input: Response text

[1325] Output: Audio data

[1326] What it does: Converts response text into speech using a text-to-speech engine (e.g., Amazon Polly).

[1327] Step 8:

[1328] Terminal: Sends audio data to smart glasses.

[1329] Input: Audio data

[1330] Output: Receives audio data

[1331] How it works: Audio data is sent over the network to smart glasses.

[1332] Step 9:

[1333] Terminal: The smart glasses play the audio data and provide it to the user.

[1334] Input: Audio data

[1335] Output: Play audio

[1336] What it does: Plays audio using the smart glasses' speakers and provides it to the user.

[1337] Step 10:

[1338] Terminal: Visually displays the generated response text.

[1339] Input: Response text

[1340] Output: Visual information

[1341] What it does: Visually display the response text on the smart glasses display to provide information to the user.

[1342] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1343] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1344] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1345] [Third embodiment]

[1346] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1347] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1348] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1349] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1350] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1351] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1352] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1353] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1354] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1355] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1356] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1357] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1358] This invention is a system that allows users to interact naturally with popular characters by analyzing input from the user and generating appropriate responses and translating them into voice. Below, the system program and its processing are explained in natural language, with specific examples.

[1359] System Overview

[1360] User authentication / login

[1361] Device: The user launches the dedicated app and logs in by entering their account information (username and password).

[1362] Server: Receives the user's authentication information and checks it against the database to see if it is correct. If authentication is successful, it starts a session and sends the information back to the terminal. The terminal checks the authentication status and displays the main screen if successful, or an error message if unsuccessful.

[1363] Character Selection and Settings

[1364] Terminal: The main screen displays a list of characters available to the user. The list is retrieved from the server.

[1365] User: Select the desired character and customize their personality, voice pitch, etc. on the detailed settings screen. The customized settings are sent from the device to the server.

[1366] Server: Stores the received configuration data with the user's profile.

[1367] Interaction Generation

[1368] User: Provides text or voice input to the character. Voice input is converted to text using the device's voice recognition function.

[1369] Terminal: Sends user text input to the server.

[1370] Server: Passes the received text to a natural language processing engine to analyze the input. Based on the analysis results, a generative AI generates an appropriate response. The response is then sent back from the server to the device.

[1371] On the device: The returned text response is passed to a speech synthesis engine to generate speech and play it back to the user.

[1372] Specific examples

[1373] Morning greetings and schedule check

[1374] User: Say "Good morning!"

[1375] Device: Converts the user's speech into text and sends it to the server.

[1376] Server: Analyzes the input text and generates a response such as "Good morning! Would you like to check today's schedule?". Then, sends the generated response back to the device.

[1377] Terminal: The returned response is converted into speech by a speech synthesis engine and played back to the user. It says, "Good morning! Would you like to check today's schedule?"

[1378] Adding a task

[1379] User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[1380] Terminal: Sends input text to the server.

[1381] Server: Parse the text and set a reminder for the specified date and time. Generate a confirmation message saying "Your reminder for the meeting has been set for tomorrow at 9 AM."

[1382] Device: Converts a confirmation message into speech and provides it to the user. Says, "I've set a reminder for a meeting tomorrow at 9 AM."

[1383] Learning and Personalization

[1384] Terminal: Sends the user's usage history to the server.

[1385] Server: Stores interaction data and analyzes it to learn user behavior patterns and preferences.

[1386] Server: Generates more personalized responses to users based on the learning results.

[1387] The device converts this personalized response into speech and provides it to the user.

[1388] In this way, the system allows users to naturally interact with the character they select, providing various support such as daily tasks and setting reminders. The interaction is personalized based on the user's preferences and behavioral patterns, providing an experience that makes it seem as if the user and the character are living together.

[1389] The processing flow will be explained below.

[1390] Step 1:

[1391] Device: The user launches the app, the app initializes, and the login screen appears.

[1392] Step 2:

[1393] User: Enter your existing account information and click the login button.

[1394] Step 3:

[1395] Terminal: Encodes the entered authentication information and sends it to the server.

[1396] Step 4:

[1397] Server: Receives the authentication information, checks it against a database, and if authentication is successful, starts the session and sends the information back to the device.

[1398] Step 5:

[1399] Terminal: Checks the authentication status, and displays the main screen if successful, or an error message if unsuccessful.

[1400] Step 6:

[1401] Terminal: On the main screen, sends a request to the server to display a list of characters available to the user.

[1402] Step 7:

[1403] Server: Retrieves available character information from the character database and returns it to the device.

[1404] Step 8:

[1405] Terminal: Displays a list of characters the user can choose from.

[1406] Step 9:

[1407] User: Select the desired character from the list and open the detailed settings screen.

[1408] Step 10:

[1409] User: Customize the character's personality, voice pitch, specific expressions, etc., and press the button to save the settings.

[1410] Step 11:

[1411] Device: Sends customized configuration data to the server.

[1412] Step 12:

[1413] Server: Stores the received configuration data with the user's profile.

[1414] Step 13:

[1415] User: Speaks to the character via text or voice.

[1416] Step 14:

[1417] Device: In the case of voice input, the voice recognition function converts the input into text and sends the text to the server.

[1418] Step 15:

[1419] Server: Passes the received text to a natural language processing engine and analyzes its content.

[1420] Step 16:

[1421] Server: Based on the results of natural language processing, a generative AI generates an appropriate response sentence.

[1422] Step 17:

[1423] Server: Sends the generated response to the terminal.

[1424] Step 18:

[1425] Terminal: The returned response sentence is passed to the speech synthesis engine to generate speech.

[1426] Step 19:

[1427] Terminal: Plays the generated audio and provides responses to the user.

[1428] Step 20:

[1429] User: Set a new task, for example, "Set a reminder for a meeting tomorrow at 9 AM."

[1430] Step 21:

[1431] Terminal: Sends a request to the server for task configuration.

[1432] Step 22:

[1433] Server: Analyzes received tasks and adds reminders to the task list at the set date and time.

[1434] Step 23:

[1435] Server: Generates a confirmation message for adding the task and sends it to the device.

[1436] Step 24:

[1437] Terminal: Converts the confirmation message into speech and provides it to the user.

[1438] Step 25:

[1439] Terminal: Periodically sends the user's interaction history to the server.

[1440] Step 26:

[1441] Server: Stores interaction data and associates it with user profiles.

[1442] Step 27:

[1443] Server: Analyzes the stored data and learns user behavior patterns and preferences.

[1444] Step 28:

[1445] Server: Generates more personalized responses based on the learning results.

[1446] Step 29:

[1447] Terminal: Converts the personalized response into speech and provides it to the user.

[1448] Example 1

[1449] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1450] Conventional dialogue systems have struggled to achieve natural interactions with users. In particular, they have had challenges in accurately analyzing user input and providing appropriate responses in a natural manner. Furthermore, technology for providing personalized responses based on user preferences and behavioral patterns has also been inadequate. This has led to problems such as a poor user experience and a decline in system utilization.

[1451] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1452] In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for using a speech recognition engine to convert the user's speech into text, means for analyzing the input using a natural language processing engine, means for generating a response using a generative artificial intelligence model, and means for converting the response into speech using a speech synthesis engine. This enables the user's input to be accurately analyzed and a response to be provided in a natural manner. Furthermore, by storing the user's profile and behavioral history and providing personalized responses based on them, the quality of the user experience can be improved and the system utilization rate can be increased.

[1453] "Means for receiving input from a user" refers to the facility by which a user provides input to the system in text or speech form.

[1454] The "means for analyzing user input and generating an appropriate response" is a function for analyzing input data received from a user and generating an appropriate response based on the content of the input data.

[1455] The "means for converting the generated response into speech and providing it to the user" is a function for converting the generated response in text format into speech format and playing it back to the user.

[1456] A "speech recognition engine that converts user speech into text" is a software component that converts user spoken speech into text format.

[1457] A "natural language processing engine" is a software component that analyzes text input from a user and understands its meaning.

[1458] A "generative artificial intelligence model" is a machine learning model for generating responses based on analyzed text.

[1459] A "speech synthesis engine" is a software component for converting generated text responses into speech form.

[1460] "Means for storing user profiles and behavioral history" refers to a function for storing users' personal information and past interaction data.

[1461] "Means for analyzing stored profiles and behavioral history and generating responses based on user preferences" refers to a function that analyzes stored data to understand the user's preferences and behavioral patterns and generates responses based on them.

[1462] The "means for learning user behavior patterns and preferences based on interaction data" is a function that utilizes the dialogue history with the user to learn the user's behavior patterns and preferences.

[1463] The "means for saving character information and providing a character based on the user's selection" is a function for saving information about each character and providing that character based on the user's selection.

[1464] "Means for customizing character personality and voice settings" is a function that allows the user to adjust settings related to the character's personality and voice.

[1465] The "means for saving customized settings" is a function for saving changes to character settings made by the user.

[1466] The present invention is a system for interacting with a user that utilizes advanced technology to achieve natural interaction by analyzing user input, generating appropriate responses, and providing them to the user in spoken form.

[1467] System Overview

[1468] Hardware and software used

[1469] This system uses the following hardware and software:

[1470] Speech recognition engine: Use a common speech recognition technology. Here, we will use the widely used Google Speech-to-Text as an example.

[1471] Natural Language Processing Engine: An engine for analyzing the meaning of text data, using advanced natural language processing techniques such as OpenAI GPT-4.

[1472] Generative AI model: A machine learning model that generates appropriate responses based on user input.

[1473] Speech synthesis engine: An engine for converting the generated text response into speech, specifically Amazon Polly.

[1474] User-System Interaction

[1475] User authentication / login

[1476] The user launches the dedicated app on their device and enters their account information (username and password) on the login screen. The device sends this information to the server, which then collates it with a database for authentication. If authentication is successful, the server creates a session and returns that information to the device. The device checks the authentication status based on the received session information and displays the main screen.

[1477] Character Selection and Settings

[1478] When the user taps the "Select Character" button on the main screen, the device sends a request for a character list to the server. The server retrieves character information from the database and sends it back to the device. The device displays the received character list on the screen, and the user selects the desired character. The user can further customize the character's personality and voice pitch on the detailed settings screen, and these settings are sent to the server. The server saves the received settings in its database.

[1479] Interaction Generation

[1480] Users can provide text or voice input to the character. For voice input, the device captures the audio and converts it into text using Google Speech-to-Text. This text is sent to a server, which passes it to a natural language processing engine (GPT-4) for analysis. Based on the analysis results, a generative AI model generates an appropriate response, which is then sent back from the server to the device. The device then converts the received response into speech using a speech synthesis engine (Amazon Polly) and plays it back to the user.

[1481] Specific examples

[1482] Morning greetings and schedule check

[1483] When a user says "Good morning!", the device converts the speech to text and sends it to the server. The server generates a response, "Good morning! Would you like to check today's schedule?" and sends it back to the device. The device converts the response to speech and plays it to the user: "Good morning! Would you like to check today's schedule?"

[1484] Adding a task

[1485] When a user types "Set a reminder for a meeting tomorrow at 9 AM," the device sends the text to the server. The server parses the instruction and sets a reminder for the specified date and time. It generates a confirmation message saying "A reminder for a meeting tomorrow at 9 AM" and sends it back to the device. The device converts it into speech and provides it to the user: "A reminder for a meeting tomorrow at 9 AM."

[1486] Learning and Personalization

[1487] Every time a user uses the system, their usage history is sent to the server and stored in a database. The server analyzes this data and learns the user's behavioral patterns and preferences. Based on this learning, the system can generate more personalized responses. For example, if a user frequently adds a specific task, the system will provide information and suggestions related to that task.

[1488] In this way, the system of the present invention uses advanced speech recognition, natural language processing, and generative artificial intelligence models to achieve natural interactions with users and support daily task management and information provision. Examples of prompt sentences include "Good morning!", "Set a reminder for a meeting tomorrow at 9:00 AM," and "What's the weather like today?"

[1489] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1490] Step 1:

[1491] Device: The user launches the dedicated app and enters their account information (username and password) on the login screen. This input data is sent to the server for login authentication.

[1492] Input: Username, Password

[1493] Output: Login request sent to the server

[1494] Step 2:

[1495] Server: Receives the user's account information and checks it against a database. If authentication is successful, the server creates a session and sends the information back to the device. If authentication fails, it sends back an error message.

[1496] Input: Username, Password

[1497] Data processing: database matching, session generation

[1498] Output: Session information or error message

[1499] Step 3:

[1500] Terminal: Checks the authentication status based on the session information received from the server and displays the main screen. If an error message is returned, the error message is displayed.

[1501] Input: Session information or error message

[1502] Output: Main screen or error message display

[1503] Step 4:

[1504] User: Tap the "Character Selection" button on the main screen.

[1505] Input: Tap

[1506] Output: Character selection request

[1507] Step 5:

[1508] Terminal: Sends a character selection request to the server.

[1509] Input: Character selection request

[1510] Output: Character selection request sent to the server

[1511] Step 6:

[1512] Server: Retrieves character information from the database and sends it back to the device.

[1513] Input: Character selection request

[1514] Data processing: Database matching, character information acquisition

[1515] Output: Retrieved character information

[1516] Step 7:

[1517] Terminal: Display the received character information on the screen.

[1518] Input: Character information

[1519] Output: Display the character list

[1520] Step 8:

[1521] User: Select the desired character from the list and customize their personality and voice pitch in the advanced settings screen.

[1522] Input: Character selection, customization settings

[1523] Output: Customization settings information

[1524] Step 9:

[1525] Device: Sends customized configuration information to the server.

[1526] Input: Customization setting information

[1527] Output: Customization settings sent to the server

[1528] Step 10:

[1529] Server: Saves customizations to a database.

[1530] Input: Customization setting information

[1531] Data processing: Database storage

[1532] Output: Save complete message

[1533] Step 11:

[1534] User: Provides text or voice input to the character.

[1535] Input: Text or voice input

[1536] Output: Input data

[1537] Step 12:

[1538] On the device: For voice input, the speech is converted to text using Google Speech-to-Text, and the converted text is sent to the server.

[1539] Input: Voice input

[1540] Data processing: speech recognition, text conversion

[1541] Output: Text data

[1542] Step 13:

[1543] Terminal: Sends text input or converted text to the server.

[1544] Input: Text data

[1545] Output: Text data sent to the server

[1546] Step 14:

[1547] Server: Passes the received text to a natural language processing engine (GPT-4) and analyzes its content.

[1548] Input: Text data

[1549] Data Processing: Natural Language Processing

[1550] Output: Analysis results

[1551] Step 15:

[1552] Server: Based on the analysis results, the generative AI model generates an appropriate response, which is then sent back to the device.

[1553] Input: Analysis results

[1554] Data processing: response generation

[1555] Output: Response data

[1556] Step 16:

[1557] Terminal: Passes the received response data to the speech synthesis engine (Amazon Polly) and converts it into speech.

[1558] Input: Response data

[1559] Data processing: voice synthesis

[1560] Output: Audio data

[1561] Step 17:

[1562] Terminal: Plays the generated audio to the user.

[1563] Input: Audio data

[1564] Output: Audio playback

[1565] Specific operation example

[1566] Examples of morning greetings and schedule confirmation

[1567] 1. User: Says "Good morning!"

[1568] 2. On the device: Capture the user's voice and convert it to text using Google Speech-to-Text.

[1569] 3. Terminal: Send the converted text "Good morning!" to the server.

[1570] 4. Server: Passes the received text "Good morning!" to the natural language processing engine (GPT-4) and instructs it to generate a response.

[1571] 5. Server: Generates a response saying "Good morning! Would you like to check today's schedule?" and sends it back to the device.

[1572] 6. Terminal: The returned response is passed to the speech synthesis engine (Amazon Polly) and converted into speech.

[1573] 7. Terminal: Play the audio to the user: "Good morning! Would you like to check today's schedule?"

[1574] Example of adding a task

[1575] 1. User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[1576] 2. Terminal: Sends the entered text to the server.

[1577] 3. Server: Passes the received text to the natural language processing engine (GPT-4) for analysis.

[1578] 4. Server: Registers the instruction "Set a meeting reminder for tomorrow at 9:00" in the scheduler.

[1579] 5. Server: Generates a confirmation message saying "Your meeting reminder has been set for tomorrow at 9 AM" and sends it back to the device.

[1580] 6. Device: Passes the confirmation message to the speech synthesis engine (Amazon Polly) and converts it into speech.

[1581] 7. Device: The generated speech "A meeting reminder has been set for tomorrow at 9:00" is provided to the user.

[1582] Examples of prompt statements

[1583] "good morning!"

[1584] "Set a meeting reminder for tomorrow at 9 AM"

[1585] "How's the weather today?"

[1586] (Application example 1)

[1587] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1588] Conventional interactive systems have difficulty in providing natural dialogue between users and characters, or in providing information based on the user's preferences. Furthermore, they lack the ability to instantly recommend appropriate information in response to user questions, limiting the user experience. This makes it difficult for users to fully enjoy the dialogue with characters and obtain personalized responses.

[1589] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1590] In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for converting speech input into text, means for analyzing the text input and acquiring and recommending information, and means for saving and reusing information based on user selection, thereby enabling the user to enjoy natural dialogue with a character and receive personalized information recommendations through the interaction.

[1591] A "means for receiving input from a user" is a device or interface that receives input when a user asks a question or gives a command by text or voice.

[1592] The "means for analyzing user input and generating an appropriate response" refers to a device or algorithm that analyzes user input and generates an optimal response for the user based on the content of the input.

[1593] The "means for converting the generated response into speech and providing it to the user" refers to a device or software for converting the generated text response into speech and playing the speech to the user.

[1594] A "voice-to-text converter" is a device or software that converts a user's speech into text in a machine-parseable format.

[1595] The "means for analyzing text input and retrieving and recommending information" refers to a device or algorithm that analyzes the content input as text, retrieves appropriate information, and recommends it to the user.

[1596] The "means for saving and reusing information based on user selection" refers to a device or software that saves information selected or set by the user and reuses it in a later interaction.

[1597] "Means for storing user profiles and behavioral histories" refers to devices or software that store users' personal information and behavioral histories in a database.

[1598] "Means for analyzing stored profile and behavioral history and generating responses based on user preferences" refers to devices or algorithms that analyze stored data and generate responses tailored to the user's preferences.

[1599] The "means for storing character information and providing a character based on a user's selection" refers to a device or software that stores information about a character and provides the character selected by the user on the system.

[1600] "Means for customizing the character's personality and voice settings" refers to devices or software that allow the user to customize the character's personality, voice pitch, etc.

[1601] The "means for personalizing information provided by a character based on user preferences" refers to a device or algorithm that customizes information provided by a character based on user preferences.

[1602] System Overview

[1603] The present invention is a system that includes a set of functions such as means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for converting speech input into text, means for analyzing the text input and retrieving and recommending information, and means for storing and reusing information based on user selection.

[1604] Hardware and software used

[1605] Hardware: Smartphone

[1606] software:

[1607] Speech recognition engine: Google Cloud Speech-to-Text

[1608] Natural Language Processing Engine: OpenAI GPT-4

[1609] Speech synthesis engine: Amazon Polly

[1610] Database: Firebase

[1611] Interface: Flutter

[1612] Operating procedure

[1613] User authentication / login

[1614] The user launches a dedicated app on their smartphone and logs in by entering their account information (username and password). The server receives the user's authentication information and verifies it against a database to ensure it is correct. If authentication is successful, the server starts a session and returns the information to the device. The device then checks the authentication status, and displays the main screen if successful, or an error message if login is unsuccessful.

[1615] Character Selection and Settings

[1616] The user can view a list of available characters on the main screen and select the desired character. It is also possible to customize the character's personality, voice pitch, etc. The customized settings are sent from the device to the server, which then stores the received settings data along with the user's profile.

[1617] Interaction Generation

[1618] The user provides text or voice input to the character. For voice input, the device converts the speech to text using Google Cloud Speech-to-Text. The converted text input is sent to the server, which passes it to a natural language processing engine (GPT-4) for analysis. Based on the analysis results, the generative AI (GPT-4) generates an appropriate response, which is sent back from the server to the device. The device then converts the returned text response into speech using Amazon Polly and plays it back to the user.

[1619] Specific examples

[1620] Movie Recommendations

[1621] A user speaks to their smartphone, saying, "Tell me some recommended action movies." The device converts the speech into text using Google Cloud Speech-to-Text and sends it to the server. The server analyzes the text, obtains the appropriate movie information, and generates a response. For example, GPT-4 generates a response such as, "The recommended action movie is 'Action Movie A.'" The generated response is converted into audio using Amazon Polly and played back to the user.

[1622] Prompt example

[1623] A specific example of a prompt is as follows:

[1624] User Question: What are some good action movies? Movie Type: Action. Please recommend both new and classic movies.

[1625] In this way, a system can be constructed that allows natural dialogue via characters and provides appropriate information in response to specific user questions.

[1626] As described above, the system of the present invention can provide personalized information recommendations while allowing the user to naturally interact with a character selected by the user, thereby improving the user experience.

[1627] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1628] Step 1:

[1629] The user launches a dedicated app on their smartphone and logs in by entering their account information (username and password). The input data is sent from the device to the server. The server receives this input, checks it against a database to see if it is correct, and returns the results. The server outputs session information if authentication is successful, or an error message if it fails.

[1630] Step 2:

[1631] If login is successful, the terminal displays the main screen to the user. A list of characters available to the user is displayed on the screen, waiting for a selection. To generate the list, the terminal obtains character information from the server and provides it to the user. Character information consists of the character's name, appearance, personality, etc.

[1632] Step 3:

[1633] The user selects the desired character and customizes the character's personality, voice pitch, etc. on the detailed settings screen. This setting data is sent from the device to the server, which saves it along with the user's profile. The server outputs a message confirming that the save was successful.

[1634] Step 4:

[1635] The user begins a dialogue with the selected character. The user speaks, "Tell me some good action movies." The voice input is converted to text by the device's Google Cloud Speech-to-Text engine. The converted text is then sent to the server.

[1636] Step 5:

[1637] The server passes the received text to a natural language processing engine (GPT-4). The server processes the data to generate a prompt, formatting it like this: "User question: What action movies do you recommend? Movie type: Action. Please recommend both new and classic movies." GPT-4 then parses this input and generates an appropriate response. The server's output is the generated text response.

[1638] Step 6:

[1639] The server returns the generated response text to the device. The device converts the response text into speech using Amazon Polly. Here, the input is the text response from GPT-4, and the output is the audio data. The device plays this audio and provides it to the user as "The recommended action movie is 'Action Movie A'."

[1640] Step 7:

[1641] When a user selects a particular movie, the selection is sent from the device to the server, which stores this information as a user history and uses it for future recommendations. The stored information is also used as feedback data to personalize future interactions.

[1642] By implementing each of the above steps in order, users will be able to receive personalized movie recommendations while enjoying natural interactions with characters.

[1643] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1644] This invention is a system that allows users to interact naturally with popular characters. It analyzes user input, generates appropriate responses, and translates them into voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized responses become possible. Below, the system's program and its processing are explained in natural language, with specific examples.

[1645] System Overview

[1646] User authentication / login

[1647] Device: The user launches the dedicated app and logs in by entering their account information (username and password).

[1648] Server: Receives the user's authentication information and checks it against the database to see if it is correct. If authentication is successful, it starts a session and sends the information back to the terminal. The terminal checks the authentication status and displays the main screen if successful, or an error message if unsuccessful.

[1649] Character Selection and Settings

[1650] Terminal: Sends a request to the server to display a list of characters available to the user on the main screen.

[1651] Server: Retrieves available character information from the character database and returns it to the device.

[1652] User: Select the desired character from the list and customize their personality, voice pitch, etc. on the detailed settings screen. The customized settings are sent from the device to the server.

[1653] Server: Stores the received configuration data with the user's profile.

[1654] Interaction Generation and Emotion Recognition

[1655] User: Enter text or voice input to the character. Voice input is converted to text using the device's voice recognition function.

[1656] Terminal: Sends user text input to the server.

[1657] Server: Passes the received text to a natural language processing engine for content analysis. Next, an emotion engine is used to recognize emotions from the user's input. Based on this emotion data, a generative AI generates an appropriate response. The response is then sent back from the server to the device.

[1658] On the device: The returned text response is passed to a speech synthesis engine to generate speech and play it back to the user.

[1659] Specific examples

[1660] Morning greetings and schedule check

[1661] User: Say "Good morning."

[1662] Device: Converts the user's speech into text and sends it to the server.

[1663] Server: Analyzes the input text and uses the emotion engine to recognize that the user is in a refreshing mood. It generates a response such as "Good morning! What a lovely morning. Would you like to check today's schedule?" and sends it to the device.

[1664] Terminal: The returned response is converted into speech using a speech synthesis engine and played back to the user. It says, "Good morning! What a beautiful morning. Would you like to check today's schedule?"

[1665] Adding a task

[1666] User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[1667] Terminal: Sends input text to the server.

[1668] Server: Analyzes the text and sets a reminder for the specified date and time. The emotion engine recognizes that the user is a little impatient and generates a confirmation message saying, "I've set a reminder for your meeting tomorrow at 9am. Is there anything else I can help you with?"

[1669] Device: Converts a confirmation message to speech and plays it to the user. Says, "I've set a reminder for your meeting tomorrow at 9:00. Is there anything else I can help you with?"

[1670] Learning and Personalization

[1671] Terminal: Periodically sends the user's interaction history to the server.

[1672] Server: Stores interaction data and emotion data, scans them, and learns user behavior patterns and emotional changes.

[1673] Server: Based on the learning results, the model is updated to generate personalized responses for the next interaction.

[1674] Terminal: Converts the personalized response into speech and provides it to the user.

[1675] In this way, the present invention generates responses based on user input and emotions, providing a natural interaction experience for the user. By responding to the user's preferences and emotions, the interaction becomes more satisfying and the relationship between the user and the character becomes deeper.

[1676] The processing flow will be explained below.

[1677] Step 1:

[1678] Device: The user launches the app, the app initializes, and the login screen appears.

[1679] Step 2:

[1680] User: Enter your existing account information and click the login button.

[1681] Step 3:

[1682] Terminal: Encodes the entered authentication information and sends it to the server.

[1683] Step 4:

[1684] Server: Receives the authentication information, checks it against a database, and if authentication is successful, starts the session and sends the information back to the device.

[1685] Step 5:

[1686] Terminal: Checks the authentication status, and displays the main screen if successful, or an error message if unsuccessful.

[1687] Step 6:

[1688] Terminal: On the main screen, sends a request to the server to display a list of characters available to the user.

[1689] Step 7:

[1690] Server: Retrieves available character information from the character database and returns it to the device.

[1691] Step 8:

[1692] Terminal: Displays a list of characters the user can choose from.

[1693] Step 9:

[1694] User: Select the desired character from the list and open the detailed settings screen.

[1695] Step 10:

[1696] User: Customize the character's personality, voice pitch, specific expressions, etc., and press the button to save the settings.

[1697] Step 11:

[1698] Device: Sends customized configuration data to the server.

[1699] Step 12:

[1700] Server: Stores the received configuration data with the user's profile.

[1701] Step 13:

[1702] User: Speaks to the character via text or voice.

[1703] Step 14:

[1704] Device: In the case of voice input, the voice recognition function converts the input into text and sends the text to the server.

[1705] Step 15:

[1706] Server: Passes the received text to a natural language processing engine and analyzes its content.

[1707] Step 16:

[1708] Server: Sends the parsed text to the emotion engine to recognize the user's emotion.

[1709] Step 17:

[1710] Server: Using the user's emotional data recognized by the emotion engine, the generative AI generates personalized responses.

[1711] Step 18:

[1712] Server: Sends the generated response to the terminal.

[1713] Step 19:

[1714] Terminal: The returned response sentence is passed to the speech synthesis engine to generate speech.

[1715] Step 20:

[1716] Terminal: Plays the generated audio and provides responses to the user.

[1717] Step 21:

[1718] User: Set a new task, for example, "Set a reminder for a meeting tomorrow at 9 AM."

[1719] Step 22:

[1720] Terminal: Sends a request to the server for task configuration.

[1721] Step 23:

[1722] Server: Analyzes received tasks and adds reminders to the task list at the set date and time.

[1723] Step 24:

[1724] Server: Generates a confirmation message for adding the task and sends it to the device.

[1725] Step 25:

[1726] Terminal: Converts the confirmation message into speech and provides it to the user.

[1727] Step 26:

[1728] Terminal: Periodically transmits the user's interaction history and emotion data to the server.

[1729] Step 27:

[1730] Server: Stores interaction and emotion data and associates it with user profiles.

[1731] Step 28:

[1732] Server: Analyzes the stored data and learns changes in user behavior patterns and emotions.

[1733] Step 29:

[1734] Server: Based on the learning results, generates personalized responses for the next interaction and updates the model.

[1735] Step 30:

[1736] Terminal: Converts the personalized response into speech and provides it to the user.

[1737] Example 2

[1738] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1739] Conventional dialogue systems simply respond to user input, making it difficult to recognize emotions in real time or generate personalized responses. Furthermore, they lack the ability to use the user's interaction history to improve future responses, resulting in low user satisfaction. Furthermore, limited options for customizing character personalities and settings make it difficult to provide a compelling experience for users.

[1740] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1741] In this invention, the server includes means for receiving voice or text input from a user, means for converting the user's voice input into text, natural language processing means for analyzing the user's input and generating an appropriate response, means for recognizing the user's emotions, and means for converting the generated response into voice and providing it to the user, thereby enabling recognition of the user's emotions in real time and generating personalized responses.

[1742] "Means for receiving voice or text input from a user" refers to a device that has an interface through which a user can provide input in the form of voice or text.

[1743] "Means for converting user voice input into text" refers to a device that uses voice recognition technology to convert speech into text form.

[1744] "Natural language processing means for analyzing user input and generating an appropriate response" refers to a device that uses natural language processing technology to analyze user input, understand the context and intent, and generate an appropriate response.

[1745] "Means for recognizing user emotions" refers to a device that uses analytical technology to extract and identify emotions from a user's voice or text.

[1746] The "means for converting the generated response into speech and providing it to the user" refers to a device that uses speech synthesis technology to convert a text response into speech and provide it to the user.

[1747] "Means for storing user profiles and behavioral history" refers to a device equipped with a database that stores users' personal information and past interaction data.

[1748] "Means for analyzing stored profiles and behavioral history to generate responses based on user preferences" refers to a device that uses technology to analyze stored data and generate individually optimized responses based on a user's preferences and past behavioral patterns.

[1749] "Means for storing character information and providing a character based on a user's selection" refers to a device that has a database and an interface for storing information on multiple characters and providing data on a character selected by a user.

[1750] "Means for customizing character personality and voice settings" refers to a device that uses an interface that allows a user to adjust or change a character's characteristics or voice features, and technology that stores those settings.

[1751] The present invention is a system for realizing natural interaction between a user and an interactive character, and is implemented by combining the following hardware and software.

[1752] System Overview

[1753] User authentication / login

[1754] Device:

[1755] The user launches a dedicated app and logs in by entering their username and password. When the user presses the "Login" button, the device encrypts the input information and sends it to the server. The hardware used is a typical computer device such as a smartphone, tablet, or PC. The software uses an encryption library (e.g., OpenSSL).

[1756] server:

[1757] The server compares the received authentication information with a database (e.g., MySQL), and if authentication is successful, it starts a session and returns the information to the terminal. The server software used can be a web server (e.g., Apache, Nginx) or a database server.

[1758] Device:

[1759] The device checks the received session token and displays the main screen if authentication is successful, or an error message if authentication is unsuccessful.The software used is a mobile app development framework (e.g., React Native, Flutter).

[1760] Character Selection and Settings

[1761] Device:

[1762] Send a request to the server to display the list of characters available to the user on the main screen, using an HTTP request.

[1763] server:

[1764] The server retrieves available character information from a character database (e.g., MongoDB) and returns it to the device. The returned data is in JSON format.

[1765] Device:

[1766] The device displays the received character information on the main screen. The user selects a character on the screen and customizes their personality, voice pitch, etc. on the detailed settings screen. The customized settings are then sent back to the server in JSON format.

[1767] server:

[1768] The server stores the received configuration data in a database together with the user's profile.

[1769] Interaction Generation and Emotion Recognition

[1770] User:

[1771] The user can input text or voice to the character. Voice input is converted to text using the device's voice recognition function (e.g., Google Speech-to-Text).

[1772] Device:

[1773] The device sends the user's text input to the server, using HTTP or WebSockets.

[1774] server:

[1775] The server passes the received text to a natural language processing engine (e.g., spaCy, Hugging Face Transformers) for content analysis. The server also uses an emotion recognition engine (e.g., a BERT-based model) to recognize the user's emotions. Based on this emotion data, the server generates an appropriate response using a generative AI model (e.g., OpenAI GPT-4) and sends it back to the device.

[1776] Device:

[1777] The device passes the returned text response to a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio and play it back to the user.

[1778] Specific examples

[1779] Morning greetings and schedule check

[1780] User:

[1781] Say "Good morning."

[1782] Device:

[1783] The user's speech is converted into text and sent to the server.

[1784] server:

[1785] The input text is analyzed using a natural language processing engine, and the emotion engine is used to recognize that the user is in a refreshing mood. The response "Good morning! What a lovely morning. Would you like to check today's schedule?" is generated and sent to the device.

[1786] Device:

[1787] The returned response is converted into speech by a speech synthesis engine and played back to the user.

[1788] Example prompts to input to a generative AI model:

[1789] "A user says good morning. He seems very refreshed. How would you respond?"

[1790] Adding a task

[1791] User:

[1792] Type "Set a meeting reminder for tomorrow at 9 AM."

[1793] Device:

[1794] Sends the input text to the server.

[1795] server:

[1796] It analyzes the text and sets a reminder for the specified date and time. The emotion engine recognizes that the user is a little impatient and generates a confirmation message saying, "I've set a reminder for your meeting tomorrow at 9:00. Is there anything else I can help you with?"

[1797] Device:

[1798] A confirmation message is converted into audio and played to the user.

[1799] Example prompts to input to a generative AI model:

[1800] "A user types that they want to set a reminder for a meeting tomorrow at 9 AM. They seem a little impatient. How would you respond?"

[1801] Learning and Personalization

[1802] Device:

[1803] The user's interaction history is periodically sent to the server.

[1804] server:

[1805] It stores interaction and emotion data, analyzes them, and learns user behavior patterns and emotional changes. Based on the learning results, it updates the model to generate personalized responses for the next interaction.

[1806] Device:

[1807] The personalized response is converted into speech and provided to the user.

[1808] In this way, the present invention generates responses based on user input and emotions, providing a natural interaction experience for the user. By responding to the user's preferences and emotions, the interaction becomes more satisfying and the relationship between the user and the character becomes deeper.

[1809] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1810] An explanation of the processing flow of this system's program, broken down into processing steps

[1811] Step 1:

[1812] User Login

[1813] Input: The user launches the dedicated app and enters their username and password.

[1814] Specific operation: When the user presses the "Login" button, the terminal encrypts the username and password and sends them to the server.

[1815] Output: Encrypted user credentials are sent to the server.

[1816] Step 2:

[1817] Authentication verification

[1818] Input: Encrypted user credentials

[1819] What happens: The server checks the received credentials against its database. It performs a database query to verify that the username and password match.

[1820] Output: Generates a result indicating whether authentication was successful or not, and sends it back to the terminal along with the session information.

[1821] Step 3:

[1822] Viewing the login results

[1823] Input: Authentication result and session information

[1824] Specific operation: The device checks the received authentication result, and displays the main screen if successful, or an error message if unsuccessful.

[1825] Output: Display of the main screen or error message.

[1826] Step 4:

[1827] Character Information Request

[1828] Input: Request required to display the main screen

[1829] Specific operation: The device sends a request to the server to obtain available character information.

[1830] Output: Sends a request to the server to get character data.

[1831] Step 5:

[1832] Acquiring and returning character information

[1833] Input: Character data acquisition request

[1834] Specific operation: The server retrieves character information from the character database and returns it to the device in JSON format.

[1835] Output: Returns character information to the terminal.

[1836] Step 6:

[1837] Displaying the character list

[1838] Input: Character information in JSON format

[1839] Specific operation: The device analyzes character information and displays it in a list on the main screen.

[1840] Output: Display the character list on the main screen.

[1841] Step 7:

[1842] Character selection and detailed settings

[1843] Input: User-selected character and customization settings

[1844] Specific operation: The user selects a character and customizes the character's personality, voice pitch, etc. in the detailed settings screen. These settings are sent from the device to the server.

[1845] Output: Sends customization information to the server.

[1846] Step 8:

[1847] Saving the configuration information

[1848] Input: Customization setting information

[1849] Specific operation: The server stores the received customization setting information in the database.

[1850] Output: Save customized settings information to a database.

[1851] Step 9:

[1852] User Input and Conversion

[1853] Input: User voice or text input

[1854] Specific operation: The user inputs to the character by voice or text. In the case of voice input, the device converts the voice into text in real time.

[1855] Output: The converted text data is sent to the server.

[1856] Step 10:

[1857] Natural Language Processing and Emotion Recognition

[1858] Input: Text data

[1859] Specific operation: The server passes the text data to a natural language processing engine for analysis, and recognizes emotions using an emotion engine.

[1860] Output: Analysis results and emotion data.

[1861] Step 11:

[1862] Generate a response

[1863] Input: Analysis results and emotion data

[1864] Specific operation: The server uses a generative AI model (e.g., OpenAI GPT-4) to generate an appropriate response.

[1865] Output: Sends the generated response to the terminal.

[1866] Step 12:

[1867] Audio generation and playback

[1868] Input: Generated response

[1869] Specific operation: The device passes the response sentence to a speech synthesis engine to generate speech and play it back to the user.

[1870] Output: Playback of the generated audio.

[1871] Example prompt

[1872] 1. Morning greeting and schedule prompt:

[1873] "A user says good morning. He seems very refreshed. How would you respond?"

[1874] 2. Prompt for adding a task:

[1875] "A user types that they want to set a reminder for a meeting tomorrow at 9 AM. They seem a little impatient. How would you respond?"

[1876] (Application example 2)

[1877] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1878] Conventional interaction systems have fixed responses to user inputs and have not been able to realize natural dialogue that takes into account the user's emotions and gaze. Furthermore, in the work environment, there is a lack of efficient and safe work support, which has led to the issue of workers being unable to quickly obtain the information they need.

[1879] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into voice and providing it to the user, means for recognizing the user's emotions, and means for tracking the user's gaze. This enables natural dialogue that takes the user's emotions and gaze into consideration. The server also includes means for saving a user's profile and behavioral history, means for analyzing the saved profile and behavioral history and generating a response based on the user's preferences, means for visually providing the generated response, and means for displaying information via a visual output device worn by the user. This enables workers to efficiently and safely obtain necessary information in the work environment and support them. The server also includes means for saving character information and providing a character based on the user's selection, means for customizing the character's personality and voice settings, and means for creating appropriate prompt sentences based on a generative AI model. This allows for more personalized responses to the user and an improved dialogue experience.

[1880] "Means for receiving input from a user" refers to hardware and software for capturing and transmitting information entered by a user through speech or text to the system.

[1881] "Means for analyzing user input and generating an appropriate response" refers to a technology that analyzes received user input information using a natural language processing engine or the like and generates a response based on that information.

[1882] The "means for converting the generated response into speech and providing it to the user" is a technology that uses a speech synthesis engine to convert the generated text response into speech and plays it aloud to the user through a speaker or the like.

[1883] "Means for recognizing user emotions" refers to technology that analyzes emotions from the user's voice, text, facial expressions, etc., and determines the user's current feelings and state.

[1884] "Means for tracking the user's gaze" refers to technology that uses a camera or sensor to track the direction or object the user is looking at and records it as gaze information.

[1885] "Means for storing a user's profile and behavior history" refers to technology for storing a user's personal information, past operation history, behavior patterns, etc. in a storage device.

[1886] "Means for analyzing stored profiles and behavioral history and generating responses based on the user's preferences" refers to technology that analyzes stored data and generates personalized responses tailored to the user's tendencies and preferences.

[1887] The "means for visually presenting the generated response" is a technique for displaying the generated response on a display device as visual information such as text, images, or videos.

[1888] "Means for displaying information via a visual output device worn by a user" refers to technology for displaying information through a device worn by a user, such as smart glasses or a head-mounted display.

[1889] "Means for saving character information and providing characters based on user selection" refers to technology that saves character attributes and settings in a database and provides the character on screen or via audio based on the user's selection.

[1890] "Means for customizing character personality and voice settings" refers to technology that allows a user to change the personality, voice pitch, tone, etc. of a character selected by the user to suit the user's preferences.

[1891] "Means for creating appropriate prompt sentences based on a generative AI model" refers to a technology that uses an AI model to generate optimal response sentences based on input from a user.

[1892] The present invention provides a system that supports factory work by enabling natural interactions with input from users. This system allows characters to interact with the user through smart glasses worn by the user, improving work efficiency and safety. Detailed embodiments for implementing the present invention are described below.

[1893] System Overview

[1894] The system consists of the following main components:

[1895] 1. A means of receiving input from the user

[1896] Hardware: Smart glasses (e.g., Google Glass)

[1897] Process: Capture the user's voice and gaze data and send it to the system.

[1898] 2. A means of analyzing user input and generating an appropriate response

[1899] Software: Speech recognition software (e.g., Google Cloud Speech-to-Text), natural language processing engines (e.g., OpenAI's GPT-4)

[1900] Process: Convert the captured audio into text, analyze the text and generate an appropriate response.

[1901] 3. A means of converting the generated response into speech and providing it to the user

[1902] Software: Speech synthesis engine (e.g. Amazon Polly)

[1903] Process: The generated text is converted into speech and provided to the user through the smart glasses speaker.

[1904] 4. How to Recognize User Emotions

[1905] Software: Emotion recognition engine (e.g., Microsoft Azure Emotion API)

[1906] Process: Analyze the user's emotions from captured voice and gaze data and provide feedback to the system.

[1907] 5. A way to track the user's gaze

[1908] Hardware: Built-in camera in smart glasses

[1909] Process: Track the user's gaze in real time and analyze the gaze data.

[1910] 6. Means of storing user profiles and activity histories

[1911] Hardware: Cloud-based database

[1912] Process: Stores user personal information, past operation history, behavioral patterns, etc.

[1913] 7. A means of analyzing stored profiles and behavioral history to generate responses based on user preferences.

[1914] Software: Data analysis engine

[1915] Process: Analyze the stored data and generate a personalized response based on the user's preferences.

[1916] 8. A visual means of providing the generated response

[1917] Hardware: Smart glasses display

[1918] Process: The generated response is visually displayed and presented to the user.

[1919] 9. Means for storing character information and providing characters based on user selection

[1920] Hardware: Cloud-based database

[1921] Process: Stores character attributes and settings and provides characters based on user selections.

[1922] 10. A way to customize character personalities and voice settings

[1923] Software: Character Setting Engine

[1924] Process: Modify the personality, voice pitch, tone, etc. of the user-selected character to suit the user's preferences.

[1925] 11. A means of creating appropriate prompts based on generative AI models

[1926] Software: Generative AI models (e.g., OpenAI's GPT-4)

[1927] Process: Use an AI model to generate optimal responses based on user input.

[1928] Specific examples

[1929] Check the checklist before starting work

[1930] 1. User: Talk to the character and say, "Check today's work."

[1931] 2. Smart glasses: Capture the user's voice and send it to voice recognition software.

[1932] 3. Server: Converts the input speech into text and passes it to the natural language processing engine.

[1933] 4. Natural language processing engine: Analyzes the text and generates a corresponding checklist.

[1934] 5. Emotion Recognition Engine: Recognizes when the user is focused.

[1935] 6. Generative AI: Generates a response such as, "Please be mindful of safety as you work today. Your first task is maintenance on Machine A. Would you like to review the checklist?"

[1936] 7. Speech synthesis engine: Converts the response into speech and provides it to the user through the smart glasses' speaker.

[1937] 8. Smart glasses: Visually display the generated text on a display.

[1938] Prompt Sentence Examples

[1939] Character Name: Factory character "Mechatrobo"

[1940] Input sentence: Tell the character "Tell me how to operate machine B"

[1941] Emotions: A little anxious

[1942] Response: Mecha-Robo: "I'll teach you how to operate Machine B. First, pull down the lever on the left. Next, press the button in the middle. If you have any trouble, please stay calm and let's go through each step one by one."

[1943] In this way, the system of the present invention improves work efficiency and safety by analyzing the user's voice and emotions and providing visual and audio responses based on those analyses. By personalizing responses based on the user's profile and behavioral history, the system provides higher satisfaction and enhances its utility in the work environment.

[1944] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1945] Step 1:

[1946] User: The user puts on the smart glasses and says, "See what I did today."

[1947] Input: User's voice

[1948] Output: Capture of audio data

[1949] How it works: The smart glasses capture the user's voice with a built-in microphone.

[1950] Step 2:

[1951] Device: The smart glasses capture and transmit the audio to the voice recognition software.

[1952] Input: Audio data

[1953] Output: Text data

[1954] How it works: Uses a speech recognition engine (e.g. Google Cloud Speech-to-Text) to convert input voice data into text.

[1955] Step 3:

[1956] Terminal: Sends text data to the server.

[1957] Input: Text data

[1958] Output: Sends text data to the server

[1959] How it works: Smart glasses send text data over the network to a server.

[1960] Step 4:

[1961] Server: The server analyzes the received text data.

[1962] Input: Text data

[1963] Output: Analysis results

[1964] How it works: It uses a natural language processing engine (e.g., OpenAI's GPT-4) to analyze text data and understand user intent.

[1965] Step 5:

[1966] Server: The server recognizes the user's emotions using an emotion recognition engine.

[1967] Input: Text data

[1968] Output: Emotion data

[1969] How it works: It uses an emotion recognition engine (e.g. Microsoft Azure's Emotion API) to analyze the user's emotions from the input text data.

[1970] Step 6:

[1971] Server: The server generates an appropriate response using a generative AI model based on the analysis results and emotion data.

[1972] Input: Text data, emotion data

[1973] Output: Response text

[1974] How it works: Uses a generative AI model (e.g., OpenAI's GPT-4) to generate a response text based on the analysis results and sentiment data.

[1975] Examples:

[1976] Character name: Factory character "Work Robot"

[1977] Input: "Check today's work"

[1978] Emotion: Focused

[1979] Response: Worker robot: "Please be mindful of safety as you work today. The first task is maintenance on Machine A. Would you like to review the checklist?"

[1980] "

[1981] Step 7:

[1982] Server: Pass the generated response text to the speech synthesis engine.

[1983] Input: Response text

[1984] Output: Audio data

[1985] What it does: Converts response text into speech using a text-to-speech engine (e.g., Amazon Polly).

[1986] Step 8:

[1987] Terminal: Sends audio data to smart glasses.

[1988] Input: Audio data

[1989] Output: Receives audio data

[1990] How it works: Audio data is sent over the network to smart glasses.

[1991] Step 9:

[1992] Terminal: The smart glasses play the audio data and provide it to the user.

[1993] Input: Audio data

[1994] Output: Play audio

[1995] What it does: Plays audio using the smart glasses' speakers and provides it to the user.

[1996] Step 10:

[1997] Terminal: Visually displays the generated response text.

[1998] Input: Response text

[1999] Output: Visual information

[2000] What it does: Visually display the response text on the smart glasses display to provide information to the user.

[2001] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2002] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2003] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2004] [Fourth embodiment]

[2005] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2006] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2007] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2008] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2009] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2010] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2011] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2012] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2013] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2014] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2015] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2016] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2017] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2018] This invention is a system that allows users to interact naturally with popular characters by analyzing input from the user and generating appropriate responses and translating them into voice. Below, the system program and its processing are explained in natural language, with specific examples.

[2019] System Overview

[2020] User authentication / login

[2021] Device: The user launches the dedicated app and logs in by entering their account information (username and password).

[2022] Server: Receives the user's authentication information and checks it against the database to see if it is correct. If authentication is successful, it starts a session and sends the information back to the terminal. The terminal checks the authentication status and displays the main screen if successful, or an error message if unsuccessful.

[2023] Character Selection and Settings

[2024] Terminal: The main screen displays a list of characters available to the user. The list is retrieved from the server.

[2025] User: Select the desired character and customize their personality, voice pitch, etc. on the detailed settings screen. The customized settings are sent from the device to the server.

[2026] Server: Stores the received configuration data with the user's profile.

[2027] Interaction Generation

[2028] User: Provides text or voice input to the character. Voice input is converted to text using the device's voice recognition function.

[2029] Terminal: Sends user text input to the server.

[2030] Server: Passes the received text to a natural language processing engine to analyze the input. Based on the analysis results, a generative AI generates an appropriate response. The response is then sent back from the server to the device.

[2031] On the device: The returned text response is passed to a speech synthesis engine to generate speech and play it back to the user.

[2032] Specific examples

[2033] Morning greetings and schedule check

[2034] User: Say "Good morning!"

[2035] Device: Converts the user's speech into text and sends it to the server.

[2036] Server: Analyzes the input text and generates a response such as "Good morning! Would you like to check today's schedule?". Then, sends the generated response back to the device.

[2037] Terminal: The returned response is converted into speech by a speech synthesis engine and played back to the user. It says, "Good morning! Would you like to check today's schedule?"

[2038] Adding a task

[2039] User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[2040] Terminal: Sends input text to the server.

[2041] Server: Parse the text and set a reminder for the specified date and time. Generate a confirmation message saying "Your reminder for the meeting has been set for tomorrow at 9 AM."

[2042] Device: Converts a confirmation message into speech and provides it to the user. Says, "I've set a reminder for a meeting tomorrow at 9 AM."

[2043] Learning and Personalization

[2044] Terminal: Sends the user's usage history to the server.

[2045] Server: Stores interaction data and analyzes it to learn user behavior patterns and preferences.

[2046] Server: Generates more personalized responses to users based on the learning results.

[2047] The device converts this personalized response into speech and provides it to the user.

[2048] In this way, the system allows users to naturally interact with the character they select, providing various support such as daily tasks and setting reminders. The interaction is personalized based on the user's preferences and behavioral patterns, providing an experience that makes it seem as if the user and the character are living together.

[2049] The processing flow will be explained below.

[2050] Step 1:

[2051] Device: The user launches the app, the app initializes, and the login screen appears.

[2052] Step 2:

[2053] User: Enter your existing account information and click the login button.

[2054] Step 3:

[2055] Terminal: Encodes the entered authentication information and sends it to the server.

[2056] Step 4:

[2057] Server: Receives the authentication information, checks it against a database, and if authentication is successful, starts the session and sends the information back to the device.

[2058] Step 5:

[2059] Terminal: Checks the authentication status, and displays the main screen if successful, or an error message if unsuccessful.

[2060] Step 6:

[2061] Terminal: On the main screen, sends a request to the server to display a list of characters available to the user.

[2062] Step 7:

[2063] Server: Retrieves available character information from the character database and returns it to the device.

[2064] Step 8:

[2065] Terminal: Displays a list of characters the user can choose from.

[2066] Step 9:

[2067] User: Select the desired character from the list and open the detailed settings screen.

[2068] Step 10:

[2069] User: Customize the character's personality, voice pitch, specific expressions, etc., and press the button to save the settings.

[2070] Step 11:

[2071] Device: Sends customized configuration data to the server.

[2072] Step 12:

[2073] Server: Stores the received configuration data with the user's profile.

[2074] Step 13:

[2075] User: Speaks to the character via text or voice.

[2076] Step 14:

[2077] Device: In the case of voice input, the voice recognition function converts the input into text and sends the text to the server.

[2078] Step 15:

[2079] Server: Passes the received text to a natural language processing engine and analyzes its content.

[2080] Step 16:

[2081] Server: Based on the results of natural language processing, a generative AI generates an appropriate response sentence.

[2082] Step 17:

[2083] Server: Sends the generated response to the terminal.

[2084] Step 18:

[2085] Terminal: The returned response sentence is passed to the speech synthesis engine to generate speech.

[2086] Step 19:

[2087] Terminal: Plays the generated audio and provides responses to the user.

[2088] Step 20:

[2089] User: Set a new task, for example, "Set a reminder for a meeting tomorrow at 9 AM."

[2090] Step 21:

[2091] Terminal: Sends a request to the server for task configuration.

[2092] Step 22:

[2093] Server: Analyzes received tasks and adds reminders to the task list at the set date and time.

[2094] Step 23:

[2095] Server: Generates a confirmation message for adding the task and sends it to the device.

[2096] Step 24:

[2097] Terminal: Converts the confirmation message into speech and provides it to the user.

[2098] Step 25:

[2099] Terminal: Periodically sends the user's interaction history to the server.

[2100] Step 26:

[2101] Server: Stores interaction data and associates it with user profiles.

[2102] Step 27:

[2103] Server: Analyzes the stored data and learns user behavior patterns and preferences.

[2104] Step 28:

[2105] Server: Generates more personalized responses based on the learning results.

[2106] Step 29:

[2107] Terminal: Converts the personalized response into speech and provides it to the user.

[2108] Example 1

[2109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2110] Conventional dialogue systems have struggled to achieve natural interactions with users. In particular, they have had challenges in accurately analyzing user input and providing appropriate responses in a natural manner. Furthermore, technology for providing personalized responses based on user preferences and behavioral patterns has also been inadequate. This has led to problems such as a poor user experience and a decline in system utilization.

[2111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2112] In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for using a speech recognition engine to convert the user's speech into text, means for analyzing the input using a natural language processing engine, means for generating a response using a generative artificial intelligence model, and means for converting the response into speech using a speech synthesis engine. This enables the user's input to be accurately analyzed and a response to be provided in a natural manner. Furthermore, by storing the user's profile and behavioral history and providing personalized responses based on them, the quality of the user experience can be improved and the system utilization rate can be increased.

[2113] "Means for receiving input from a user" refers to the facility by which a user provides input to the system in text or speech form.

[2114] The "means for analyzing user input and generating an appropriate response" is a function for analyzing input data received from a user and generating an appropriate response based on the content of the input data.

[2115] The "means for converting the generated response into speech and providing it to the user" is a function for converting the generated response in text format into speech format and playing it back to the user.

[2116] A "speech recognition engine that converts user speech into text" is a software component that converts user spoken speech into text format.

[2117] A "natural language processing engine" is a software component that analyzes text input from a user and understands its meaning.

[2118] A "generative artificial intelligence model" is a machine learning model for generating responses based on analyzed text.

[2119] A "speech synthesis engine" is a software component for converting generated text responses into speech form.

[2120] "Means for storing user profiles and behavioral history" refers to a function for storing users' personal information and past interaction data.

[2121] "Means for analyzing stored profiles and behavioral history and generating responses based on user preferences" refers to a function that analyzes stored data to understand the user's preferences and behavioral patterns and generates responses based on them.

[2122] The "means for learning user behavior patterns and preferences based on interaction data" is a function that utilizes the dialogue history with the user to learn the user's behavior patterns and preferences.

[2123] The "means for saving character information and providing a character based on the user's selection" is a function for saving information about each character and providing that character based on the user's selection.

[2124] "Means for customizing character personality and voice settings" is a function that allows the user to adjust settings related to the character's personality and voice.

[2125] The "means for saving customized settings" is a function for saving changes to character settings made by the user.

[2126] The present invention is a system for interacting with a user that utilizes advanced technology to achieve natural interaction by analyzing user input, generating appropriate responses, and providing them to the user in spoken form.

[2127] System Overview

[2128] Hardware and software used

[2129] This system uses the following hardware and software:

[2130] Speech recognition engine: Use a common speech recognition technology. Here, we will use the widely used Google Speech-to-Text as an example.

[2131] Natural Language Processing Engine: An engine for analyzing the meaning of text data, using advanced natural language processing techniques such as OpenAI GPT-4.

[2132] Generative AI model: A machine learning model that generates appropriate responses based on user input.

[2133] Speech synthesis engine: An engine for converting the generated text response into speech, specifically Amazon Polly.

[2134] User-System Interaction

[2135] User authentication / login

[2136] The user launches the dedicated app on their device and enters their account information (username and password) on the login screen. The device sends this information to the server, which then collates it with a database for authentication. If authentication is successful, the server creates a session and returns that information to the device. The device checks the authentication status based on the received session information and displays the main screen.

[2137] Character Selection and Settings

[2138] When the user taps the "Select Character" button on the main screen, the device sends a request for a character list to the server. The server retrieves character information from the database and sends it back to the device. The device displays the received character list on the screen, and the user selects the desired character. The user can further customize the character's personality and voice pitch on the detailed settings screen, and these settings are sent to the server. The server saves the received settings in its database.

[2139] Interaction Generation

[2140] Users can provide text or voice input to the character. For voice input, the device captures the audio and converts it into text using Google Speech-to-Text. This text is sent to a server, which passes it to a natural language processing engine (GPT-4) for analysis. Based on the analysis results, a generative AI model generates an appropriate response, which is then sent back from the server to the device. The device then converts the received response into speech using a speech synthesis engine (Amazon Polly) and plays it back to the user.

[2141] Specific examples

[2142] Morning greetings and schedule check

[2143] When a user says "Good morning!", the device converts the speech to text and sends it to the server. The server generates a response, "Good morning! Would you like to check today's schedule?" and sends it back to the device. The device converts the response to speech and plays it to the user: "Good morning! Would you like to check today's schedule?"

[2144] Adding a task

[2145] When a user types "Set a reminder for a meeting tomorrow at 9 AM," the device sends the text to the server. The server parses the instruction and sets a reminder for the specified date and time. It generates a confirmation message saying "A reminder for a meeting tomorrow at 9 AM" and sends it back to the device. The device converts it into speech and provides it to the user: "A reminder for a meeting tomorrow at 9 AM."

[2146] Learning and Personalization

[2147] Every time a user uses the system, their usage history is sent to the server and stored in a database. The server analyzes this data and learns the user's behavioral patterns and preferences. Based on this learning, the system can generate more personalized responses. For example, if a user frequently adds a specific task, the system will provide information and suggestions related to that task.

[2148] In this way, the system of the present invention uses advanced speech recognition, natural language processing, and generative artificial intelligence models to achieve natural interactions with users and support daily task management and information provision. Examples of prompt sentences include "Good morning!", "Set a reminder for a meeting tomorrow at 9:00 AM," and "What's the weather like today?"

[2149] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2150] Step 1:

[2151] Device: The user launches the dedicated app and enters their account information (username and password) on the login screen. This input data is sent to the server for login authentication.

[2152] Input: Username, Password

[2153] Output: Login request sent to the server

[2154] Step 2:

[2155] Server: Receives the user's account information and checks it against a database. If authentication is successful, the server creates a session and sends the information back to the device. If authentication fails, it sends back an error message.

[2156] Input: Username, Password

[2157] Data processing: database matching, session generation

[2158] Output: Session information or error message

[2159] Step 3:

[2160] Terminal: Checks the authentication status based on the session information received from the server and displays the main screen. If an error message is returned, the error message is displayed.

[2161] Input: Session information or error message

[2162] Output: Main screen or error message display

[2163] Step 4:

[2164] User: Tap the "Character Selection" button on the main screen.

[2165] Input: Tap

[2166] Output: Character selection request

[2167] Step 5:

[2168] Terminal: Sends a character selection request to the server.

[2169] Input: Character selection request

[2170] Output: Character selection request sent to the server

[2171] Step 6:

[2172] Server: Retrieves character information from the database and sends it back to the device.

[2173] Input: Character selection request

[2174] Data processing: Database matching, character information acquisition

[2175] Output: Retrieved character information

[2176] Step 7:

[2177] Terminal: Display the received character information on the screen.

[2178] Input: Character information

[2179] Output: Display the character list

[2180] Step 8:

[2181] User: Select the desired character from the list and customize their personality and voice pitch in the advanced settings screen.

[2182] Input: Character selection, customization settings

[2183] Output: Customization settings information

[2184] Step 9:

[2185] Device: Sends customized configuration information to the server.

[2186] Input: Customization setting information

[2187] Output: Customization settings sent to the server

[2188] Step 10:

[2189] Server: Saves customizations to a database.

[2190] Input: Customization setting information

[2191] Data processing: Database storage

[2192] Output: Save complete message

[2193] Step 11:

[2194] User: Provides text or voice input to the character.

[2195] Input: Text or voice input

[2196] Output: Input data

[2197] Step 12:

[2198] On the device: For voice input, the speech is converted to text using Google Speech-to-Text, and the converted text is sent to the server.

[2199] Input: Voice input

[2200] Data processing: speech recognition, text conversion

[2201] Output: Text data

[2202] Step 13:

[2203] Terminal: Sends text input or converted text to the server.

[2204] Input: Text data

[2205] Output: Text data sent to the server

[2206] Step 14:

[2207] Server: Passes the received text to a natural language processing engine (GPT-4) and analyzes its content.

[2208] Input: Text data

[2209] Data Processing: Natural Language Processing

[2210] Output: Analysis results

[2211] Step 15:

[2212] Server: Based on the analysis results, the generative AI model generates an appropriate response, which is then sent back to the device.

[2213] Input: Analysis results

[2214] Data processing: response generation

[2215] Output: Response data

[2216] Step 16:

[2217] Terminal: Passes the received response data to the speech synthesis engine (Amazon Polly) and converts it into speech.

[2218] Input: Response data

[2219] Data processing: voice synthesis

[2220] Output: Audio data

[2221] Step 17:

[2222] Terminal: Plays the generated audio to the user.

[2223] Input: Audio data

[2224] Output: Audio playback

[2225] Specific operation example

[2226] Examples of morning greetings and schedule confirmation

[2227] 1. User: Says "Good morning!"

[2228] 2. On the device: Capture the user's voice and convert it to text using Google Speech-to-Text.

[2229] 3. Terminal: Send the converted text "Good morning!" to the server.

[2230] 4. Server: Passes the received text "Good morning!" to the natural language processing engine (GPT-4) and instructs it to generate a response.

[2231] 5. Server: Generates a response saying "Good morning! Would you like to check today's schedule?" and sends it back to the device.

[2232] 6. Terminal: The returned response is passed to the speech synthesis engine (Amazon Polly) and converted into speech.

[2233] 7. Terminal: Play the audio to the user: "Good morning! Would you like to check today's schedule?"

[2234] Example of adding a task

[2235] 1. User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[2236] 2. Terminal: Sends the entered text to the server.

[2237] 3. Server: Passes the received text to the natural language processing engine (GPT-4) for analysis.

[2238] 4. Server: Registers the instruction "Set a meeting reminder for tomorrow at 9:00" in the scheduler.

[2239] 5. Server: Generates a confirmation message saying "Your meeting reminder has been set for tomorrow at 9 AM" and sends it back to the device.

[2240] 6. Device: Passes the confirmation message to the speech synthesis engine (Amazon Polly) and converts it into speech.

[2241] 7. Device: The generated speech "A meeting reminder has been set for tomorrow at 9:00" is provided to the user.

[2242] Examples of prompt statements

[2243] "good morning!"

[2244] "Set a meeting reminder for tomorrow at 9 AM"

[2245] "How's the weather today?"

[2246] (Application example 1)

[2247] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2248] Conventional interactive systems have difficulty in providing natural dialogue between users and characters, or in providing information based on the user's preferences. Furthermore, they lack the ability to instantly recommend appropriate information in response to user questions, limiting the user experience. This makes it difficult for users to fully enjoy the dialogue with characters and obtain personalized responses.

[2249] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2250] In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for converting speech input into text, means for analyzing the text input and acquiring and recommending information, and means for saving and reusing information based on user selection, thereby enabling the user to enjoy natural dialogue with a character and receive personalized information recommendations through the interaction.

[2251] A "means for receiving input from a user" is a device or interface that receives input when a user asks a question or gives a command by text or voice.

[2252] The "means for analyzing user input and generating an appropriate response" refers to a device or algorithm that analyzes user input and generates an optimal response for the user based on the content of the input.

[2253] The "means for converting the generated response into speech and providing it to the user" refers to a device or software for converting the generated text response into speech and playing the speech to the user.

[2254] A "voice-to-text converter" is a device or software that converts a user's speech into text in a machine-parseable format.

[2255] The "means for analyzing text input and retrieving and recommending information" refers to a device or algorithm that analyzes the content input as text, retrieves appropriate information, and recommends it to the user.

[2256] The "means for saving and reusing information based on user selection" refers to a device or software that saves information selected or set by the user and reuses it in a later interaction.

[2257] "Means for storing user profiles and behavioral histories" refers to devices or software that store users' personal information and behavioral histories in a database.

[2258] "Means for analyzing stored profile and behavioral history and generating responses based on user preferences" refers to devices or algorithms that analyze stored data and generate responses tailored to the user's preferences.

[2259] The "means for storing character information and providing a character based on a user's selection" refers to a device or software that stores information about a character and provides the character selected by the user on the system.

[2260] "Means for customizing the character's personality and voice settings" refers to devices or software that allow the user to customize the character's personality, voice pitch, etc.

[2261] The "means for personalizing information provided by a character based on user preferences" refers to a device or algorithm that customizes information provided by a character based on user preferences.

[2262] System Overview

[2263] The present invention is a system that includes a set of functions such as means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into speech and providing it to the user, means for converting speech input into text, means for analyzing the text input and retrieving and recommending information, and means for storing and reusing information based on user selection.

[2264] Hardware and software used

[2265] Hardware: Smartphone

[2266] software:

[2267] Speech recognition engine: Google Cloud Speech-to-Text

[2268] Natural Language Processing Engine: OpenAI GPT-4

[2269] Speech synthesis engine: Amazon Polly

[2270] Database: Firebase

[2271] Interface: Flutter

[2272] Operating procedure

[2273] User authentication / login

[2274] The user launches a dedicated app on their smartphone and logs in by entering their account information (username and password). The server receives the user's authentication information and verifies it against a database to ensure it is correct. If authentication is successful, the server starts a session and returns the information to the device. The device then checks the authentication status, and displays the main screen if successful, or an error message if login is unsuccessful.

[2275] Character Selection and Settings

[2276] The user can view a list of available characters on the main screen and select the desired character. It is also possible to customize the character's personality, voice pitch, etc. The customized settings are sent from the device to the server, which then stores the received settings data along with the user's profile.

[2277] Interaction Generation

[2278] The user provides text or voice input to the character. For voice input, the device converts the speech to text using Google Cloud Speech-to-Text. The converted text input is sent to the server, which passes it to a natural language processing engine (GPT-4) for analysis. Based on the analysis results, the generative AI (GPT-4) generates an appropriate response, which is sent back from the server to the device. The device then converts the returned text response into speech using Amazon Polly and plays it back to the user.

[2279] Specific examples

[2280] Movie Recommendations

[2281] A user speaks to their smartphone, saying, "Tell me some recommended action movies." The device converts the speech into text using Google Cloud Speech-to-Text and sends it to the server. The server analyzes the text, obtains the appropriate movie information, and generates a response. For example, GPT-4 generates a response such as, "The recommended action movie is 'Action Movie A.'" The generated response is converted into audio using Amazon Polly and played back to the user.

[2282] Prompt example

[2283] A specific example of a prompt is as follows:

[2284] User Question: What are some good action movies? Movie Type: Action. Please recommend both new and classic movies.

[2285] In this way, a system can be constructed that allows natural dialogue via characters and provides appropriate information in response to specific user questions.

[2286] As described above, the system of the present invention can provide personalized information recommendations while allowing the user to naturally interact with a character selected by the user, thereby improving the user experience.

[2287] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2288] Step 1:

[2289] The user launches a dedicated app on their smartphone and logs in by entering their account information (username and password). The input data is sent from the device to the server. The server receives this input, checks it against a database to see if it is correct, and returns the results. The server outputs session information if authentication is successful, or an error message if it fails.

[2290] Step 2:

[2291] If login is successful, the terminal displays the main screen to the user. A list of characters available to the user is displayed on the screen, waiting for a selection. To generate the list, the terminal obtains character information from the server and provides it to the user. Character information consists of the character's name, appearance, personality, etc.

[2292] Step 3:

[2293] The user selects the desired character and customizes the character's personality, voice pitch, etc. on the detailed settings screen. This setting data is sent from the device to the server, which saves it along with the user's profile. The server outputs a message confirming that the save was successful.

[2294] Step 4:

[2295] The user begins a dialogue with the selected character. The user speaks, "Tell me some good action movies." The voice input is converted to text by the device's Google Cloud Speech-to-Text engine. The converted text is then sent to the server.

[2296] Step 5:

[2297] The server passes the received text to a natural language processing engine (GPT-4). The server processes the data to generate a prompt, formatting it like this: "User question: What action movies do you recommend? Movie type: Action. Please recommend both new and classic movies." GPT-4 then parses this input and generates an appropriate response. The server's output is the generated text response.

[2298] Step 6:

[2299] The server returns the generated response text to the device. The device converts the response text into speech using Amazon Polly. Here, the input is the text response from GPT-4, and the output is the audio data. The device plays this audio and provides it to the user as "The recommended action movie is 'Action Movie A'."

[2300] Step 7:

[2301] When a user selects a particular movie, the selection is sent from the device to the server, which stores this information as a user history and uses it for future recommendations. The stored information is also used as feedback data to personalize future interactions.

[2302] By implementing each of the above steps in order, users will be able to receive personalized movie recommendations while enjoying natural interactions with characters.

[2303] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2304] This invention is a system that allows users to interact naturally with popular characters. It analyzes user input, generates appropriate responses, and translates them into voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized responses become possible. Below, the system's program and its processing are explained in natural language, with specific examples.

[2305] System Overview

[2306] User authentication / login

[2307] Device: The user launches the dedicated app and logs in by entering their account information (username and password).

[2308] Server: Receives the user's authentication information and checks it against the database to see if it is correct. If authentication is successful, it starts a session and sends the information back to the terminal. The terminal checks the authentication status and displays the main screen if successful, or an error message if unsuccessful.

[2309] Character Selection and Settings

[2310] Terminal: Sends a request to the server to display a list of characters available to the user on the main screen.

[2311] Server: Retrieves available character information from the character database and returns it to the device.

[2312] User: Select the desired character from the list and customize their personality, voice pitch, etc. on the detailed settings screen. The customized settings are sent from the device to the server.

[2313] Server: Stores the received configuration data with the user's profile.

[2314] Interaction Generation and Emotion Recognition

[2315] User: Enter text or voice input to the character. Voice input is converted to text using the device's voice recognition function.

[2316] Terminal: Sends user text input to the server.

[2317] Server: Passes the received text to a natural language processing engine for content analysis. Next, an emotion engine is used to recognize emotions from the user's input. Based on this emotion data, a generative AI generates an appropriate response. The response is then sent back from the server to the device.

[2318] On the device: The returned text response is passed to a speech synthesis engine to generate speech and play it back to the user.

[2319] Specific examples

[2320] Morning greetings and schedule check

[2321] User: Say "Good morning."

[2322] Device: Converts the user's speech into text and sends it to the server.

[2323] Server: Analyzes the input text and uses the emotion engine to recognize that the user is in a refreshing mood. It generates a response such as "Good morning! What a lovely morning. Would you like to check today's schedule?" and sends it to the device.

[2324] Terminal: The returned response is converted into speech using a speech synthesis engine and played back to the user. It says, "Good morning! What a beautiful morning. Would you like to check today's schedule?"

[2325] Adding a task

[2326] User: Type "Set a reminder for a meeting tomorrow at 9 AM."

[2327] Terminal: Sends input text to the server.

[2328] Server: Analyzes the text and sets a reminder for the specified date and time. The emotion engine recognizes that the user is a little impatient and generates a confirmation message saying, "I've set a reminder for your meeting tomorrow at 9am. Is there anything else I can help you with?"

[2329] Device: Converts a confirmation message to speech and plays it to the user. Says, "I've set a reminder for your meeting tomorrow at 9:00. Is there anything else I can help you with?"

[2330] Learning and Personalization

[2331] Terminal: Periodically sends the user's interaction history to the server.

[2332] Server: Stores interaction data and emotion data, scans them, and learns user behavior patterns and emotional changes.

[2333] Server: Based on the learning results, the model is updated to generate personalized responses for the next interaction.

[2334] Terminal: Converts the personalized response into speech and provides it to the user.

[2335] In this way, the present invention generates responses based on user input and emotions, providing a natural interaction experience for the user. By responding to the user's preferences and emotions, the interaction becomes more satisfying and the relationship between the user and the character becomes deeper.

[2336] The processing flow will be explained below.

[2337] Step 1:

[2338] Device: The user launches the app, the app initializes, and the login screen appears.

[2339] Step 2:

[2340] User: Enter your existing account information and click the login button.

[2341] Step 3:

[2342] Terminal: Encodes the entered authentication information and sends it to the server.

[2343] Step 4:

[2344] Server: Receives the authentication information, checks it against a database, and if authentication is successful, starts the session and sends the information back to the device.

[2345] Step 5:

[2346] Terminal: Checks the authentication status, and displays the main screen if successful, or an error message if unsuccessful.

[2347] Step 6:

[2348] Terminal: On the main screen, sends a request to the server to display a list of characters available to the user.

[2349] Step 7:

[2350] Server: Retrieves available character information from the character database and returns it to the device.

[2351] Step 8:

[2352] Terminal: Displays a list of characters the user can choose from.

[2353] Step 9:

[2354] User: Select the desired character from the list and open the detailed settings screen.

[2355] Step 10:

[2356] User: Customize the character's personality, voice pitch, specific expressions, etc., and press the button to save the settings.

[2357] Step 11:

[2358] Device: Sends customized configuration data to the server.

[2359] Step 12:

[2360] Server: Stores the received configuration data with the user's profile.

[2361] Step 13:

[2362] User: Speaks to the character via text or voice.

[2363] Step 14:

[2364] Device: In the case of voice input, the voice recognition function converts the input into text and sends the text to the server.

[2365] Step 15:

[2366] Server: Passes the received text to a natural language processing engine and analyzes its content.

[2367] Step 16:

[2368] Server: Sends the parsed text to the emotion engine to recognize the user's emotion.

[2369] Step 17:

[2370] Server: Using the user's emotional data recognized by the emotion engine, the generative AI generates personalized responses.

[2371] Step 18:

[2372] Server: Sends the generated response to the terminal.

[2373] Step 19:

[2374] Terminal: The returned response sentence is passed to the speech synthesis engine to generate speech.

[2375] Step 20:

[2376] Terminal: Plays the generated audio and provides responses to the user.

[2377] Step 21:

[2378] User: Set a new task, for example, "Set a reminder for a meeting tomorrow at 9 AM."

[2379] Step 22:

[2380] Terminal: Sends a request to the server for task configuration.

[2381] Step 23:

[2382] Server: Analyzes received tasks and adds reminders to the task list at the set date and time.

[2383] Step 24:

[2384] Server: Generates a confirmation message for adding the task and sends it to the device.

[2385] Step 25:

[2386] Terminal: Converts the confirmation message into speech and provides it to the user.

[2387] Step 26:

[2388] Terminal: Periodically transmits the user's interaction history and emotion data to the server.

[2389] Step 27:

[2390] Server: Stores interaction and emotion data and associates it with user profiles.

[2391] Step 28:

[2392] Server: Analyzes the stored data and learns changes in user behavior patterns and emotions.

[2393] Step 29:

[2394] Server: Based on the learning results, generates personalized responses for the next interaction and updates the model.

[2395] Step 30:

[2396] Terminal: Converts the personalized response into speech and provides it to the user.

[2397] Example 2

[2398] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2399] Conventional dialogue systems simply respond to user input, making it difficult to recognize emotions in real time or generate personalized responses. Furthermore, they lack the ability to use the user's interaction history to improve future responses, resulting in low user satisfaction. Furthermore, limited options for customizing character personalities and settings make it difficult to provide a compelling experience for users.

[2400] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2401] In this invention, the server includes means for receiving voice or text input from a user, means for converting the user's voice input into text, natural language processing means for analyzing the user's input and generating an appropriate response, means for recognizing the user's emotions, and means for converting the generated response into voice and providing it to the user, thereby enabling recognition of the user's emotions in real time and generating personalized responses.

[2402] "Means for receiving voice or text input from a user" refers to a device that has an interface through which a user can provide input in the form of voice or text.

[2403] "Means for converting user voice input into text" refers to a device that uses voice recognition technology to convert speech into text form.

[2404] "Natural language processing means for analyzing user input and generating an appropriate response" refers to a device that uses natural language processing technology to analyze user input, understand the context and intent, and generate an appropriate response.

[2405] "Means for recognizing user emotions" refers to a device that uses analytical technology to extract and identify emotions from a user's voice or text.

[2406] The "means for converting the generated response into speech and providing it to the user" refers to a device that uses speech synthesis technology to convert a text response into speech and provide it to the user.

[2407] "Means for storing user profiles and behavioral history" refers to a device equipped with a database that stores users' personal information and past interaction data.

[2408] "Means for analyzing stored profiles and behavioral history to generate responses based on user preferences" refers to a device that uses technology to analyze stored data and generate individually optimized responses based on a user's preferences and past behavioral patterns.

[2409] "Means for storing character information and providing a character based on a user's selection" refers to a device that has a database and an interface for storing information on multiple characters and providing data on a character selected by a user.

[2410] "Means for customizing character personality and voice settings" refers to a device that uses an interface that allows a user to adjust or change a character's characteristics or voice features, and technology that stores those settings.

[2411] The present invention is a system for realizing natural interaction between a user and an interactive character, and is implemented by combining the following hardware and software.

[2412] System Overview

[2413] User authentication / login

[2414] Device:

[2415] The user launches a dedicated app and logs in by entering their username and password. When the user presses the "Login" button, the device encrypts the input information and sends it to the server. The hardware used is a typical computer device such as a smartphone, tablet, or PC. The software uses an encryption library (e.g., OpenSSL).

[2416] server:

[2417] The server compares the received authentication information with a database (e.g., MySQL), and if authentication is successful, it starts a session and returns the information to the terminal. The server software used can be a web server (e.g., Apache, Nginx) or a database server.

[2418] Device:

[2419] The device checks the received session token and displays the main screen if authentication is successful, or an error message if authentication is unsuccessful.The software used is a mobile app development framework (e.g., React Native, Flutter).

[2420] Character Selection and Settings

[2421] Device:

[2422] Send a request to the server to display the list of characters available to the user on the main screen, using an HTTP request.

[2423] server:

[2424] The server retrieves available character information from a character database (e.g., MongoDB) and returns it to the device. The returned data is in JSON format.

[2425] Device:

[2426] The device displays the received character information on the main screen. The user selects a character on the screen and customizes their personality, voice pitch, etc. on the detailed settings screen. The customized settings are then sent back to the server in JSON format.

[2427] server:

[2428] The server stores the received configuration data in a database together with the user's profile.

[2429] Interaction Generation and Emotion Recognition

[2430] User:

[2431] The user can input text or voice to the character. Voice input is converted to text using the device's voice recognition function (e.g., Google Speech-to-Text).

[2432] Device:

[2433] The device sends the user's text input to the server, using HTTP or WebSockets.

[2434] server:

[2435] The server passes the received text to a natural language processing engine (e.g., spaCy, Hugging Face Transformers) for content analysis. The server also uses an emotion recognition engine (e.g., a BERT-based model) to recognize the user's emotions. Based on this emotion data, the server generates an appropriate response using a generative AI model (e.g., OpenAI GPT-4) and sends it back to the device.

[2436] Device:

[2437] The device passes the returned text response to a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio and play it back to the user.

[2438] Specific examples

[2439] Morning greetings and schedule check

[2440] User:

[2441] Say "Good morning."

[2442] Device:

[2443] The user's speech is converted into text and sent to the server.

[2444] server:

[2445] The input text is analyzed using a natural language processing engine, and the emotion engine is used to recognize that the user is in a refreshing mood. The response "Good morning! What a lovely morning. Would you like to check today's schedule?" is generated and sent to the device.

[2446] Device:

[2447] The returned response is converted into speech by a speech synthesis engine and played back to the user.

[2448] Example prompts to input to a generative AI model:

[2449] "A user says good morning. He seems very refreshed. How would you respond?"

[2450] Adding a task

[2451] User:

[2452] Type "Set a meeting reminder for tomorrow at 9 AM."

[2453] Device:

[2454] Sends the input text to the server.

[2455] server:

[2456] It analyzes the text and sets a reminder for the specified date and time. The emotion engine recognizes that the user is a little impatient and generates a confirmation message saying, "I've set a reminder for your meeting tomorrow at 9:00. Is there anything else I can help you with?"

[2457] Device:

[2458] A confirmation message is converted into audio and played to the user.

[2459] Example prompts to input to a generative AI model:

[2460] "A user types that they want to set a reminder for a meeting tomorrow at 9 AM. They seem a little impatient. How would you respond?"

[2461] Learning and Personalization

[2462] Device:

[2463] The user's interaction history is periodically sent to the server.

[2464] server:

[2465] It stores interaction and emotion data, analyzes them, and learns user behavior patterns and emotional changes. Based on the learning results, it updates the model to generate personalized responses for the next interaction.

[2466] Device:

[2467] The personalized response is converted into speech and provided to the user.

[2468] In this way, the present invention generates responses based on user input and emotions, providing a natural interaction experience for the user. By responding to the user's preferences and emotions, the interaction becomes more satisfying and the relationship between the user and the character becomes deeper.

[2469] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2470] An explanation of the processing flow of this system's program, broken down into processing steps

[2471] Step 1:

[2472] User Login

[2473] Input: The user launches the dedicated app and enters their username and password.

[2474] Specific operation: When the user presses the "Login" button, the terminal encrypts the username and password and sends them to the server.

[2475] Output: Encrypted user credentials are sent to the server.

[2476] Step 2:

[2477] Authentication verification

[2478] Input: Encrypted user credentials

[2479] What happens: The server checks the received credentials against its database. It performs a database query to verify that the username and password match.

[2480] Output: Generates a result indicating whether authentication was successful or not, and sends it back to the terminal along with the session information.

[2481] Step 3:

[2482] Viewing the login results

[2483] Input: Authentication result and session information

[2484] Specific operation: The device checks the received authentication result, and displays the main screen if successful, or an error message if unsuccessful.

[2485] Output: Display of the main screen or error message.

[2486] Step 4:

[2487] Character Information Request

[2488] Input: Request required to display the main screen

[2489] Specific operation: The device sends a request to the server to obtain available character information.

[2490] Output: Sends a request to the server to get character data.

[2491] Step 5:

[2492] Acquiring and returning character information

[2493] Input: Character data acquisition request

[2494] Specific operation: The server retrieves character information from the character database and returns it to the device in JSON format.

[2495] Output: Returns character information to the terminal.

[2496] Step 6:

[2497] Displaying the character list

[2498] Input: Character information in JSON format

[2499] Specific operation: The device analyzes character information and displays it in a list on the main screen.

[2500] Output: Display the character list on the main screen.

[2501] Step 7:

[2502] Character selection and detailed settings

[2503] Input: User-selected character and customization settings

[2504] Specific operation: The user selects a character and customizes the character's personality, voice pitch, etc. in the detailed settings screen. These settings are sent from the device to the server.

[2505] Output: Sends customization information to the server.

[2506] Step 8:

[2507] Saving the configuration information

[2508] Input: Customization setting information

[2509] Specific operation: The server stores the received customization setting information in the database.

[2510] Output: Save customized settings information to a database.

[2511] Step 9:

[2512] User Input and Conversion

[2513] Input: User voice or text input

[2514] Specific operation: The user inputs to the character by voice or text. In the case of voice input, the device converts the voice into text in real time.

[2515] Output: The converted text data is sent to the server.

[2516] Step 10:

[2517] Natural Language Processing and Emotion Recognition

[2518] Input: Text data

[2519] Specific operation: The server passes the text data to a natural language processing engine for analysis, and recognizes emotions using an emotion engine.

[2520] Output: Analysis results and emotion data.

[2521] Step 11:

[2522] Generate a response

[2523] Input: Analysis results and emotion data

[2524] Specific operation: The server uses a generative AI model (e.g., OpenAI GPT-4) to generate an appropriate response.

[2525] Output: Sends the generated response to the terminal.

[2526] Step 12:

[2527] Audio generation and playback

[2528] Input: Generated response

[2529] Specific operation: The device passes the response sentence to a speech synthesis engine to generate speech and play it back to the user.

[2530] Output: Playback of the generated audio.

[2531] Example prompt

[2532] 1. Morning greeting and schedule prompt:

[2533] "A user says good morning. He seems very refreshed. How would you respond?"

[2534] 2. Prompt for adding a task:

[2535] "A user types that they want to set a reminder for a meeting tomorrow at 9 AM. They seem a little impatient. How would you respond?"

[2536] (Application example 2)

[2537] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2538] Conventional interaction systems have fixed responses to user inputs and have not been able to realize natural dialogue that takes into account the user's emotions and gaze. Furthermore, in the work environment, there is a lack of efficient and safe work support, which has led to the issue of workers being unable to quickly obtain the information they need.

[2539] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving input from a user, means for analyzing the user's input and generating an appropriate response, means for converting the generated response into voice and providing it to the user, means for recognizing the user's emotions, and means for tracking the user's gaze. This enables natural dialogue that takes the user's emotions and gaze into consideration. The server also includes means for saving a user's profile and behavioral history, means for analyzing the saved profile and behavioral history and generating a response based on the user's preferences, means for visually providing the generated response, and means for displaying information via a visual output device worn by the user. This enables workers to efficiently and safely obtain necessary information in the work environment and support them. The server also includes means for saving character information and providing a character based on the user's selection, means for customizing the character's personality and voice settings, and means for creating appropriate prompt sentences based on a generative AI model. This allows for more personalized responses to the user and an improved dialogue experience.

[2540] "Means for receiving input from a user" refers to hardware and software for capturing and transmitting information entered by a user through speech or text to the system.

[2541] "Means for analyzing user input and generating an appropriate response" refers to a technology that analyzes received user input information using a natural language processing engine or the like and generates a response based on that information.

[2542] The "means for converting the generated response into speech and providing it to the user" is a technology that uses a speech synthesis engine to convert the generated text response into speech and plays it aloud to the user through a speaker or the like.

[2543] "Means for recognizing user emotions" refers to technology that analyzes emotions from the user's voice, text, facial expressions, etc., and determines the user's current feelings and state.

[2544] "Means for tracking the user's gaze" refers to technology that uses a camera or sensor to track the direction or object the user is looking at and records it as gaze information.

[2545] "Means for storing a user's profile and behavior history" refers to technology for storing a user's personal information, past operation history, behavior patterns, etc. in a storage device.

[2546] "Means for analyzing stored profiles and behavioral history and generating responses based on the user's preferences" refers to technology that analyzes stored data and generates personalized responses tailored to the user's tendencies and preferences.

[2547] The "means for visually presenting the generated response" is a technique for displaying the generated response on a display device as visual information such as text, images, or videos.

[2548] "Means for displaying information via a visual output device worn by a user" refers to technology for displaying information through a device worn by a user, such as smart glasses or a head-mounted display.

[2549] "Means for saving character information and providing characters based on user selection" refers to technology that saves character attributes and settings in a database and provides the character on screen or via audio based on the user's selection.

[2550] "Means for customizing character personality and voice settings" refers to technology that allows a user to change the personality, voice pitch, tone, etc. of a character selected by the user to suit the user's preferences.

[2551] "Means for creating appropriate prompt sentences based on a generative AI model" refers to a technology that uses an AI model to generate optimal response sentences based on input from a user.

[2552] The present invention provides a system that supports factory work by enabling natural interactions with input from users. This system allows characters to interact with the user through smart glasses worn by the user, improving work efficiency and safety. Detailed embodiments for implementing the present invention are described below.

[2553] System Overview

[2554] The system consists of the following main components:

[2555] 1. A means of receiving input from the user

[2556] Hardware: Smart glasses (e.g., Google Glass)

[2557] Process: Capture the user's voice and gaze data and send it to the system.

[2558] 2. A means of analyzing user input and generating an appropriate response

[2559] Software: Speech recognition software (e.g., Google Cloud Speech-to-Text), natural language processing engines (e.g., OpenAI's GPT-4)

[2560] Process: Convert the captured audio into text, analyze the text and generate an appropriate response.

[2561] 3. A means of converting the generated response into speech and providing it to the user

[2562] Software: Speech synthesis engine (e.g. Amazon Polly)

[2563] Process: The generated text is converted into speech and provided to the user through the smart glasses speaker.

[2564] 4. How to Recognize User Emotions

[2565] Software: Emotion recognition engine (e.g., Microsoft Azure Emotion API)

[2566] Process: Analyze the user's emotions from captured voice and gaze data and provide feedback to the system.

[2567] 5. A way to track the user's gaze

[2568] Hardware: Built-in camera in smart glasses

[2569] Process: Track the user's gaze in real time and analyze the gaze data.

[2570] 6. Means of storing user profiles and activity histories

[2571] Hardware: Cloud-based database

[2572] Pro...

Claims

1. means for receiving input from a user; means for parsing user input and generating an appropriate response; means for converting the generated response into speech and providing it to the user; A system including:

2. a means for storing a user's profile and behavior history; means for analyzing the stored profile and behavioral history to generate a response based on the user's preferences; The system of claim 1 further comprising:

3. means for storing character information and providing characters based on user selection; A way to customize your character's personality and voice settings, The system of claim 1 further comprising:

Images