System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting contents of a consultation from a user; means for transmitting the inputted contents of the consultation to a cloud server; means for analyzing the contents of the consultation on the cloud server and passing the contents of the consultation to a generative AI model as an input; means for generating advice or a proposal based on the contents of the consultation by the generative AI model; and means for providing the generated advice to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Elderly people have a wide range of physical anxieties and worries about daily life, but there are limited ways to receive appropriate support. In addition, with the increase in single-person households, there are fewer opportunities to easily find someone to talk to or share entertainment, and they often feel mentally isolated and lonely. To solve this situation, a system is needed that can efficiently solve the problems that elderly people face in their daily lives and also provide mental support. [Means for solving the problem]

[0005] The present invention is a system that includes a means for inputting consultation details from a user, a means for transmitting the input consultation details to a cloud server, a means for analyzing the consultation details on the cloud server and passing them as input to a generative AI model, a means for the generative AI model to generate advice and suggestions based on the consultation details, and a means for providing the generated advice to the user. This system can effectively resolve physical anxieties and lifestyle concerns that elderly people have.

[0006] Furthermore, by including a means for the cloud server to generate content for a conversation partner mode or a game mode based on the user's selection and provide it to the user, psychological support can also be provided. Also, by including a means for the user to set the AI's characteristics and direction, and for the cloud server to automatically adjust the responses and suggestions of the generated AI model based on the settings, optimal services tailored to individual users can be provided.

[0007] In addition, the system includes a means for the user to select additional features and for the cloud server to process the payment based on that information, and a means for applying the additional features to the user account when the payment is successful, thereby enabling the continuous provision of services while establishing a revenue model.

[0008] "Users" refer to elderly people and individuals living alone who use the system.

[0009] "Consultation content" refers to information entered by the user, such as physical anxieties or worries about daily life.

[0010] "Cloud server" refers to a computer system that is remotely managed and operated over the Internet.

[0011] A "generative AI model" refers to an artificial intelligence algorithm that generates appropriate advice, suggestions, and conversation content based on user input.

[0012] "Advice" refers to specific solutions or suggestions that the generative AI model provides based on the user's inquiry.

[0013] "Conversation partner mode" refers to the function in which the cloud server uses the generated AI model to communicate with the user.

[0014] "Game mode" refers to game content provided by the cloud server for users to enjoy entertainment.

[0015] "Training settings" refers to the setting information that the user inputs to adjust the characteristics and direction of the generated AI model.

[0016] "Additional Features" refers to new features and services within the system that are made available to users through a fee.

[0017] "Payment processing" refers to the payment processing carried out by the cloud server for the user to use additional functions.

[0018] "Account" refers to a database used to manage a user's personal information and usage history within the system. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention provides a system that allows users to use an application linked to a cloud server to seek advice about health conditions or lifestyle concerns or to enjoy entertainment. Specific embodiments of this system will be described below.

[0041] Process to receive consultation details from the user

[0042] First, the user launches the application on their smartphone or tablet and inputs their consultation details by voice or text. The application then sends the input to a cloud server, which analyzes the received data, converts it into an appropriate format, and passes it to the generative AI model.

[0043] AI responds to inquiries

[0044] On the cloud server, the generative AI model analyzes the consultation content received from the user and generates appropriate advice and suggestions. The generated advice is then sent back from the cloud server to the user's device and provided to the user via an application. This process occurs in real time, allowing users to receive advice quickly.

[0045] Providing daily conversation partners and entertainment

[0046] When a user selects "conversation partner mode" or "game mode" in an application, the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and sends it to the user's device. In "conversation partner mode," the generative AI model generates conversation content with the user and displays it in the application. In "game mode," game content is provided that can be enjoyed together with the user.

[0047] AI training function

[0048] Users can set the AI's characteristics and direction through the application's training screen. The training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. This allows users to receive personalized responses tailored to their preferences.

[0049] Revenue model and billing process

[0050] This system also provides a means for users to select additional features and for the cloud server to process billing based on that information. The user selects the additional features on a billing page within the application, and that information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[0051] Specific examples

[0052] For example, if User A complains of a sore lower back, the cloud server passes the information to the generative AI model. The generative AI model generates advice such as "Try some moderate stretching. Specifically, try these movements," and sends it to User A's device. Similarly, if User B thinks, "I'm feeling a little lonely today," and selects "Conversation mode," the generative AI model generates conversation content such as "What did you do today?" and provides it to User B.

[0053] In this way, the system of the present invention provides a concrete means for elderly users and single-person household users to lead richer daily lives.

[0054] The processing flow will be explained below.

[0055] Process to receive consultation details from the user

[0056] Step 1:

[0057] The user launches the app on their smartphone and inputs the details of their consultation either by voice or text.

[0058] Step 2:

[0059] The terminal converts the input consultation content into JSON format and sends an HTTP POST request to the cloud server.

[0060] AI responds to inquiries

[0061] Step 3:

[0062] The server parses the received JSON data of the consultation content and converts it into a specified format.

[0063] Step 4:

[0064] The server passes the consultation content after format conversion as input to the generative AI model.

[0065] Step 5:

[0066] A generative AI model generates advice and suggestions based on the consultation content.

[0067] Step 6:

[0068] The server converts the generated advice into JSON format and sends it to the user's device.

[0069] Step 7:

[0070] The device parses the received advice and displays it to the user within the application, and may also use speech synthesis to communicate it aloud if necessary.

[0071] Providing daily conversation partners and entertainment

[0072] Step 1:

[0073] The user selects either "talkie mode" or "game mode" in the application.

[0074] Step 2:

[0075] The device converts the selected mode information into JSON format and sends it to the cloud server via an HTTP POST request.

[0076] Step 3:

[0077] The server invokes a generative AI model or a game content generation system based on the received mode information.

[0078] Step 4:

[0079] In the conversation partner mode, the server uses the generative AI model to generate appropriate conversation content and send it to the user's device.In the game mode, the server generates game content and sends it to the user's device.

[0080] Step 5:

[0081] In the conversation partner mode, the terminal displays the generated conversation content, and in the game mode, the terminal starts a game so that the user can play together.

[0082] AI training function

[0083] Step 1:

[0084] Users set the AI's characteristics and direction through the application's development screen.

[0085] Step 2:

[0086] The device converts the development setting information into JSON format and sends it to the cloud server via an HTTP POST request.

[0087] Step 3:

[0088] The server automatically adjusts the response and proposal content of the generated AI model based on the received training setting information.

[0089] Revenue model and billing process

[0090] Step 1:

[0091] The user selects additional features from the application's billing page.

[0092] Step 2:

[0093] The device converts the purchase information into JSON format and sends it to the cloud server via an HTTP POST request.

[0094] Step 3:

[0095] The server calls a payment gateway based on the received purchase information and executes the payment process.

[0096] Step 4:

[0097] If the payment is successful, the server applies the additional function to the user's account and notifies the user's terminal of that information.

[0098] Step 5:

[0099] Based on the received information about the additional function, the terminal performs settings so that the user can use the function within the application.

[0100] Example 1

[0101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] In modern life, people are looking for fast and effective solutions to their health and daily worries. Furthermore, with the increase in elderly people and single-person households, there is a need for mental health care and entertainment for those living alone. Systems that provide fast and personalized responses to these issues are lacking, and technological solutions are needed to meet user needs.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0104] In this invention, the server includes means for inputting consultation details from a user, means for transmitting the input consultation details to a cloud server, means for the cloud server to analyze the received data, convert it into an appropriate format, and pass it to the generative AI model, means for the generative AI model to generate advice and suggestions based on the consultation details, and means for transmitting the generated advice again from the cloud server to the user's terminal and providing it to the user through an application. This enables users to receive quick and personalized advice and suggestions in real time, providing a powerful means for elderly people and single-person households to live richer daily lives.

[0105] "User" refers to a person who uses this system to input consultation details, select modes, and set up AI training.

[0106] "Consultation content" refers to information entered by the user regarding their health condition or concerns about their daily life.

[0107] A "cloud server" refers to a server that performs data analysis, sends data to generative AI models, and receives responses via the Internet.

[0108] A "generative AI model" refers to an artificial intelligence model that generates advice and suggestions based on the input consultation content.

[0109] "Application" refers to software that runs on the user's smartphone or tablet and allows them to input consultation details, select modes, and set up AI training.

[0110] "Conversation partner mode" refers to a mode in which the cloud server uses a generated AI model to provide the content of the conversation to the user.

[0111] "Game mode" refers to a mode in which the cloud server uses a generated AI model to provide users with entertainment content such as games and quizzes.

[0112] "Setting the AI's characteristics and direction" refers to the user making settings through the application to customize the responses and suggestions of the generated AI model.

[0113] "Training setting information" refers to information about the characteristics and direction of the AI ​​set by the user.

[0114] "Payment gateway" refers to an Internet service that allows a server to process payments based on billing information from users.

[0115] This invention provides a system that allows users to use an application linked to a cloud server to seek advice about health conditions or lifestyle concerns or to enjoy entertainment. Specific embodiments of this system will be described below.

[0116] Required Hardware and Software

[0117] User device: A computing device such as a smartphone or tablet.

[0118] Application: Software installed on a device that provides a user interface.

[0119] Cloud server: A server that performs data analysis and runs generative AI models via the internet.

[0120] Generative AI model: An artificial intelligence model that generates advice and suggestions based on the content of a consultation.

[0121] System Operation

[0122] First, the user launches the application on their smartphone or tablet and inputs the content of their consultation by voice or text. At this time, the voice data input by the user is converted into text data within the application. The application then sends the input content of the consultation to a cloud server. The cloud server analyzes the received data and converts it into a format that is easy for the generative AI model to understand.

[0123] The generative AI model on the cloud server analyzes the consultation content received from the user and generates appropriate advice and suggestions. The generated advice is sent from the cloud server to the user's device and provided to the user through an application. This process is carried out in real time, allowing users to receive advice quickly.

[0124] For example, if a user enters the text "My lower back hurts" and taps the send button, the device sends the input data to the cloud server. The cloud server analyzes the received data and passes it to the generative AI model. The generative AI model generates advice such as "Try some moderate stretching. Specifically, try these movements," and the cloud server sends the generated advice to the user's device. The device decodes the received advice and displays it on the screen. The user then checks the advice displayed on the screen.

[0125] Furthermore, when the user selects "conversation partner mode" or "game mode," the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and sends it to the user's device. In "conversation partner mode," the generative AI model generates conversation content with the user and displays it in the application. In "game mode," game content that can be enjoyed together with the user is provided.

[0126] Furthermore, users can set the AI's characteristics and direction through the application's training screen. The training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. This allows users to receive personalized responses tailored to their preferences.

[0127] As a revenue model, a method is also provided in which the user selects additional features and the cloud server processes billing based on that information. The user selects the additional features on the billing page within the application and the information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[0128] Examples of prompt statements

[0129] Consultation prompt:

[0130] Tell the AI ​​the following questions to generate appropriate advice:

[0131] Consultation: "My lower back hurts"

[0132] Interactive mode input prompt:

[0133] Continue the conversation with the user as follows:

[0134] Conversation topic: "I'm feeling a little lonely today"

[0135] Game mode input prompt:

[0136] Give your users a fun and easy quiz.

[0137] Theme: "General Knowledge"

[0138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0139] Step 1:

[0140] The user launches the application on their smartphone or tablet and inputs the content of their consultation by voice or text. The input content is converted into text data by the application. After the user's voice input is converted into text, text data is generated. This is the output of Step 1.

[0141] Step 2:

[0142] The device transmits the text data, which is the output of step 1, to the cloud server. The transmitted text data is received by the cloud server. The data received by the cloud server is the input of step 2.

[0143] Step 3:

[0144] The server analyzes the received text data (input in step 2) and converts it into a format that is easy for the generative AI model to understand. For example, it converts it into JSON format. This format conversion is performed as data processing, and as a result, prepared data is generated to be passed to the generative AI model. This is the output of step 3.

[0145] Step 4:

[0146] The server passes the format-converted data, which is the output of step 3, to the generative AI model. The passed data becomes the input for the generative AI model, and at this stage the generative AI model begins analysis.

[0147] Step 5:

[0148] The generative AI model analyzes the format-converted data, which is the input in step 4, and generates appropriate advice and suggestions. Here, advice generated based on the prompt sentence is output. For example, for the input "My lower back hurts," specific advice such as "Try some moderate stretching" is generated.

[0149] Step 6:

[0150] The server sends the generated advice, which is the output of step 5, to the user's device. At this stage, the data is again encoded and transmitted securely using the appropriate communication protocol. The transmitted data is the input here.

[0151] Step 7:

[0152] The terminal receives and decodes the generated advice, which is the input of step 6. It displays the received advice to the user. This decoded advice is the output of step 7, and the user sees the information on the terminal screen.

[0153] Step 8:

[0154] The user selects "conversation mode" or "game mode" within the application. The selected mode information is input.

[0155] Step 9:

[0156] The terminal transmits mode information to the cloud server, which is the input of step 8. The transmitted mode information is received by the cloud server. The received mode information is the output of step 9.

[0157] Step 10:

[0158] The server generates appropriate content using the generative AI model based on the received mode information, which is the output of step 9. For example, in "conversation partner mode," it generates appropriate conversation content for the user, and in "game mode," it generates games and quizzes that can be enjoyed together with the user. The generated content is the output of step 10.

[0159] Step 11:

[0160] The server transmits the generated content, which is the output of step 10, to the user's terminal. The transmitted content becomes the input of step 11.

[0161] Step 12:

[0162] The terminal receives the generated content, which is the input of step 11, and displays it on the screen. This displayed content is the output of step 12, and the user can view, manipulate, and enjoy it.

[0163] Step 13:

[0164] Users can set the AI's characteristics and direction through the application's training screen, and this information is used as input.

[0165] Step 14:

[0166] The terminal transmits the development setting information, which is the input of step 13, to the cloud server. The transmitted development setting information is the output of step 14.

[0167] Step 15:

[0168] The server automatically adjusts the response and proposal content of the generated AI model based on the training setting information output in step 14. These adjusted response and proposal content are the output of step 15.

[0169] Step 16:

[0170] The user selects an additional feature on the billing page within the application, and the selected billing information is input.

[0171] Step 17:

[0172] The terminal transmits the billing information, which is the input of step 16, to the cloud server. The transmitted billing information is the output of step 17.

[0173] Step 18:

[0174] The server receives the billing information output from step 17 and passes it to the payment gateway for payment processing. If the payment is successful, it applies additional features to the user account. The applied additional features are output from step 18.

[0175] Step 19:

[0176] The terminal displays the applied additional function, which is the output of step 18, to the user and makes it available for use. The user can use the new function through the terminal application.

[0177] (Application example 1)

[0178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] Maintaining a healthy diet and managing your health can be challenging in today's hectic lifestyles. Getting accurate advice based on your health status and individual dietary preferences is particularly challenging, and there are no systems that can directly order meals. Conventional food delivery systems simply deliver the meals selected by the user, but are unable to provide suggestions or advice based on the user's health status. Furthermore, there is a lack of systems that can quickly provide appropriate advice and even order meals on the spot.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0181] In this invention, the server includes means for inputting consultation details from a user, means for transmitting the input consultation details to a cloud server, means for analyzing the consultation details on the cloud server and passing them as input to a generative AI model, means for the generative AI model to generate advice and suggestions based on the consultation details, means for providing the generated advice to the user, means for generating meal suggestions based on health status, and means for ordering the suggested meals from a food delivery system. This enables the user to quickly receive appropriate advice based on their health status and then order the suggested meals from a food delivery system.

[0182] A "user" is someone who uses this system to input their consultation details and receive health advice and dietary suggestions.

[0183] The "cloud server" is a remote server that receives and analyzes the consultation content sent by users via the Internet and generates advice and suggestions using a generative AI model.

[0184] "Consultation content" refers to worries, questions, and requests about health and diet entered by the user.

[0185] A "generative AI model" is an artificial intelligence (AI) model that runs on a cloud server and generates advice and suggestions based on the input consultation content.

[0186] "Advice and suggestions" are specific recommendations and instructions generated by the generative AI model based on the user's consultation.

[0187] "Health status" refers to information about the user's physical condition, food preferences, and nutritional balance.

[0188] "Meal suggestions" are specific meal menu suggestions recommended by the generative AI model based on the user's health status.

[0189] A "food delivery system" is an online service that allows users to order and have food delivered from a specified menu.

[0190] This system allows users to consult about their health and diet using devices such as smartphones and tablets, and receives appropriate advice and suggestions in cooperation with a cloud server. Furthermore, based on the suggestions, users can order meals via a food delivery system.

[0191] Overall system configuration

[0192] The system mainly consists of the following components:

[0193] 1. User Device

[0194] 2. Cloud Server

[0195] 3. Generative AI Models

[0196] 4. Food delivery system

[0197] Operation on the user device

[0198] Users start up a dedicated application on their smartphone or tablet and first input their health and dietary concerns by voice or text. This input is then sent from the device to a cloud server.

[0199] Analysis on a cloud server

[0200] The cloud server analyzes the consultation content received from the user, converts it into an appropriate format, and passes it to a generative AI model. For example, OpenAI's GPT-3 (generative AI model) is used as the generative AI model. This model generates advice and suggestions based on the user's consultation content.

[0201] Generate advice and suggestions

[0202] The generative AI model analyzes the consultation content sent from the cloud server and generates appropriate advice and suggestions. For example, it may suggest an appropriate diet based on the user's health condition. This generated advice and suggestions are then sent back to the user's device via the cloud server.

[0203] Meal suggestions based on health status

[0204] The cloud server generates meal suggestions appropriate for the user's health condition based on the advice generated by the generative AI model. For example, the advice generated might be, "Try to eat a balanced meal centered around vegetables today."

[0205] Ordering from a food delivery system

[0206] The user terminal transmits a meal order to the food delivery system based on the suggestions, which includes the meal menu, delivery address, payment method, etc. specified by the user.

[0207] Specific examples

[0208] For example, if a user types into their device, "What is the best meal for my health today?", the cloud server passes this information to the generative AI model. The generative AI model generates advice such as, "Try to eat a balanced meal with a focus on vegetables today." The cloud server then uses this advice to suggest a "healthy plate of grilled vegetables and chicken" and orders this meal through a food delivery system.

[0209] Prompt Sentence Examples

[0210] "Provide appropriate meal suggestions based on the user's health status. Advice: 'Aim for a balanced meal with a focus on vegetables today.'"

[0211] As described above, the present invention is a system that supports the user's health management and promptly suggests and delivers appropriate meals.

[0212] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0213] Step 1:

[0214] The user launches a dedicated application on their smartphone or tablet and inputs the details of their consultation by voice or text. The inputted details are sent from the device to the cloud server. The inputted data is sent in the form of text data.

[0215] Step 2:

[0216] The cloud server analyzes the received consultation content and converts it into an appropriate format. Specifically, it uses natural language processing to analyze the text data and converts it into a format that can be understood by a generative AI model (e.g., OpenAI's GPT-3). This process involves extracting the intent and keywords of the consultation content. The input is the text data of the user's consultation content, and the output is data formatted for the generative AI model.

[0217] Step 3:

[0218] A generative AI model on a cloud server generates advice and suggestions based on the formatted consultation content. The generative AI model uses an internal algorithm to analyze the user's consultation content and construct optimal advice and suggestions. In this process, the advice is generated using prompt text. The input is the formatted consultation content data, and the output is the generated advice or suggestion text.

[0219] Step 4:

[0220] The generated advice and suggestions are then sent to the user's device via the cloud server. The cloud server converts the output data from the generative AI model into a format that is easy for the user to understand and sends it. The input is the advice and suggestion text generated by the generative AI model, and the output is the advice text sent to the user's device.

[0221] Step 5:

[0222] The user device displays the advice and suggestions received from the cloud server. The user can check this through the application. Specifically, the received text data is displayed on the screen. The input is the advice text from the cloud server, and the output is the advice content displayed on the device.

[0223] Step 6:

[0224] The cloud server generates meal suggestions suitable for the user's health condition based on the advice created by the generative AI model. Specifically, it generates prompt sentences based on the generated advice and makes food suggestions. The input is text data of the advice content, and the output is text data of the meal suggestions.

[0225] Step 7:

[0226] The user terminal receives the meal suggestions sent from the cloud server and sends the order directly to the food delivery system. Specifically, it generates order information including the suggested meal menu, delivery address, and payment method and sends it to the food delivery system. The input is text data of the meal suggestions, and the output is order data sent to the food delivery system.

[0227] Through the above steps, the user can receive appropriate advice for health management and order meals based on that advice.

[0228] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0229] This invention is a system that combines a generative AI model and an emotion engine to support users' daily lives, provide appropriate advice for consultations, and respond based on emotion recognition. Specific embodiments of this system are described below.

[0230] Process to receive consultation details from the user

[0231] The user launches the application on their smartphone or tablet and inputs their consultation details by voice or text. The device then sends the input consultation details and voice data to a cloud server. The cloud server analyzes the received data, converts it into an appropriate format, and passes it to the generative AI model and emotion engine.

[0232] Emotion recognition and advice generation using an emotion engine

[0233] On the cloud server, the emotion engine recognizes emotions from the consultation content and voice data entered by the user. The recognized emotion information is reflected in the generative AI model, which then generates advice and suggestions based on the user's emotions. The generated advice is then sent back from the cloud server to the user's device and provided to the user via an application.

[0234] Providing daily conversation partners and entertainment

[0235] When a user selects "conversation partner mode" or "game mode" in the application, the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and emotions and sends it to the user's device. In "conversation partner mode," the emotion engine generates and provides conversation content based on the emotions recognized by the emotion engine. In "game mode," the difficulty and content of the game are adjusted according to the user's emotional state.

[0236] AI training function

[0237] Users can set the AI's characteristics and direction through the application's training screen. Training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. By combining emotion engines, the user's emotional state is also taken into account, allowing for the provision of optimal services tailored to each individual user.

[0238] Revenue model and billing process

[0239] This system also provides a means for users to select additional features and for the cloud server to process billing based on that information. The user selects the additional features on a billing page within the application, and that information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[0240] Specific examples

[0241] For example, if User A complains of a "pain in the lower back," the cloud server passes the information to the emotion engine. The emotion engine recognizes "anxiety" from User A's voice data and provides this information to the generative AI model. The generative AI model then generates a message that takes User A's emotions into consideration, such as "Try some moderate stretching. Specifically, try these movements," along with advice such as "Try it slowly, without overdoing it," and sends it to User A's device.

[0242] If User B thinks, "I'm feeling a little lonely today," and selects "Conversational Partner Mode," the emotion engine will recognize this feeling. The generative AI model will start the conversation with something like, "What did you do today?", and then generate content that takes emotion into consideration, such as, "Shall we have a fun conversation?", and provide it to User B.

[0243] In this way, the system of the present invention provides a concrete means for elderly users and single-person household users to lead richer daily lives.

[0244] The processing flow will be explained below.

[0245] Process to receive consultation details from the user

[0246] Step 1:

[0247] The user starts the smartphone app and inputs the details of their consultation. The user can input the details of their consultation by voice or text.

[0248] Step 2:

[0249] The device converts the input consultation content and voice data into JSON format and sends it to the cloud server via an HTTP POST request.

[0250] AI-based consultation processing and emotion recognition

[0251] Step 3:

[0252] The server parses the JSON data of the received consultation content and voice data and converts it into a specified format.

[0253] Step 4:

[0254] The server passes the consultation content after format conversion to the generative AI model and emotion engine.

[0255] Step 5:

[0256] The emotion engine recognizes emotions from the user's voice data and text. For example, emotions such as "anxiety" and "loneliness" can be recognized through voice analysis.

[0257] Step 6:

[0258] The generative AI model generates advice and suggestions based on the recognized emotional information and the content of the consultation. Specifically, in addition to advice such as "Try some moderate stretching. Specifically, try these movements," it generates messages that take the user's emotions into consideration, such as "Try it little by little without overdoing it."

[0259] Step 7:

[0260] The server converts the generated advice and emotion-sensitive messages into JSON format and sends them to the user's device.

[0261] Step 8:

[0262] The device parses the received advice and messages and displays them to the user within the application, and may also use speech synthesis to communicate them aloud if necessary.

[0263] Providing daily conversation partners and entertainment

[0264] Step 1:

[0265] The user selects either "talkie mode" or "game mode" in the application.

[0266] Step 2:

[0267] The device converts the selected mode information into JSON format and sends it to the cloud server via an HTTP POST request.

[0268] Step 3:

[0269] The server invokes a generative AI model and an emotion engine based on the received mode information.

[0270] Step 4:

[0271] The emotion engine recognizes emotions from the user's voice data and text and provides that information to the generative AI model.

[0272] Step 5:

[0273] The generative AI model generates appropriate conversation content in conversation partner mode and appropriate game content in game mode based on emotion information and mode information from the emotion engine.

[0274] Step 6:

[0275] The server converts the generated conversation content or game content into JSON format and sends it to the user's device.

[0276] Step 7:

[0277] In the conversation partner mode, the terminal displays the generated conversation content, and in the game mode, the terminal starts the generated game so that the user can play together with the user.

[0278] AI training function

[0279] Step 1:

[0280] Users set the AI's characteristics and direction through the application's development screen.

[0281] Step 2:

[0282] The device converts the development setting information into JSON format and sends it to the cloud server via an HTTP POST request.

[0283] Step 3:

[0284] The server automatically adjusts the responses and suggestions of the generative AI model based on the received training setting information, taking into account information from the emotion engine.

[0285] Revenue model and billing process

[0286] Step 1:

[0287] The user selects additional features from the application's billing page.

[0288] Step 2:

[0289] The device converts the purchase information into JSON format and sends it to the cloud server via an HTTP POST request.

[0290] Step 3:

[0291] The server calls a payment gateway based on the received purchase information and executes the payment process.

[0292] Step 4:

[0293] If the payment is successful, the server applies the additional features to the user's account and sends the information to the user's device via an HTTP POST request.

[0294] Step 5:

[0295] Based on the received additional function information, the terminal performs settings so that the user can use the function within the application.

[0296] Example 2

[0297] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0298] In modern society, users face various worries and problems in their daily lives, but it is difficult to quickly obtain appropriate advice. Furthermore, there is a lack of appropriate responses that take emotions into consideration, and people with whom people can easily talk, leading to feelings of loneliness. To address these issues, there is a need for a system that can understand users' emotions and provide appropriate advice and entertainment.

[0299] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0300] In this invention, the server includes means for inputting consultation content from a user, means for transmitting the input consultation content to a cloud server, means for analyzing the consultation content on the cloud server, converting it into an appropriate format, and passing it as input to the generative AI model and emotion identification device, means for the emotion identification device to recognize emotions from the consultation content and voice data, means for the generative AI model to generate advice and suggestions based on the recognized emotion information, and means for providing the generated advice to the user from the cloud server. This allows users to receive appropriate advice and entertainment according to their emotions at any given time, thereby improving the quality of their daily lives.

[0301] "User" refers to an individual who uses the system to input their inquiry and receive advice and suggestions.

[0302] "Consultation content" refers to the worries, questions, and other inquiries that the user inputs into the system by voice or text.

[0303] A "cloud server" is a server that can be accessed remotely via the Internet and is a central element of a system that analyzes and processes data.

[0304] A "generative AI model" is a generative model created by artificial intelligence, and refers to a program that creates advice and suggestions based on user input and emotional information.

[0305] An "emotion identification device" refers to a device or program that recognizes emotions from user input or voice data and reflects them in the analysis results.

[0306] "Advice and suggestions" refers to specific advice and recommendations generated by the generative AI model based on the user's consultation content and emotional information.

[0307] The "conversation partner mode" refers to a mode in which the user selects how the system will respond as a conversation partner, and conversation content based on emotions is provided.

[0308] "Play mode" refers to a mode in which the user selects a mode that provides games and entertainment content to the system, and the difficulty and content of the game are adjusted according to the user's emotional state.

[0309] "Setting the characteristics and direction of artificial intelligence" refers to the operation in which the user sets specific parameters through a settings screen within the system to adjust the response and proposal content of the generated AI model.

[0310] "Payment gateway" refers to an interface through which the cloud server processes the user's payment information and performs billing processing.

[0311] This invention is a system that allows users to input their consultation details using an application on a smartphone or tablet and receive appropriate advice and suggestions in response. The system operates on a cloud server using a generative AI model and an emotion recognition device. A specific embodiment of this system is described below.

[0312] The user launches an application installed on their smartphone or tablet and inputs the consultation details by voice or text. The device then sends the input consultation details to a cloud server. This transmission uses a communication protocol (e.g., HTTP, HTTPS) via an internet connection.

[0313] The cloud server parses the received data and converts it into an appropriate format (e.g., JSON format) using, for example, a natural language processing library or speech recognition technology (e.g., Google Speech-to-Text API).The converted data is then passed to a generative AI model and an emotion recognition device.

[0314] The emotion recognition device recognizes emotions from the consultation content and voice data entered by the user. The technology used can be, for example, IBM Watson's Natural Language Understanding. The emotion recognition device provides the recognized emotional information to the generative AI model.

[0315] The generative AI model generates advice and suggestions for the user based on this emotional information. For example, OpenAI's GPT model is used for generation. The generative AI model generates specific suggestions and response messages that reflect the user's emotional state.

[0316] The generated advice and suggestions are sent from the cloud server to the user's device. The device then provides the received advice to the user through an application. The advice is displayed using a visually easy-to-understand interface (e.g., text message, pop-up message).

[0317] For example, if a user complains of a "pain in the lower back," the cloud server passes the information to an emotion recognition device. The emotion recognition device recognizes "anxiety" from the voice data. This information is provided to a generative AI model, which then generates advice such as "Try some moderate stretching. Specifically, try the following movements," along with a message that takes the user's emotions into consideration, such as "Try doing it gradually, without overdoing it." The message is then sent to the user's device.

[0318] Furthermore, if the user selects "conversation partner mode," the emotion recognition device recognizes "loneliness." In this case, the generative AI model generates emotion-sensitive content, such as "What did you do today?" and "Shall we have a little fun conversation?", and provides these to the user.

[0319] An example of a specific prompt is as follows: "If a user complains of lower back pain, what advice would the emotion recognizer generate for the generative AI model by recognizing anxiety from the voice data?"

[0320] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0321] Step 1:

[0322] The user launches the application on their smartphone or tablet. Here, the user inputs the details of their consultation by voice or text. The input data is temporarily stored within the device. An example of input data is the voice input of "My lower back hurts."

[0323] Step 2:

[0324] The device converts the voice data entered by the user into text data. This conversion uses voice recognition technology (e.g., Google Speech-to-Text API). The converted text data becomes "My lower back hurts." This text data is then sent to a cloud server. HTTP or HTTPS is used as the transmission protocol.

[0325] Step 3:

[0326] The server parses the received text data using a natural language processing library (e.g., NLTK). The parsed data is then converted into an appropriate format (e.g., JSON). This formatted data is then passed as input to the generative AI model and emotion recognition device.

[0327] Step 4:

[0328] The emotion recognition device recognizes emotions from the user's text data. The technology used is, for example, IBM Watson's Natural Language Understanding. From the input data "My back hurts," the emotion "anxiety" is recognized. This recognized emotion data is provided to the generative AI model.

[0329] Step 5:

[0330] The generative AI model generates advice and suggestions for the user based on the emotional information received from the emotion recognition device. The generative AI model uses OpenAI's GPT model. Based on the input data "My lower back hurts" and the emotional information "anxiety," the advice message generated is "Try some moderate stretching. Specifically, do the following movements." This generated advice is sent back to the cloud server.

[0331] Step 6:

[0332] The server sends the advice received from the generative AI model to the user's device using HTTP or HTTPS as the transmission protocol. The input data includes the generated advice message.

[0333] Step 7:

[0334] The device provides the user with the advice received from the server. The advice is displayed as a text message on the application interface. Specifically, the message displayed is, "Try some moderate stretching. Don't push yourself too hard, and try it little by little."

[0335] Step 8 (optional):

[0336] To use additional features, the user selects them on the billing page within the application, and this selection information is sent from the terminal to the cloud server.

[0337] Step 9 (Optional):

[0338] The server processes the payment via a payment gateway (e.g., Stripe API). If the payment is successful, the add-on is applied to the user's account. Input data includes the user's payment information and the selected add-on.

[0339] (Application example 2)

[0340] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0341] In conventional store operations, in-store staff are responsible for all customer interactions and service provision, which results in a heavy workload. Furthermore, it is difficult to respond flexibly to customers' emotions and moods, creating a need for a system that can provide optimal service to individual customers. In particular, there is a growing need for a system that can recognize customer emotions and automatically generate appropriate advice and services based on them.

[0342] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0343] In this invention, the server includes means for inputting consultation content from a user, means for transmitting the input consultation content to a cloud server, means for analyzing the consultation content on the cloud server and passing it as input to a generative AI model, means for the generative AI model to generate advice or suggestions based on the consultation content, means for providing the generated advice to the user, means for recognizing the user's emotions using a camera, means for transmitting the emotion recognition result to the cloud server, means for the generative AI model to generate appropriate advice or suggestions for the user based on the emotion recognition result, and means for providing the advice or suggestions generated by an in-store robot. This makes it possible to efficiently handle customer service in the store and provide optimal services according to customer emotions.

[0344] A "user" is an individual or group that uses the system and is the entity that inputs the consultation content and voice data.

[0345] "Consultation content" refers to information such as questions, requests, and opinions that users provide to the system.

[0346] A "cloud server" is a computer system that provides computing resources that are remotely accessible via the Internet.

[0347] A "generative AI model" is an artificial intelligence algorithm that generates advice or suggestions based on specific input data (such as consultation content or emotion recognition results).

[0348] A "camera" is a device for capturing images and videos, and is used to recognize the user's emotions.

[0349] "Emotion recognition" is the process of analyzing information such as a user's facial expressions and voice to identify their emotional state (e.g., joy, sadness, anxiety, etc.).

[0350] A "robot" is a mechanical device that is programmed to work automatically or semi-automatically and perform specific tasks.

[0351] "Advice" is specific instructions or suggestions provided by the generative AI model based on the user's consultation content and emotions.

[0352] "Suggestions" are suggestions to the user for options or strategies to consider, generated based on the consultation and perceived emotions.

[0353] A "store" is a physical business location that offers goods and services to consumers.

[0354] "Staff" refers to the people who work in the store, dealing with customers and performing other tasks.

[0355] This invention is a system that combines generative AI models and emotion recognition technology to improve customer service and operational efficiency in brick-and-mortar stores. To realize this system, the following hardware and software are used to process and calculate data.

[0356] First, users (customers or staff) can input their concerns through a robot in a physical store. Input is done by voice or text using the robot's microphone or touch panel. The robot then sends the inputted concerns and voice data to a cloud server.

[0357] The cloud server analyzes the received data and passes it as input to the generative AI model. For emotion recognition, a camera mounted on the robot captures the user's facial expression and sends the image data to the cloud server. The cloud server then uses emotion recognition software (e.g., EmotionRecognition library) to recognize the user's emotion from the sent image data.

[0358] The recognized emotional information is fed into a generative AI model (e.g., GPT-2 model), which generates advice and suggestions based on the user's emotions and the content of the consultation. The generated advice and suggestions are then provided to the user from the cloud server via the robot's display and speakers.

[0359] For example, if a user says, "I'm not in the mood today," the robot will analyze their facial expressions and tone of voice and send them to the cloud server. If the cloud server recognizes the emotion and determines that the user is "tired," it will use the generative AI model to generate advice such as "I'll play some relaxing music," and provide it to the user via the robot.

[0360] Usage examples and prompt statements

[0361] Here is an example prompt:

[0362] The customer said: "I'm tired today." They seem to be feeling tired. What advice would be helpful for them?

[0363] In this way, the system of the present invention can improve the efficiency of customer service in stores and provide optimal services according to customer emotions.By using a generative AI model, it is possible to flexibly generate advice and suggestions customized for each individual user.

[0364] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0365] Step 1:

[0366] The user inputs the content of the consultation through the robot. The content of the consultation, input in voice or text format, is captured using the robot's microphone and touch panel.

[0367] Input: Consultation content (voice or text)

[0368] Output: The consultation content converted into data format by the robot

[0369] Step 2:

[0370] The robot sends the acquired consultation information to a cloud server using an internet connection.

[0371] Input: Consultation content converted into data format

[0372] Output: Data sent to the cloud server

[0373] Step 3:

[0374] The cloud server analyzes the received consultation content and starts emotion recognition.

[0375] Input: Consultation details sent

[0376] Output: Analysis results and data for emotion recognition

[0377] Step 4:

[0378] The cloud server uses the image data sent from the robot to recognize the user's emotions using emotion recognition software.

[0379] Input: Image data

[0380] Output: Recognized emotion information (e.g., joy, sadness, anxiety)

[0381] Step 5:

[0382] The cloud server inputs the consultation content and emotion recognition results into a generative AI model to generate appropriate advice and suggestions for the user.

[0383] Input: Consultation content, recognized emotion information

[0384] Output: Advice or suggestions generated by the generative AI model

[0385] Step 6:

[0386] The cloud server then sends the generated advice and suggestions back to the robot.

[0387] Input: Generated advice and suggestions

[0388] Output: Data sent to the robot

[0389] Step 7:

[0390] The robot provides the user with advice and suggestions received from the cloud server, which are communicated through the robot's display and speakers.

[0391] Input: Advice and suggestions received from the cloud server

[0392] Output: Advice and suggestions presented to the user

[0393] Step 8:

[0394] If desired, the user can interact with the robot again to provide additional consultation or feedback, and the process repeats.

[0395] Input: Additional consultation details, feedback

[0396] Output: Updated consultation details and feedback information

[0397] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0398] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0399] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0400] [Second embodiment]

[0401] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0402] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0403] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0404] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0405] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0406] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0407] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0408] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0409] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0410] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0411] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0412] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0413] This invention provides a system that allows users to use an application linked to a cloud server to seek advice about health conditions or lifestyle concerns or to enjoy entertainment. Specific embodiments of this system will be described below.

[0414] Process to receive consultation details from the user

[0415] First, the user launches the application on their smartphone or tablet and inputs their consultation details by voice or text. The application then sends the input to a cloud server, which analyzes the received data, converts it into an appropriate format, and passes it to the generative AI model.

[0416] AI responds to inquiries

[0417] On the cloud server, the generative AI model analyzes the consultation content received from the user and generates appropriate advice and suggestions. The generated advice is then sent back from the cloud server to the user's device and provided to the user via an application. This process occurs in real time, allowing users to receive advice quickly.

[0418] Providing daily conversation partners and entertainment

[0419] When a user selects "conversation partner mode" or "game mode" in an application, the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and sends it to the user's device. In "conversation partner mode," the generative AI model generates conversation content with the user and displays it in the application. In "game mode," game content is provided that can be enjoyed together with the user.

[0420] AI training function

[0421] Users can set the AI's characteristics and direction through the application's training screen. The training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. This allows users to receive personalized responses tailored to their preferences.

[0422] Revenue model and billing process

[0423] This system also provides a means for users to select additional features and for the cloud server to process billing based on that information. The user selects the additional features on a billing page within the application, and that information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[0424] Specific examples

[0425] For example, if User A complains of a sore lower back, the cloud server passes the information to the generative AI model. The generative AI model generates advice such as "Try some moderate stretching. Specifically, try these movements," and sends it to User A's device. Similarly, if User B thinks, "I'm feeling a little lonely today," and selects "Conversation mode," the generative AI model generates conversation content such as "What did you do today?" and provides it to User B.

[0426] In this way, the system of the present invention provides a concrete means for elderly users and single-person household users to lead richer daily lives.

[0427] The processing flow will be explained below.

[0428] Process to receive consultation details from the user

[0429] Step 1:

[0430] The user launches the app on their smartphone and inputs the details of their consultation either by voice or text.

[0431] Step 2:

[0432] The terminal converts the input consultation content into JSON format and sends an HTTP POST request to the cloud server.

[0433] AI responds to inquiries

[0434] Step 3:

[0435] The server parses the received JSON data of the consultation content and converts it into a specified format.

[0436] Step 4:

[0437] The server passes the consultation content after format conversion as input to the generative AI model.

[0438] Step 5:

[0439] A generative AI model generates advice and suggestions based on the consultation content.

[0440] Step 6:

[0441] The server converts the generated advice into JSON format and sends it to the user's device.

[0442] Step 7:

[0443] The device parses the received advice and displays it to the user within the application, and may also use speech synthesis to communicate it aloud if necessary.

[0444] Providing daily conversation partners and entertainment

[0445] Step 1:

[0446] The user selects either "talkie mode" or "game mode" in the application.

[0447] Step 2:

[0448] The device converts the selected mode information into JSON format and sends it to the cloud server via an HTTP POST request.

[0449] Step 3:

[0450] The server invokes a generative AI model or a game content generation system based on the received mode information.

[0451] Step 4:

[0452] In the conversation partner mode, the server uses the generative AI model to generate appropriate conversation content and send it to the user's device.In the game mode, the server generates game content and sends it to the user's device.

[0453] Step 5:

[0454] In the conversation partner mode, the terminal displays the generated conversation content, and in the game mode, the terminal starts a game so that the user can play together.

[0455] AI training function

[0456] Step 1:

[0457] Users set the AI's characteristics and direction through the application's development screen.

[0458] Step 2:

[0459] The device converts the development setting information into JSON format and sends it to the cloud server via an HTTP POST request.

[0460] Step 3:

[0461] The server automatically adjusts the response and proposal content of the generated AI model based on the received training setting information.

[0462] Revenue model and billing process

[0463] Step 1:

[0464] The user selects additional features from the application's billing page.

[0465] Step 2:

[0466] The device converts the purchase information into JSON format and sends it to the cloud server via an HTTP POST request.

[0467] Step 3:

[0468] The server calls a payment gateway based on the received purchase information and executes the payment process.

[0469] Step 4:

[0470] If the payment is successful, the server applies the additional function to the user's account and notifies the user's terminal of that information.

[0471] Step 5:

[0472] Based on the received information about the additional function, the terminal performs settings so that the user can use the function within the application.

[0473] Example 1

[0474] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0475] In modern life, people are looking for fast and effective solutions to their health and daily worries. Furthermore, with the increase in elderly people and single-person households, there is a need for mental health care and entertainment for those living alone. Systems that provide fast and personalized responses to these issues are lacking, and technological solutions are needed to meet user needs.

[0476] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0477] In this invention, the server includes means for inputting consultation details from a user, means for transmitting the input consultation details to a cloud server, means for the cloud server to analyze the received data, convert it into an appropriate format, and pass it to the generative AI model, means for the generative AI model to generate advice and suggestions based on the consultation details, and means for transmitting the generated advice again from the cloud server to the user's terminal and providing it to the user through an application. This enables users to receive quick and personalized advice and suggestions in real time, providing a powerful means for elderly people and single-person households to live richer daily lives.

[0478] "User" refers to a person who uses this system to input consultation details, select modes, and set up AI training.

[0479] "Consultation content" refers to information entered by the user regarding their health condition or concerns about their daily life.

[0480] A "cloud server" refers to a server that performs data analysis, sends data to generative AI models, and receives responses via the Internet.

[0481] A "generative AI model" refers to an artificial intelligence model that generates advice and suggestions based on the input consultation content.

[0482] "Application" refers to software that runs on the user's smartphone or tablet and allows them to input consultation details, select modes, and set up AI training.

[0483] "Conversation partner mode" refers to a mode in which the cloud server uses a generated AI model to provide the content of the conversation to the user.

[0484] "Game mode" refers to a mode in which the cloud server uses a generated AI model to provide users with entertainment content such as games and quizzes.

[0485] "Setting the AI's characteristics and direction" refers to the user making settings through the application to customize the responses and suggestions of the generated AI model.

[0486] "Training setting information" refers to information about the characteristics and direction of the AI ​​set by the user.

[0487] "Payment gateway" refers to an Internet service that allows a server to process payments based on billing information from users.

[0488] This invention provides a system that allows users to use an application linked to a cloud server to seek advice about health conditions or lifestyle concerns or to enjoy entertainment. Specific embodiments of this system will be described below.

[0489] Required Hardware and Software

[0490] User device: A computing device such as a smartphone or tablet.

[0491] Application: Software installed on a device that provides a user interface.

[0492] Cloud server: A server that performs data analysis and runs generative AI models via the internet.

[0493] Generative AI model: An artificial intelligence model that generates advice and suggestions based on the content of a consultation.

[0494] System Operation

[0495] First, the user launches the application on their smartphone or tablet and inputs the content of their consultation by voice or text. At this time, the voice data input by the user is converted into text data within the application. The application then sends the input content of the consultation to a cloud server. The cloud server analyzes the received data and converts it into a format that is easy for the generative AI model to understand.

[0496] The generative AI model on the cloud server analyzes the consultation content received from the user and generates appropriate advice and suggestions. The generated advice is sent from the cloud server to the user's device and provided to the user through an application. This process is carried out in real time, allowing users to receive advice quickly.

[0497] For example, if a user enters the text "My lower back hurts" and taps the send button, the device sends the input data to the cloud server. The cloud server analyzes the received data and passes it to the generative AI model. The generative AI model generates advice such as "Try some moderate stretching. Specifically, try these movements," and the cloud server sends the generated advice to the user's device. The device decodes the received advice and displays it on the screen. The user then checks the advice displayed on the screen.

[0498] Furthermore, when the user selects "conversation partner mode" or "game mode," the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and sends it to the user's device. In "conversation partner mode," the generative AI model generates conversation content with the user and displays it in the application. In "game mode," game content that can be enjoyed together with the user is provided.

[0499] Furthermore, users can set the AI's characteristics and direction through the application's training screen. The training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. This allows users to receive personalized responses tailored to their preferences.

[0500] As a revenue model, a method is also provided in which the user selects additional features and the cloud server processes billing based on that information. The user selects the additional features on the billing page within the application and the information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[0501] Examples of prompt statements

[0502] Consultation prompt:

[0503] Tell the AI ​​the following questions to generate appropriate advice:

[0504] Consultation: "My lower back hurts"

[0505] Interactive mode input prompt:

[0506] Continue the conversation with the user as follows:

[0507] Conversation topic: "I'm feeling a little lonely today"

[0508] Game mode input prompt:

[0509] Give your users a fun and easy quiz.

[0510] Theme: "General Knowledge"

[0511] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0512] Step 1:

[0513] The user launches the application on their smartphone or tablet and inputs the content of their consultation by voice or text. The input content is converted into text data by the application. After the user's voice input is converted into text, text data is generated. This is the output of Step 1.

[0514] Step 2:

[0515] The device transmits the text data, which is the output of step 1, to the cloud server. The transmitted text data is received by the cloud server. The data received by the cloud server is the input of step 2.

[0516] Step 3:

[0517] The server analyzes the received text data (input in step 2) and converts it into a format that is easy for the generative AI model to understand. For example, it converts it into JSON format. This format conversion is performed as data processing, and as a result, prepared data is generated to be passed to the generative AI model. This is the output of step 3.

[0518] Step 4:

[0519] The server passes the format-converted data, which is the output of step 3, to the generative AI model. The passed data becomes the input for the generative AI model, and at this stage the generative AI model begins analysis.

[0520] Step 5:

[0521] The generative AI model analyzes the format-converted data, which is the input in step 4, and generates appropriate advice and suggestions. Here, advice generated based on the prompt sentence is output. For example, for the input "My lower back hurts," specific advice such as "Try some moderate stretching" is generated.

[0522] Step 6:

[0523] The server sends the generated advice, which is the output of step 5, to the user's device. At this stage, the data is again encoded and transmitted securely using the appropriate communication protocol. The transmitted data is the input here.

[0524] Step 7:

[0525] The terminal receives and decodes the generated advice, which is the input of step 6. It displays the received advice to the user. This decoded advice is the output of step 7, and the user sees the information on the terminal screen.

[0526] Step 8:

[0527] The user selects "conversation mode" or "game mode" within the application. The selected mode information is input.

[0528] Step 9:

[0529] The terminal transmits mode information to the cloud server, which is the input of step 8. The transmitted mode information is received by the cloud server. The received mode information is the output of step 9.

[0530] Step 10:

[0531] The server generates appropriate content using the generative AI model based on the received mode information, which is the output of step 9. For example, in "conversation partner mode," it generates appropriate conversation content for the user, and in "game mode," it generates games and quizzes that can be enjoyed together with the user. The generated content is the output of step 10.

[0532] Step 11:

[0533] The server transmits the generated content, which is the output of step 10, to the user's terminal. The transmitted content becomes the input of step 11.

[0534] Step 12:

[0535] The terminal receives the generated content, which is the input of step 11, and displays it on the screen. This displayed content is the output of step 12, and the user can view, manipulate, and enjoy it.

[0536] Step 13:

[0537] Users can set the AI's characteristics and direction through the application's training screen, and this information is used as input.

[0538] Step 14:

[0539] The terminal transmits the development setting information, which is the input of step 13, to the cloud server. The transmitted development setting information is the output of step 14.

[0540] Step 15:

[0541] The server automatically adjusts the response and proposal content of the generated AI model based on the training setting information output in step 14. These adjusted response and proposal content are the output of step 15.

[0542] Step 16:

[0543] The user selects an additional feature on the billing page within the application, and the selected billing information is input.

[0544] Step 17:

[0545] The terminal transmits the billing information, which is the input of step 16, to the cloud server. The transmitted billing information is the output of step 17.

[0546] Step 18:

[0547] The server receives the billing information output from step 17 and passes it to the payment gateway for payment processing. If the payment is successful, it applies additional features to the user account. The applied additional features are output from step 18.

[0548] Step 19:

[0549] The terminal displays the applied additional function, which is the output of step 18, to the user and makes it available for use. The user can use the new function through the terminal application.

[0550] (Application example 1)

[0551] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0552] Maintaining a healthy diet and managing your health can be challenging in today's hectic lifestyles. Getting accurate advice based on your health status and individual dietary preferences is particularly challenging, and there are no systems that can directly order meals. Conventional food delivery systems simply deliver the meals selected by the user, but are unable to provide suggestions or advice based on the user's health status. Furthermore, there is a lack of systems that can quickly provide appropriate advice and even order meals on the spot.

[0553] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0554] In this invention, the server includes means for inputting consultation details from a user, means for transmitting the input consultation details to a cloud server, means for analyzing the consultation details on the cloud server and passing them as input to a generative AI model, means for the generative AI model to generate advice and suggestions based on the consultation details, means for providing the generated advice to the user, means for generating meal suggestions based on health status, and means for ordering the suggested meals from a food delivery system. This enables the user to quickly receive appropriate advice based on their health status and then order the suggested meals from a food delivery system.

[0555] A "user" is someone who uses this system to input their consultation details and receive health advice and dietary suggestions.

[0556] The "cloud server" is a remote server that receives and analyzes the consultation content sent by users via the Internet and generates advice and suggestions using a generative AI model.

[0557] "Consultation content" refers to worries, questions, and requests about health and diet entered by the user.

[0558] A "generative AI model" is an artificial intelligence (AI) model that runs on a cloud server and generates advice and suggestions based on the input consultation content.

[0559] "Advice and suggestions" are specific recommendations and instructions generated by the generative AI model based on the user's consultation.

[0560] "Health status" refers to information about the user's physical condition, food preferences, and nutritional balance.

[0561] "Meal suggestions" are specific meal menu suggestions recommended by the generative AI model based on the user's health status.

[0562] A "food delivery system" is an online service that allows users to order and have food delivered from a specified menu.

[0563] This system allows users to consult about their health and diet using devices such as smartphones and tablets, and receives appropriate advice and suggestions in cooperation with a cloud server. Furthermore, based on the suggestions, users can order meals via a food delivery system.

[0564] Overall system configuration

[0565] The system mainly consists of the following components:

[0566] 1. User Device

[0567] 2. Cloud Server

[0568] 3. Generative AI Models

[0569] 4. Food delivery system

[0570] Operation on the user device

[0571] Users start up a dedicated application on their smartphone or tablet and first input their health and dietary concerns by voice or text. This input is then sent from the device to a cloud server.

[0572] Analysis on a cloud server

[0573] The cloud server analyzes the consultation content received from the user, converts it into an appropriate format, and passes it to a generative AI model. For example, OpenAI's GPT-3 (generative AI model) is used as the generative AI model. This model generates advice and suggestions based on the user's consultation content.

[0574] Generate advice and suggestions

[0575] The generative AI model analyzes the consultation content sent from the cloud server and generates appropriate advice and suggestions. For example, it may suggest an appropriate diet based on the user's health condition. This generated advice and suggestions are then sent back to the user's device via the cloud server.

[0576] Meal suggestions based on health status

[0577] The cloud server generates meal suggestions appropriate for the user's health condition based on the advice generated by the generative AI model. For example, the advice generated might be, "Try to eat a balanced meal centered around vegetables today."

[0578] Ordering from a food delivery system

[0579] The user terminal transmits a meal order to the food delivery system based on the suggestions, which includes the meal menu, delivery address, payment method, etc. specified by the user.

[0580] Specific examples

[0581] For example, if a user types into their device, "What is the best meal for my health today?", the cloud server passes this information to the generative AI model. The generative AI model generates advice such as, "Try to eat a balanced meal with a focus on vegetables today." The cloud server then uses this advice to suggest a "healthy plate of grilled vegetables and chicken" and orders this meal through a food delivery system.

[0582] Prompt Sentence Examples

[0583] "Provide appropriate meal suggestions based on the user's health status. Advice: 'Aim for a balanced meal with a focus on vegetables today.'"

[0584] As described above, the present invention is a system that supports the user's health management and promptly suggests and delivers appropriate meals.

[0585] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0586] Step 1:

[0587] The user launches a dedicated application on their smartphone or tablet and inputs the details of their consultation by voice or text. The inputted details are sent from the device to the cloud server. The inputted data is sent in the form of text data.

[0588] Step 2:

[0589] The cloud server analyzes the received consultation content and converts it into an appropriate format. Specifically, it uses natural language processing to analyze the text data and converts it into a format that can be understood by a generative AI model (e.g., OpenAI's GPT-3). This process involves extracting the intent and keywords of the consultation content. The input is the text data of the user's consultation content, and the output is data formatted for the generative AI model.

[0590] Step 3:

[0591] A generative AI model on a cloud server generates advice and suggestions based on the formatted consultation content. The generative AI model uses an internal algorithm to analyze the user's consultation content and construct optimal advice and suggestions. In this process, the advice is generated using prompt text. The input is the formatted consultation content data, and the output is the generated advice or suggestion text.

[0592] Step 4:

[0593] The generated advice and suggestions are then sent to the user's device via the cloud server. The cloud server converts the output data from the generative AI model into a format that is easy for the user to understand and sends it. The input is the advice and suggestion text generated by the generative AI model, and the output is the advice text sent to the user's device.

[0594] Step 5:

[0595] The user device displays the advice and suggestions received from the cloud server. The user can check this through the application. Specifically, the received text data is displayed on the screen. The input is the advice text from the cloud server, and the output is the advice content displayed on the device.

[0596] Step 6:

[0597] The cloud server generates meal suggestions suitable for the user's health condition based on the advice created by the generative AI model. Specifically, it generates prompt sentences based on the generated advice and makes food suggestions. The input is text data of the advice content, and the output is text data of the meal suggestions.

[0598] Step 7:

[0599] The user terminal receives the meal suggestions sent from the cloud server and sends the order directly to the food delivery system. Specifically, it generates order information including the suggested meal menu, delivery address, and payment method and sends it to the food delivery system. The input is text data of the meal suggestions, and the output is order data sent to the food delivery system.

[0600] Through the above steps, the user can receive appropriate advice for health management and order meals based on that advice.

[0601] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0602] This invention is a system that combines a generative AI model and an emotion engine to support users' daily lives, provide appropriate advice for consultations, and respond based on emotion recognition. Specific embodiments of this system are described below.

[0603] Process to receive consultation details from the user

[0604] The user launches the application on their smartphone or tablet and inputs their consultation details by voice or text. The device then sends the input consultation details and voice data to a cloud server. The cloud server analyzes the received data, converts it into an appropriate format, and passes it to the generative AI model and emotion engine.

[0605] Emotion recognition and advice generation using an emotion engine

[0606] On the cloud server, the emotion engine recognizes emotions from the consultation content and voice data entered by the user. The recognized emotion information is reflected in the generative AI model, which then generates advice and suggestions based on the user's emotions. The generated advice is then sent back from the cloud server to the user's device and provided to the user via an application.

[0607] Providing daily conversation partners and entertainment

[0608] When a user selects "conversation partner mode" or "game mode" in the application, the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and emotions and sends it to the user's device. In "conversation partner mode," the emotion engine generates and provides conversation content based on the emotions recognized by the emotion engine. In "game mode," the difficulty and content of the game are adjusted according to the user's emotional state.

[0609] AI training function

[0610] Users can set the AI's characteristics and direction through the application's training screen. Training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. By combining emotion engines, the user's emotional state is also taken into account, allowing for the provision of optimal services tailored to each individual user.

[0611] Revenue model and billing process

[0612] This system also provides a means for users to select additional features and for the cloud server to process billing based on that information. The user selects the additional features on a billing page within the application, and that information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[0613] Specific examples

[0614] For example, if User A complains of a "pain in the lower back," the cloud server passes the information to the emotion engine. The emotion engine recognizes "anxiety" from User A's voice data and provides this information to the generative AI model. The generative AI model then generates a message that takes User A's emotions into consideration, such as "Try some moderate stretching. Specifically, try these movements," along with advice such as "Try it slowly, without overdoing it," and sends it to User A's device.

[0615] If User B thinks, "I'm feeling a little lonely today," and selects "Conversational Partner Mode," the emotion engine will recognize this feeling. The generative AI model will start the conversation with something like, "What did you do today?", and then generate content that takes emotion into consideration, such as, "Shall we have a fun conversation?", and provide it to User B.

[0616] In this way, the system of the present invention provides a concrete means for elderly users and single-person household users to lead richer daily lives.

[0617] The processing flow will be explained below.

[0618] Process to receive consultation details from the user

[0619] Step 1:

[0620] The user starts the smartphone app and inputs the details of their consultation. The user can input the details of their consultation by voice or text.

[0621] Step 2:

[0622] The device converts the input consultation content and voice data into JSON format and sends it to the cloud server via an HTTP POST request.

[0623] AI-based consultation processing and emotion recognition

[0624] Step 3:

[0625] The server parses the JSON data of the received consultation content and voice data and converts it into a specified format.

[0626] Step 4:

[0627] The server passes the consultation content after format conversion to the generative AI model and emotion engine.

[0628] Step 5:

[0629] The emotion engine recognizes emotions from the user's voice data and text. For example, emotions such as "anxiety" and "loneliness" can be recognized through voice analysis.

[0630] Step 6:

[0631] The generative AI model generates advice and suggestions based on the recognized emotional information and the content of the consultation. Specifically, in addition to advice such as "Try some moderate stretching. Specifically, try these movements," it generates messages that take the user's emotions into consideration, such as "Try it little by little without overdoing it."

[0632] Step 7:

[0633] The server converts the generated advice and emotion-sensitive messages into JSON format and sends them to the user's device.

[0634] Step 8:

[0635] The device parses the received advice and messages and displays them to the user within the application, and may also use speech synthesis to communicate them aloud if necessary.

[0636] Providing daily conversation partners and entertainment

[0637] Step 1:

[0638] The user selects either "talkie mode" or "game mode" in the application.

[0639] Step 2:

[0640] The device converts the selected mode information into JSON format and sends it to the cloud server via an HTTP POST request.

[0641] Step 3:

[0642] The server invokes a generative AI model and an emotion engine based on the received mode information.

[0643] Step 4:

[0644] The emotion engine recognizes emotions from the user's voice data and text and provides that information to the generative AI model.

[0645] Step 5:

[0646] The generative AI model generates appropriate conversation content in conversation partner mode and appropriate game content in game mode based on emotion information and mode information from the emotion engine.

[0647] Step 6:

[0648] The server converts the generated conversation content or game content into JSON format and sends it to the user's device.

[0649] Step 7:

[0650] In the conversation partner mode, the terminal displays the generated conversation content, and in the game mode, the terminal starts the generated game so that the user can play together with the user.

[0651] AI training function

[0652] Step 1:

[0653] Users set the AI's characteristics and direction through the application's development screen.

[0654] Step 2:

[0655] The device converts the development setting information into JSON format and sends it to the cloud server via an HTTP POST request.

[0656] Step 3:

[0657] The server automatically adjusts the responses and suggestions of the generative AI model based on the received training setting information, taking into account information from the emotion engine.

[0658] Revenue model and billing process

[0659] Step 1:

[0660] The user selects additional features from the application's billing page.

[0661] Step 2:

[0662] The device converts the purchase information into JSON format and sends it to the cloud server via an HTTP POST request.

[0663] Step 3:

[0664] The server calls a payment gateway based on the received purchase information and executes the payment process.

[0665] Step 4:

[0666] If the payment is successful, the server applies the additional features to the user's account and sends the information to the user's device via an HTTP POST request.

[0667] Step 5:

[0668] Based on the received additional function information, the terminal performs settings so that the user can use the function within the application.

[0669] Example 2

[0670] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0671] In modern society, users face various worries and problems in their daily lives, but it is difficult to quickly obtain appropriate advice. Furthermore, there is a lack of appropriate responses that take emotions into consideration, and people with whom people can easily talk, leading to feelings of loneliness. To address these issues, there is a need for a system that can understand users' emotions and provide appropriate advice and entertainment.

[0672] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0673] In this invention, the server includes means for inputting consultation content from a user, means for transmitting the input consultation content to a cloud server, means for analyzing the consultation content on the cloud server, converting it into an appropriate format, and passing it as input to the generative AI model and emotion identification device, means for the emotion identification device to recognize emotions from the consultation content and voice data, means for the generative AI model to generate advice and suggestions based on the recognized emotion information, and means for providing the generated advice to the user from the cloud server. This allows users to receive appropriate advice and entertainment according to their emotions at any given time, thereby improving the quality of their daily lives.

[0674] "User" refers to an individual who uses the system to input their inquiry and receive advice and suggestions.

[0675] "Consultation content" refers to the worries, questions, and other inquiries that the user inputs into the system by voice or text.

[0676] A "cloud server" is a server that can be accessed remotely via the Internet and is a central element of a system that analyzes and processes data.

[0677] A "generative AI model" is a generative model created by artificial intelligence, and refers to a program that creates advice and suggestions based on user input and emotional information.

[0678] An "emotion identification device" refers to a device or program that recognizes emotions from user input or voice data and reflects them in the analysis results.

[0679] "Advice and suggestions" refers to specific advice and recommendations generated by the generative AI model based on the user's consultation content and emotional information.

[0680] The "conversation partner mode" refers to a mode in which the user selects how the system will respond as a conversation partner, and conversation content based on emotions is provided.

[0681] "Play mode" refers to a mode in which the user selects a mode that provides games and entertainment content to the system, and the difficulty and content of the game are adjusted according to the user's emotional state.

[0682] "Setting the characteristics and direction of artificial intelligence" refers to the operation in which the user sets specific parameters through a settings screen within the system to adjust the response and proposal content of the generated AI model.

[0683] "Payment gateway" refers to an interface through which the cloud server processes the user's payment information and performs billing processing.

[0684] This invention is a system that allows users to input their consultation details using an application on a smartphone or tablet and receive appropriate advice and suggestions in response. The system operates on a cloud server using a generative AI model and an emotion recognition device. A specific embodiment of this system is described below.

[0685] The user launches an application installed on their smartphone or tablet and inputs the consultation details by voice or text. The device then sends the input consultation details to a cloud server. This transmission uses a communication protocol (e.g., HTTP, HTTPS) via an internet connection.

[0686] The cloud server parses the received data and converts it into an appropriate format (e.g., JSON format) using, for example, a natural language processing library or speech recognition technology (e.g., Google Speech-to-Text API).The converted data is then passed to a generative AI model and an emotion recognition device.

[0687] The emotion recognition device recognizes emotions from the consultation content and voice data entered by the user. The technology used can be, for example, IBM Watson's Natural Language Understanding. The emotion recognition device provides the recognized emotional information to the generative AI model.

[0688] The generative AI model generates advice and suggestions for the user based on this emotional information. For example, OpenAI's GPT model is used for generation. The generative AI model generates specific suggestions and response messages that reflect the user's emotional state.

[0689] The generated advice and suggestions are sent from the cloud server to the user's device. The device then provides the received advice to the user through an application. The advice is displayed using a visually easy-to-understand interface (e.g., text message, pop-up message).

[0690] For example, if a user complains of a "pain in the lower back," the cloud server passes the information to an emotion recognition device. The emotion recognition device recognizes "anxiety" from the voice data. This information is provided to a generative AI model, which then generates advice such as "Try some moderate stretching. Specifically, try the following movements," along with a message that takes the user's emotions into consideration, such as "Try doing it gradually, without overdoing it." The message is then sent to the user's device.

[0691] Furthermore, if the user selects "conversation partner mode," the emotion recognition device recognizes "loneliness." In this case, the generative AI model generates emotion-sensitive content, such as "What did you do today?" and "Shall we have a little fun conversation?", and provides these to the user.

[0692] An example of a specific prompt is as follows: "If a user complains of lower back pain, what advice would the emotion recognizer generate for the generative AI model by recognizing anxiety from the voice data?"

[0693] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0694] Step 1:

[0695] The user launches the application on their smartphone or tablet. Here, the user inputs the details of their consultation by voice or text. The input data is temporarily stored within the device. An example of input data is the voice input of "My lower back hurts."

[0696] Step 2:

[0697] The device converts the voice data entered by the user into text data. This conversion uses voice recognition technology (e.g., Google Speech-to-Text API). The converted text data becomes "My lower back hurts." This text data is then sent to a cloud server. HTTP or HTTPS is used as the transmission protocol.

[0698] Step 3:

[0699] The server parses the received text data using a natural language processing library (e.g., NLTK). The parsed data is then converted into an appropriate format (e.g., JSON). This formatted data is then passed as input to the generative AI model and emotion recognition device.

[0700] Step 4:

[0701] The emotion recognition device recognizes emotions from the user's text data. The technology used is, for example, IBM Watson's Natural Language Understanding. From the input data "My back hurts," the emotion "anxiety" is recognized. This recognized emotion data is provided to the generative AI model.

[0702] Step 5:

[0703] The generative AI model generates advice and suggestions for the user based on the emotional information received from the emotion recognition device. The generative AI model uses OpenAI's GPT model. Based on the input data "My lower back hurts" and the emotional information "anxiety," the advice message generated is "Try some moderate stretching. Specifically, do the following movements." This generated advice is sent back to the cloud server.

[0704] Step 6:

[0705] The server sends the advice received from the generative AI model to the user's device using HTTP or HTTPS as the transmission protocol. The input data includes the generated advice message.

[0706] Step 7:

[0707] The device provides the user with the advice received from the server. The advice is displayed as a text message on the application interface. Specifically, the message displayed is, "Try some moderate stretching. Don't push yourself too hard, and try it little by little."

[0708] Step 8 (optional):

[0709] To use additional features, the user selects them on the billing page within the application, and this selection information is sent from the terminal to the cloud server.

[0710] Step 9 (Optional):

[0711] The server processes the payment via a payment gateway (e.g., Stripe API). If the payment is successful, the add-on is applied to the user's account. Input data includes the user's payment information and the selected add-on.

[0712] (Application example 2)

[0713] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0714] In conventional store operations, in-store staff are responsible for all customer interactions and service provision, which results in a heavy workload. Furthermore, it is difficult to respond flexibly to customers' emotions and moods, creating a need for a system that can provide optimal service to individual customers. In particular, there is a growing need for a system that can recognize customer emotions and automatically generate appropriate advice and services based on them.

[0715] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0716] In this invention, the server includes means for inputting consultation content from a user, means for transmitting the input consultation content to a cloud server, means for analyzing the consultation content on the cloud server and passing it as input to a generative AI model, means for the generative AI model to generate advice or suggestions based on the consultation content, means for providing the generated advice to the user, means for recognizing the user's emotions using a camera, means for transmitting the emotion recognition result to the cloud server, means for the generative AI model to generate appropriate advice or suggestions for the user based on the emotion recognition result, and means for providing the advice or suggestions generated by an in-store robot. This makes it possible to efficiently handle customer service in the store and provide optimal services according to customer emotions.

[0717] A "user" is an individual or group that uses the system and is the entity that inputs the consultation content and voice data.

[0718] "Consultation content" refers to information such as questions, requests, and opinions that users provide to the system.

[0719] A "cloud server" is a computer system that provides computing resources that are remotely accessible via the Internet.

[0720] A "generative AI model" is an artificial intelligence algorithm that generates advice or suggestions based on specific input data (such as consultation content or emotion recognition results).

[0721] A "camera" is a device for capturing images and videos, and is used to recognize the user's emotions.

[0722] "Emotion recognition" is the process of analyzing information such as a user's facial expressions and voice to identify their emotional state (e.g., joy, sadness, anxiety, etc.).

[0723] A "robot" is a mechanical device that is programmed to work automatically or semi-automatically and perform specific tasks.

[0724] "Advice" is specific instructions or suggestions provided by the generative AI model based on the user's consultation content and emotions.

[0725] "Suggestions" are suggestions to the user for options or strategies to consider, generated based on the consultation and perceived emotions.

[0726] A "store" is a physical business location that offers goods and services to consumers.

[0727] "Staff" refers to the people who work in the store, dealing with customers and performing other tasks.

[0728] This invention is a system that combines generative AI models and emotion recognition technology to improve customer service and operational efficiency in brick-and-mortar stores. To realize this system, the following hardware and software are used to process and calculate data.

[0729] First, users (customers or staff) can input their concerns through a robot in a physical store. Input is done by voice or text using the robot's microphone or touch panel. The robot then sends the inputted concerns and voice data to a cloud server.

[0730] The cloud server analyzes the received data and passes it as input to the generative AI model. For emotion recognition, a camera mounted on the robot captures the user's facial expression and sends the image data to the cloud server. The cloud server then uses emotion recognition software (e.g., EmotionRecognition library) to recognize the user's emotion from the sent image data.

[0731] The recognized emotional information is fed into a generative AI model (e.g., GPT-2 model), which generates advice and suggestions based on the user's emotions and the content of the consultation. The generated advice and suggestions are then provided to the user from the cloud server via the robot's display and speakers.

[0732] For example, if a user says, "I'm not in the mood today," the robot will analyze their facial expressions and tone of voice and send them to the cloud server. If the cloud server recognizes the emotion and determines that the user is "tired," it will use the generative AI model to generate advice such as "I'll play some relaxing music," and provide it to the user via the robot.

[0733] Usage examples and prompt statements

[0734] Here is an example prompt:

[0735] The customer said: "I'm tired today." They seem to be feeling tired. What advice would be helpful for them?

[0736] In this way, the system of the present invention can improve the efficiency of customer service in stores and provide optimal services according to customer emotions.By using a generative AI model, it is possible to flexibly generate advice and suggestions customized for each individual user.

[0737] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0738] Step 1:

[0739] The user inputs the content of the consultation through the robot. The content of the consultation, input in voice or text format, is captured using the robot's microphone and touch panel.

[0740] Input: Consultation content (voice or text)

[0741] Output: The consultation content converted into data format by the robot

[0742] Step 2:

[0743] The robot sends the acquired consultation information to a cloud server using an internet connection.

[0744] Input: Consultation content converted into data format

[0745] Output: Data sent to the cloud server

[0746] Step 3:

[0747] The cloud server analyzes the received consultation content and starts emotion recognition.

[0748] Input: Consultation details sent

[0749] Output: Analysis results and data for emotion recognition

[0750] Step 4:

[0751] The cloud server uses the image data sent from the robot to recognize the user's emotions using emotion recognition software.

[0752] Input: Image data

[0753] Output: Recognized emotion information (e.g., joy, sadness, anxiety)

[0754] Step 5:

[0755] The cloud server inputs the consultation content and emotion recognition results into a generative AI model to generate appropriate advice and suggestions for the user.

[0756] Input: Consultation content, recognized emotion information

[0757] Output: Advice or suggestions generated by the generative AI model

[0758] Step 6:

[0759] The cloud server then sends the generated advice and suggestions back to the robot.

[0760] Input: Generated advice and suggestions

[0761] Output: Data sent to the robot

[0762] Step 7:

[0763] The robot provides the user with advice and suggestions received from the cloud server, which are communicated through the robot's display and speakers.

[0764] Input: Advice and suggestions received from the cloud server

[0765] Output: Advice and suggestions presented to the user

[0766] Step 8:

[0767] If desired, the user can interact with the robot again to provide additional consultation or feedback, and the process repeats.

[0768] Input: Additional consultation details, feedback

[0769] Output: Updated consultation details and feedback information

[0770] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0771] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0772] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0773] [Third embodiment]

[0774] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0775] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0776] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0777] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0778] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0779] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0780] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0781] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0782] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0783] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0784] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0785] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0786] This invention provides a system that allows users to use an application linked to a cloud server to seek advice about health conditions or lifestyle concerns or to enjoy entertainment. Specific embodiments of this system will be described below.

[0787] Process to receive consultation details from the user

[0788] First, the user launches the application on their smartphone or tablet and inputs their consultation details by voice or text. The application then sends the input to a cloud server, which analyzes the received data, converts it into an appropriate format, and passes it to the generative AI model.

[0789] AI responds to inquiries

[0790] On the cloud server, the generative AI model analyzes the consultation content received from the user and generates appropriate advice and suggestions. The generated advice is then sent back from the cloud server to the user's device and provided to the user via an application. This process occurs in real time, allowing users to receive advice quickly.

[0791] Providing daily conversation partners and entertainment

[0792] When a user selects "conversation partner mode" or "game mode" in an application, the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and sends it to the user's device. In "conversation partner mode," the generative AI model generates conversation content with the user and displays it in the application. In "game mode," game content is provided that can be enjoyed together with the user.

[0793] AI training function

[0794] Users can set the AI's characteristics and direction through the application's training screen. The training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. This allows users to receive personalized responses tailored to their preferences.

[0795] Revenue model and billing process

[0796] This system also provides a means for users to select additional features and for the cloud server to process billing based on that information. The user selects the additional features on a billing page within the application, and that information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[0797] Specific examples

[0798] For example, if User A complains of a sore lower back, the cloud server passes the information to the generative AI model. The generative AI model generates advice such as "Try some moderate stretching. Specifically, try these movements," and sends it to User A's device. Similarly, if User B thinks, "I'm feeling a little lonely today," and selects "Conversation mode," the generative AI model generates conversation content such as "What did you do today?" and provides it to User B.

[0799] In this way, the system of the present invention provides a concrete means for elderly users and single-person household users to lead richer daily lives.

[0800] The processing flow will be explained below.

[0801] Process to receive consultation details from the user

[0802] Step 1:

[0803] The user launches the app on their smartphone and inputs the details of their consultation either by voice or text.

[0804] Step 2:

[0805] The terminal converts the input consultation content into JSON format and sends an HTTP POST request to the cloud server.

[0806] AI responds to inquiries

[0807] Step 3:

[0808] The server parses the received JSON data of the consultation content and converts it into a specified format.

[0809] Step 4:

[0810] The server passes the consultation content after format conversion as input to the generative AI model.

[0811] Step 5:

[0812] A generative AI model generates advice and suggestions based on the consultation content.

[0813] Step 6:

[0814] The server converts the generated advice into JSON format and sends it to the user's device.

[0815] Step 7:

[0816] The device parses the received advice and displays it to the user within the application, and may also use speech synthesis to communicate it aloud if necessary.

[0817] Providing daily conversation partners and entertainment

[0818] Step 1:

[0819] The user selects either "talkie mode" or "game mode" in the application.

[0820] Step 2:

[0821] The device converts the selected mode information into JSON format and sends it to the cloud server via an HTTP POST request.

[0822] Step 3:

[0823] The server invokes a generative AI model or a game content generation system based on the received mode information.

[0824] Step 4:

[0825] In the conversation partner mode, the server uses the generative AI model to generate appropriate conversation content and send it to the user's device.In the game mode, the server generates game content and sends it to the user's device.

[0826] Step 5:

[0827] In the conversation partner mode, the terminal displays the generated conversation content, and in the game mode, the terminal starts a game so that the user can play together.

[0828] AI training function

[0829] Step 1:

[0830] Users set the AI's characteristics and direction through the application's development screen.

[0831] Step 2:

[0832] The device converts the development setting information into JSON format and sends it to the cloud server via an HTTP POST request.

[0833] Step 3:

[0834] The server automatically adjusts the response and proposal content of the generated AI model based on the received training setting information.

[0835] Revenue model and billing process

[0836] Step 1:

[0837] The user selects additional features from the application's billing page.

[0838] Step 2:

[0839] The device converts the purchase information into JSON format and sends it to the cloud server via an HTTP POST request.

[0840] Step 3:

[0841] The server calls a payment gateway based on the received purchase information and executes the payment process.

[0842] Step 4:

[0843] If the payment is successful, the server applies the additional function to the user's account and notifies the user's terminal of that information.

[0844] Step 5:

[0845] Based on the received information about the additional function, the terminal performs settings so that the user can use the function within the application.

[0846] Example 1

[0847] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0848] In modern life, people are looking for fast and effective solutions to their health and daily worries. Furthermore, with the increase in elderly people and single-person households, there is a need for mental health care and entertainment for those living alone. Systems that provide fast and personalized responses to these issues are lacking, and technological solutions are needed to meet user needs.

[0849] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0850] In this invention, the server includes means for inputting consultation details from a user, means for transmitting the input consultation details to a cloud server, means for the cloud server to analyze the received data, convert it into an appropriate format, and pass it to the generative AI model, means for the generative AI model to generate advice and suggestions based on the consultation details, and means for transmitting the generated advice again from the cloud server to the user's terminal and providing it to the user through an application. This enables users to receive quick and personalized advice and suggestions in real time, providing a powerful means for elderly people and single-person households to live richer daily lives.

[0851] "User" refers to a person who uses this system to input consultation details, select modes, and set up AI training.

[0852] "Consultation content" refers to information entered by the user regarding their health condition or concerns about their daily life.

[0853] A "cloud server" refers to a server that performs data analysis, sends data to generative AI models, and receives responses via the Internet.

[0854] A "generative AI model" refers to an artificial intelligence model that generates advice and suggestions based on the input consultation content.

[0855] "Application" refers to software that runs on the user's smartphone or tablet and allows them to input consultation details, select modes, and set up AI training.

[0856] "Conversation partner mode" refers to a mode in which the cloud server uses a generated AI model to provide the content of the conversation to the user.

[0857] "Game mode" refers to a mode in which the cloud server uses a generated AI model to provide users with entertainment content such as games and quizzes.

[0858] "Setting the AI's characteristics and direction" refers to the user making settings through the application to customize the responses and suggestions of the generated AI model.

[0859] "Training setting information" refers to information about the characteristics and direction of the AI ​​set by the user.

[0860] "Payment gateway" refers to an Internet service that allows a server to process payments based on billing information from users.

[0861] This invention provides a system that allows users to use an application linked to a cloud server to seek advice about health conditions or lifestyle concerns or to enjoy entertainment. Specific embodiments of this system will be described below.

[0862] Required Hardware and Software

[0863] User device: A computing device such as a smartphone or tablet.

[0864] Application: Software installed on a device that provides a user interface.

[0865] Cloud server: A server that performs data analysis and runs generative AI models via the internet.

[0866] Generative AI model: An artificial intelligence model that generates advice and suggestions based on the content of a consultation.

[0867] System Operation

[0868] First, the user launches the application on their smartphone or tablet and inputs the content of their consultation by voice or text. At this time, the voice data input by the user is converted into text data within the application. The application then sends the input content of the consultation to a cloud server. The cloud server analyzes the received data and converts it into a format that is easy for the generative AI model to understand.

[0869] The generative AI model on the cloud server analyzes the consultation content received from the user and generates appropriate advice and suggestions. The generated advice is sent from the cloud server to the user's device and provided to the user through an application. This process is carried out in real time, allowing users to receive advice quickly.

[0870] For example, if a user enters the text "My lower back hurts" and taps the send button, the device sends the input data to the cloud server. The cloud server analyzes the received data and passes it to the generative AI model. The generative AI model generates advice such as "Try some moderate stretching. Specifically, try these movements," and the cloud server sends the generated advice to the user's device. The device decodes the received advice and displays it on the screen. The user then checks the advice displayed on the screen.

[0871] Furthermore, when the user selects "conversation partner mode" or "game mode," the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and sends it to the user's device. In "conversation partner mode," the generative AI model generates conversation content with the user and displays it in the application. In "game mode," game content that can be enjoyed together with the user is provided.

[0872] Furthermore, users can set the AI's characteristics and direction through the application's training screen. The training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. This allows users to receive personalized responses tailored to their preferences.

[0873] As a revenue model, a method is also provided in which the user selects additional features and the cloud server processes billing based on that information. The user selects the additional features on the billing page within the application and the information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[0874] Examples of prompt statements

[0875] Consultation prompt:

[0876] Tell the AI ​​the following questions to generate appropriate advice:

[0877] Consultation: "My lower back hurts"

[0878] Interactive mode input prompt:

[0879] Continue the conversation with the user as follows:

[0880] Conversation topic: "I'm feeling a little lonely today"

[0881] Game mode input prompt:

[0882] Give your users a fun and easy quiz.

[0883] Theme: "General Knowledge"

[0884] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0885] Step 1:

[0886] The user launches the application on their smartphone or tablet and inputs the content of their consultation by voice or text. The input content is converted into text data by the application. After the user's voice input is converted into text, text data is generated. This is the output of Step 1.

[0887] Step 2:

[0888] The device transmits the text data, which is the output of step 1, to the cloud server. The transmitted text data is received by the cloud server. The data received by the cloud server is the input of step 2.

[0889] Step 3:

[0890] The server analyzes the received text data (input in step 2) and converts it into a format that is easy for the generative AI model to understand. For example, it converts it into JSON format. This format conversion is performed as data processing, and as a result, prepared data is generated to be passed to the generative AI model. This is the output of step 3.

[0891] Step 4:

[0892] The server passes the format-converted data, which is the output of step 3, to the generative AI model. The passed data becomes the input for the generative AI model, and at this stage the generative AI model begins analysis.

[0893] Step 5:

[0894] The generative AI model analyzes the format-converted data, which is the input in step 4, and generates appropriate advice and suggestions. Here, advice generated based on the prompt sentence is output. For example, for the input "My lower back hurts," specific advice such as "Try some moderate stretching" is generated.

[0895] Step 6:

[0896] The server sends the generated advice, which is the output of step 5, to the user's device. At this stage, the data is again encoded and transmitted securely using the appropriate communication protocol. The transmitted data is the input here.

[0897] Step 7:

[0898] The terminal receives and decodes the generated advice, which is the input of step 6. It displays the received advice to the user. This decoded advice is the output of step 7, and the user sees the information on the terminal screen.

[0899] Step 8:

[0900] The user selects "conversation mode" or "game mode" within the application. The selected mode information is input.

[0901] Step 9:

[0902] The terminal transmits mode information to the cloud server, which is the input of step 8. The transmitted mode information is received by the cloud server. The received mode information is the output of step 9.

[0903] Step 10:

[0904] The server generates appropriate content using the generative AI model based on the received mode information, which is the output of step 9. For example, in "conversation partner mode," it generates appropriate conversation content for the user, and in "game mode," it generates games and quizzes that can be enjoyed together with the user. The generated content is the output of step 10.

[0905] Step 11:

[0906] The server transmits the generated content, which is the output of step 10, to the user's terminal. The transmitted content becomes the input of step 11.

[0907] Step 12:

[0908] The terminal receives the generated content, which is the input of step 11, and displays it on the screen. This displayed content is the output of step 12, and the user can view, manipulate, and enjoy it.

[0909] Step 13:

[0910] Users can set the AI's characteristics and direction through the application's training screen, and this information is used as input.

[0911] Step 14:

[0912] The terminal transmits the development setting information, which is the input of step 13, to the cloud server. The transmitted development setting information is the output of step 14.

[0913] Step 15:

[0914] The server automatically adjusts the response and proposal content of the generated AI model based on the training setting information output in step 14. These adjusted response and proposal content are the output of step 15.

[0915] Step 16:

[0916] The user selects an additional feature on the billing page within the application, and the selected billing information is input.

[0917] Step 17:

[0918] The terminal transmits the billing information, which is the input of step 16, to the cloud server. The transmitted billing information is the output of step 17.

[0919] Step 18:

[0920] The server receives the billing information output from step 17 and passes it to the payment gateway for payment processing. If the payment is successful, it applies additional features to the user account. The applied additional features are output from step 18.

[0921] Step 19:

[0922] The terminal displays the applied additional function, which is the output of step 18, to the user and makes it available for use. The user can use the new function through the terminal application.

[0923] (Application example 1)

[0924] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0925] Maintaining a healthy diet and managing your health can be challenging in today's hectic lifestyles. Getting accurate advice based on your health status and individual dietary preferences is particularly challenging, and there are no systems that can directly order meals. Conventional food delivery systems simply deliver the meals selected by the user, but are unable to provide suggestions or advice based on the user's health status. Furthermore, there is a lack of systems that can quickly provide appropriate advice and even order meals on the spot.

[0926] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0927] In this invention, the server includes means for inputting consultation details from a user, means for transmitting the input consultation details to a cloud server, means for analyzing the consultation details on the cloud server and passing them as input to a generative AI model, means for the generative AI model to generate advice and suggestions based on the consultation details, means for providing the generated advice to the user, means for generating meal suggestions based on health status, and means for ordering the suggested meals from a food delivery system. This enables the user to quickly receive appropriate advice based on their health status and then order the suggested meals from a food delivery system.

[0928] A "user" is someone who uses this system to input their consultation details and receive health advice and dietary suggestions.

[0929] The "cloud server" is a remote server that receives and analyzes the consultation content sent by users via the Internet and generates advice and suggestions using a generative AI model.

[0930] "Consultation content" refers to worries, questions, and requests about health and diet entered by the user.

[0931] A "generative AI model" is an artificial intelligence (AI) model that runs on a cloud server and generates advice and suggestions based on the input consultation content.

[0932] "Advice and suggestions" are specific recommendations and instructions generated by the generative AI model based on the user's consultation.

[0933] "Health status" refers to information about the user's physical condition, food preferences, and nutritional balance.

[0934] "Meal suggestions" are specific meal menu suggestions recommended by the generative AI model based on the user's health status.

[0935] A "food delivery system" is an online service that allows users to order and have food delivered from a specified menu.

[0936] This system allows users to consult about their health and diet using devices such as smartphones and tablets, and receives appropriate advice and suggestions in cooperation with a cloud server. Furthermore, based on the suggestions, users can order meals via a food delivery system.

[0937] Overall system configuration

[0938] The system mainly consists of the following components:

[0939] 1. User Device

[0940] 2. Cloud Server

[0941] 3. Generative AI Models

[0942] 4. Food delivery system

[0943] Operation on the user device

[0944] Users start up a dedicated application on their smartphone or tablet and first input their health and dietary concerns by voice or text. This input is then sent from the device to a cloud server.

[0945] Analysis on a cloud server

[0946] The cloud server analyzes the consultation content received from the user, converts it into an appropriate format, and passes it to a generative AI model. For example, OpenAI's GPT-3 (generative AI model) is used as the generative AI model. This model generates advice and suggestions based on the user's consultation content.

[0947] Generate advice and suggestions

[0948] The generative AI model analyzes the consultation content sent from the cloud server and generates appropriate advice and suggestions. For example, it may suggest an appropriate diet based on the user's health condition. This generated advice and suggestions are then sent back to the user's device via the cloud server.

[0949] Meal suggestions based on health status

[0950] The cloud server generates meal suggestions appropriate for the user's health condition based on the advice generated by the generative AI model. For example, the advice generated might be, "Try to eat a balanced meal centered around vegetables today."

[0951] Ordering from a food delivery system

[0952] The user terminal transmits a meal order to the food delivery system based on the suggestions, which includes the meal menu, delivery address, payment method, etc. specified by the user.

[0953] Specific examples

[0954] For example, if a user types into their device, "What is the best meal for my health today?", the cloud server passes this information to the generative AI model. The generative AI model generates advice such as, "Try to eat a balanced meal with a focus on vegetables today." The cloud server then uses this advice to suggest a "healthy plate of grilled vegetables and chicken" and orders this meal through a food delivery system.

[0955] Prompt Sentence Examples

[0956] "Provide appropriate meal suggestions based on the user's health status. Advice: 'Aim for a balanced meal with a focus on vegetables today.'"

[0957] As described above, the present invention is a system that supports the user's health management and promptly suggests and delivers appropriate meals.

[0958] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0959] Step 1:

[0960] The user launches a dedicated application on their smartphone or tablet and inputs the details of their consultation by voice or text. The inputted details are sent from the device to the cloud server. The inputted data is sent in the form of text data.

[0961] Step 2:

[0962] The cloud server analyzes the received consultation content and converts it into an appropriate format. Specifically, it uses natural language processing to analyze the text data and converts it into a format that can be understood by a generative AI model (e.g., OpenAI's GPT-3). This process involves extracting the intent and keywords of the consultation content. The input is the text data of the user's consultation content, and the output is data formatted for the generative AI model.

[0963] Step 3:

[0964] A generative AI model on a cloud server generates advice and suggestions based on the formatted consultation content. The generative AI model uses an internal algorithm to analyze the user's consultation content and construct optimal advice and suggestions. In this process, the advice is generated using prompt text. The input is the formatted consultation content data, and the output is the generated advice or suggestion text.

[0965] Step 4:

[0966] The generated advice and suggestions are then sent to the user's device via the cloud server. The cloud server converts the output data from the generative AI model into a format that is easy for the user to understand and sends it. The input is the advice and suggestion text generated by the generative AI model, and the output is the advice text sent to the user's device.

[0967] Step 5:

[0968] The user device displays the advice and suggestions received from the cloud server. The user can check this through the application. Specifically, the received text data is displayed on the screen. The input is the advice text from the cloud server, and the output is the advice content displayed on the device.

[0969] Step 6:

[0970] The cloud server generates meal suggestions suitable for the user's health condition based on the advice created by the generative AI model. Specifically, it generates prompt sentences based on the generated advice and makes food suggestions. The input is text data of the advice content, and the output is text data of the meal suggestions.

[0971] Step 7:

[0972] The user terminal receives the meal suggestions sent from the cloud server and sends the order directly to the food delivery system. Specifically, it generates order information including the suggested meal menu, delivery address, and payment method and sends it to the food delivery system. The input is text data of the meal suggestions, and the output is order data sent to the food delivery system.

[0973] Through the above steps, the user can receive appropriate advice for health management and order meals based on that advice.

[0974] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0975] This invention is a system that combines a generative AI model and an emotion engine to support users' daily lives, provide appropriate advice for consultations, and respond based on emotion recognition. Specific embodiments of this system are described below.

[0976] Process to receive consultation details from the user

[0977] The user launches the application on their smartphone or tablet and inputs their consultation details by voice or text. The device then sends the input consultation details and voice data to a cloud server. The cloud server analyzes the received data, converts it into an appropriate format, and passes it to the generative AI model and emotion engine.

[0978] Emotion recognition and advice generation using an emotion engine

[0979] On the cloud server, the emotion engine recognizes emotions from the consultation content and voice data entered by the user. The recognized emotion information is reflected in the generative AI model, which then generates advice and suggestions based on the user's emotions. The generated advice is then sent back from the cloud server to the user's device and provided to the user via an application.

[0980] Providing daily conversation partners and entertainment

[0981] When a user selects "conversation partner mode" or "game mode" in the application, the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and emotions and sends it to the user's device. In "conversation partner mode," the emotion engine generates and provides conversation content based on the emotions recognized by the emotion engine. In "game mode," the difficulty and content of the game are adjusted according to the user's emotional state.

[0982] AI training function

[0983] Users can set the AI's characteristics and direction through the application's training screen. Training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. By combining emotion engines, the user's emotional state is also taken into account, allowing for the provision of optimal services tailored to each individual user.

[0984] Revenue model and billing process

[0985] This system also provides a means for users to select additional features and for the cloud server to process billing based on that information. The user selects the additional features on a billing page within the application, and that information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[0986] Specific examples

[0987] For example, if User A complains of a "pain in the lower back," the cloud server passes the information to the emotion engine. The emotion engine recognizes "anxiety" from User A's voice data and provides this information to the generative AI model. The generative AI model then generates a message that takes User A's emotions into consideration, such as "Try some moderate stretching. Specifically, try these movements," along with advice such as "Try it slowly, without overdoing it," and sends it to User A's device.

[0988] If User B thinks, "I'm feeling a little lonely today," and selects "Conversational Partner Mode," the emotion engine will recognize this feeling. The generative AI model will start the conversation with something like, "What did you do today?", and then generate content that takes emotion into consideration, such as, "Shall we have a fun conversation?", and provide it to User B.

[0989] In this way, the system of the present invention provides a concrete means for elderly users and single-person household users to lead richer daily lives.

[0990] The processing flow will be explained below.

[0991] Process to receive consultation details from the user

[0992] Step 1:

[0993] The user starts the smartphone app and inputs the details of their consultation. The user can input the details of their consultation by voice or text.

[0994] Step 2:

[0995] The device converts the input consultation content and voice data into JSON format and sends it to the cloud server via an HTTP POST request.

[0996] AI-based consultation processing and emotion recognition

[0997] Step 3:

[0998] The server parses the JSON data of the received consultation content and voice data and converts it into a specified format.

[0999] Step 4:

[1000] The server passes the consultation content after format conversion to the generative AI model and emotion engine.

[1001] Step 5:

[1002] The emotion engine recognizes emotions from the user's voice data and text. For example, emotions such as "anxiety" and "loneliness" can be recognized through voice analysis.

[1003] Step 6:

[1004] The generative AI model generates advice and suggestions based on the recognized emotional information and the content of the consultation. Specifically, in addition to advice such as "Try some moderate stretching. Specifically, try these movements," it generates messages that take the user's emotions into consideration, such as "Try it little by little without overdoing it."

[1005] Step 7:

[1006] The server converts the generated advice and emotion-sensitive messages into JSON format and sends them to the user's device.

[1007] Step 8:

[1008] The device parses the received advice and messages and displays them to the user within the application, and may also use speech synthesis to communicate them aloud if necessary.

[1009] Providing daily conversation partners and entertainment

[1010] Step 1:

[1011] The user selects either "talkie mode" or "game mode" in the application.

[1012] Step 2:

[1013] The device converts the selected mode information into JSON format and sends it to the cloud server via an HTTP POST request.

[1014] Step 3:

[1015] The server invokes a generative AI model and an emotion engine based on the received mode information.

[1016] Step 4:

[1017] The emotion engine recognizes emotions from the user's voice data and text and provides that information to the generative AI model.

[1018] Step 5:

[1019] The generative AI model generates appropriate conversation content in conversation partner mode and appropriate game content in game mode based on emotion information and mode information from the emotion engine.

[1020] Step 6:

[1021] The server converts the generated conversation content or game content into JSON format and sends it to the user's device.

[1022] Step 7:

[1023] In the conversation partner mode, the terminal displays the generated conversation content, and in the game mode, the terminal starts the generated game so that the user can play together with the user.

[1024] AI training function

[1025] Step 1:

[1026] Users set the AI's characteristics and direction through the application's development screen.

[1027] Step 2:

[1028] The device converts the development setting information into JSON format and sends it to the cloud server via an HTTP POST request.

[1029] Step 3:

[1030] The server automatically adjusts the responses and suggestions of the generative AI model based on the received training setting information, taking into account information from the emotion engine.

[1031] Revenue model and billing process

[1032] Step 1:

[1033] The user selects additional features from the application's billing page.

[1034] Step 2:

[1035] The device converts the purchase information into JSON format and sends it to the cloud server via an HTTP POST request.

[1036] Step 3:

[1037] The server calls a payment gateway based on the received purchase information and executes the payment process.

[1038] Step 4:

[1039] If the payment is successful, the server applies the additional features to the user's account and sends the information to the user's device via an HTTP POST request.

[1040] Step 5:

[1041] Based on the received additional function information, the terminal performs settings so that the user can use the function within the application.

[1042] Example 2

[1043] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1044] In modern society, users face various worries and problems in their daily lives, but it is difficult to quickly obtain appropriate advice. Furthermore, there is a lack of appropriate responses that take emotions into consideration, and people with whom people can easily talk, leading to feelings of loneliness. To address these issues, there is a need for a system that can understand users' emotions and provide appropriate advice and entertainment.

[1045] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1046] In this invention, the server includes means for inputting consultation content from a user, means for transmitting the input consultation content to a cloud server, means for analyzing the consultation content on the cloud server, converting it into an appropriate format, and passing it as input to the generative AI model and emotion identification device, means for the emotion identification device to recognize emotions from the consultation content and voice data, means for the generative AI model to generate advice and suggestions based on the recognized emotion information, and means for providing the generated advice to the user from the cloud server. This allows users to receive appropriate advice and entertainment according to their emotions at any given time, thereby improving the quality of their daily lives.

[1047] "User" refers to an individual who uses the system to input their inquiry and receive advice and suggestions.

[1048] "Consultation content" refers to the worries, questions, and other inquiries that the user inputs into the system by voice or text.

[1049] A "cloud server" is a server that can be accessed remotely via the Internet and is a central element of a system that analyzes and processes data.

[1050] A "generative AI model" is a generative model created by artificial intelligence, and refers to a program that creates advice and suggestions based on user input and emotional information.

[1051] An "emotion identification device" refers to a device or program that recognizes emotions from user input or voice data and reflects them in the analysis results.

[1052] "Advice and suggestions" refers to specific advice and recommendations generated by the generative AI model based on the user's consultation content and emotional information.

[1053] The "conversation partner mode" refers to a mode in which the user selects how the system will respond as a conversation partner, and conversation content based on emotions is provided.

[1054] "Play mode" refers to a mode in which the user selects a mode that provides games and entertainment content to the system, and the difficulty and content of the game are adjusted according to the user's emotional state.

[1055] "Setting the characteristics and direction of artificial intelligence" refers to the operation in which the user sets specific parameters through a settings screen within the system to adjust the response and proposal content of the generated AI model.

[1056] "Payment gateway" refers to an interface through which the cloud server processes the user's payment information and performs billing processing.

[1057] This invention is a system that allows users to input their consultation details using an application on a smartphone or tablet and receive appropriate advice and suggestions in response. The system operates on a cloud server using a generative AI model and an emotion recognition device. A specific embodiment of this system is described below.

[1058] The user launches an application installed on their smartphone or tablet and inputs the consultation details by voice or text. The device then sends the input consultation details to a cloud server. This transmission uses a communication protocol (e.g., HTTP, HTTPS) via an internet connection.

[1059] The cloud server parses the received data and converts it into an appropriate format (e.g., JSON format) using, for example, a natural language processing library or speech recognition technology (e.g., Google Speech-to-Text API).The converted data is then passed to a generative AI model and an emotion recognition device.

[1060] The emotion recognition device recognizes emotions from the consultation content and voice data entered by the user. The technology used can be, for example, IBM Watson's Natural Language Understanding. The emotion recognition device provides the recognized emotional information to the generative AI model.

[1061] The generative AI model generates advice and suggestions for the user based on this emotional information. For example, OpenAI's GPT model is used for generation. The generative AI model generates specific suggestions and response messages that reflect the user's emotional state.

[1062] The generated advice and suggestions are sent from the cloud server to the user's device. The device then provides the received advice to the user through an application. The advice is displayed using a visually easy-to-understand interface (e.g., text message, pop-up message).

[1063] For example, if a user complains of a "pain in the lower back," the cloud server passes the information to an emotion recognition device. The emotion recognition device recognizes "anxiety" from the voice data. This information is provided to a generative AI model, which then generates advice such as "Try some moderate stretching. Specifically, try the following movements," along with a message that takes the user's emotions into consideration, such as "Try doing it gradually, without overdoing it." The message is then sent to the user's device.

[1064] Furthermore, if the user selects "conversation partner mode," the emotion recognition device recognizes "loneliness." In this case, the generative AI model generates emotion-sensitive content, such as "What did you do today?" and "Shall we have a little fun conversation?", and provides these to the user.

[1065] An example of a specific prompt is as follows: "If a user complains of lower back pain, what advice would the emotion recognizer generate for the generative AI model by recognizing anxiety from the voice data?"

[1066] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1067] Step 1:

[1068] The user launches the application on their smartphone or tablet. Here, the user inputs the details of their consultation by voice or text. The input data is temporarily stored within the device. An example of input data is the voice input of "My lower back hurts."

[1069] Step 2:

[1070] The device converts the voice data entered by the user into text data. This conversion uses voice recognition technology (e.g., Google Speech-to-Text API). The converted text data becomes "My lower back hurts." This text data is then sent to a cloud server. HTTP or HTTPS is used as the transmission protocol.

[1071] Step 3:

[1072] The server parses the received text data using a natural language processing library (e.g., NLTK). The parsed data is then converted into an appropriate format (e.g., JSON). This formatted data is then passed as input to the generative AI model and emotion recognition device.

[1073] Step 4:

[1074] The emotion recognition device recognizes emotions from the user's text data. The technology used is, for example, IBM Watson's Natural Language Understanding. From the input data "My back hurts," the emotion "anxiety" is recognized. This recognized emotion data is provided to the generative AI model.

[1075] Step 5:

[1076] The generative AI model generates advice and suggestions for the user based on the emotional information received from the emotion recognition device. The generative AI model uses OpenAI's GPT model. Based on the input data "My lower back hurts" and the emotional information "anxiety," the advice message generated is "Try some moderate stretching. Specifically, do the following movements." This generated advice is sent back to the cloud server.

[1077] Step 6:

[1078] The server sends the advice received from the generative AI model to the user's device using HTTP or HTTPS as the transmission protocol. The input data includes the generated advice message.

[1079] Step 7:

[1080] The device provides the user with the advice received from the server. The advice is displayed as a text message on the application interface. Specifically, the message displayed is, "Try some moderate stretching. Don't push yourself too hard, and try it little by little."

[1081] Step 8 (optional):

[1082] To use additional features, the user selects them on the billing page within the application, and this selection information is sent from the terminal to the cloud server.

[1083] Step 9 (Optional):

[1084] The server processes the payment via a payment gateway (e.g., Stripe API). If the payment is successful, the add-on is applied to the user's account. Input data includes the user's payment information and the selected add-on.

[1085] (Application example 2)

[1086] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1087] In conventional store operations, in-store staff are responsible for all customer interactions and service provision, which results in a heavy workload. Furthermore, it is difficult to respond flexibly to customers' emotions and moods, creating a need for a system that can provide optimal service to individual customers. In particular, there is a growing need for a system that can recognize customer emotions and automatically generate appropriate advice and services based on them.

[1088] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1089] In this invention, the server includes means for inputting consultation content from a user, means for transmitting the input consultation content to a cloud server, means for analyzing the consultation content on the cloud server and passing it as input to a generative AI model, means for the generative AI model to generate advice or suggestions based on the consultation content, means for providing the generated advice to the user, means for recognizing the user's emotions using a camera, means for transmitting the emotion recognition result to the cloud server, means for the generative AI model to generate appropriate advice or suggestions for the user based on the emotion recognition result, and means for providing the advice or suggestions generated by an in-store robot. This makes it possible to efficiently handle customer service in the store and provide optimal services according to customer emotions.

[1090] A "user" is an individual or group that uses the system and is the entity that inputs the consultation content and voice data.

[1091] "Consultation content" refers to information such as questions, requests, and opinions that users provide to the system.

[1092] A "cloud server" is a computer system that provides computing resources that are remotely accessible via the Internet.

[1093] A "generative AI model" is an artificial intelligence algorithm that generates advice or suggestions based on specific input data (such as consultation content or emotion recognition results).

[1094] A "camera" is a device for capturing images and videos, and is used to recognize the user's emotions.

[1095] "Emotion recognition" is the process of analyzing information such as a user's facial expressions and voice to identify their emotional state (e.g., joy, sadness, anxiety, etc.).

[1096] A "robot" is a mechanical device that is programmed to work automatically or semi-automatically and perform specific tasks.

[1097] "Advice" is specific instructions or suggestions provided by the generative AI model based on the user's consultation content and emotions.

[1098] "Suggestions" are suggestions to the user for options or strategies to consider, generated based on the consultation and perceived emotions.

[1099] A "store" is a physical business location that offers goods and services to consumers.

[1100] "Staff" refers to the people who work in the store, dealing with customers and performing other tasks.

[1101] This invention is a system that combines generative AI models and emotion recognition technology to improve customer service and operational efficiency in brick-and-mortar stores. To realize this system, the following hardware and software are used to process and calculate data.

[1102] First, users (customers or staff) can input their concerns through a robot in a physical store. Input is done by voice or text using the robot's microphone or touch panel. The robot then sends the inputted concerns and voice data to a cloud server.

[1103] The cloud server analyzes the received data and passes it as input to the generative AI model. For emotion recognition, a camera mounted on the robot captures the user's facial expression and sends the image data to the cloud server. The cloud server then uses emotion recognition software (e.g., EmotionRecognition library) to recognize the user's emotion from the sent image data.

[1104] The recognized emotional information is fed into a generative AI model (e.g., GPT-2 model), which generates advice and suggestions based on the user's emotions and the content of the consultation. The generated advice and suggestions are then provided to the user from the cloud server via the robot's display and speakers.

[1105] For example, if a user says, "I'm not in the mood today," the robot will analyze their facial expressions and tone of voice and send them to the cloud server. If the cloud server recognizes the emotion and determines that the user is "tired," it will use the generative AI model to generate advice such as "I'll play some relaxing music," and provide it to the user via the robot.

[1106] Usage examples and prompt statements

[1107] Here is an example prompt:

[1108] The customer said: "I'm tired today." They seem to be feeling tired. What advice would be helpful for them?

[1109] In this way, the system of the present invention can improve the efficiency of customer service in stores and provide optimal services according to customer emotions.By using a generative AI model, it is possible to flexibly generate advice and suggestions customized for each individual user.

[1110] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1111] Step 1:

[1112] The user inputs the content of the consultation through the robot. The content of the consultation, input in voice or text format, is captured using the robot's microphone and touch panel.

[1113] Input: Consultation content (voice or text)

[1114] Output: The consultation content converted into data format by the robot

[1115] Step 2:

[1116] The robot sends the acquired consultation information to a cloud server using an internet connection.

[1117] Input: Consultation content converted into data format

[1118] Output: Data sent to the cloud server

[1119] Step 3:

[1120] The cloud server analyzes the received consultation content and starts emotion recognition.

[1121] Input: Consultation details sent

[1122] Output: Analysis results and data for emotion recognition

[1123] Step 4:

[1124] The cloud server uses the image data sent from the robot to recognize the user's emotions using emotion recognition software.

[1125] Input: Image data

[1126] Output: Recognized emotion information (e.g., joy, sadness, anxiety)

[1127] Step 5:

[1128] The cloud server inputs the consultation content and emotion recognition results into a generative AI model to generate appropriate advice and suggestions for the user.

[1129] Input: Consultation content, recognized emotion information

[1130] Output: Advice or suggestions generated by the generative AI model

[1131] Step 6:

[1132] The cloud server then sends the generated advice and suggestions back to the robot.

[1133] Input: Generated advice and suggestions

[1134] Output: Data sent to the robot

[1135] Step 7:

[1136] The robot provides the user with advice and suggestions received from the cloud server, which are communicated through the robot's display and speakers.

[1137] Input: Advice and suggestions received from the cloud server

[1138] Output: Advice and suggestions presented to the user

[1139] Step 8:

[1140] If desired, the user can interact with the robot again to provide additional consultation or feedback, and the process repeats.

[1141] Input: Additional consultation details, feedback

[1142] Output: Updated consultation details and feedback information

[1143] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1144] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1145] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1146] [Fourth embodiment]

[1147] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1148] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1149] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1150] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1151] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1152] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1153] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1154] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1155] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1156] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1157] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1158] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1159] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1160] This invention provides a system that allows users to use an application linked to a cloud server to seek advice about health conditions or lifestyle concerns or to enjoy entertainment. Specific embodiments of this system will be described below.

[1161] Process to receive consultation details from the user

[1162] First, the user launches the application on their smartphone or tablet and inputs their consultation details by voice or text. The application then sends the input to a cloud server, which analyzes the received data, converts it into an appropriate format, and passes it to the generative AI model.

[1163] AI responds to inquiries

[1164] On the cloud server, the generative AI model analyzes the consultation content received from the user and generates appropriate advice and suggestions. The generated advice is then sent back from the cloud server to the user's device and provided to the user via an application. This process occurs in real time, allowing users to receive advice quickly.

[1165] Providing daily conversation partners and entertainment

[1166] When a user selects "conversation partner mode" or "game mode" in an application, the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and sends it to the user's device. In "conversation partner mode," the generative AI model generates conversation content with the user and displays it in the application. In "game mode," game content is provided that can be enjoyed together with the user.

[1167] AI training function

[1168] Users can set the AI's characteristics and direction through the application's training screen. The training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. This allows users to receive personalized responses tailored to their preferences.

[1169] Revenue model and billing process

[1170] This system also provides a means for users to select additional features and for the cloud server to process billing based on that information. The user selects the additional features on a billing page within the application, and that information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[1171] Specific examples

[1172] For example, if User A complains of a sore lower back, the cloud server passes the information to the generative AI model. The generative AI model generates advice such as "Try some moderate stretching. Specifically, try these movements," and sends it to User A's device. Similarly, if User B thinks, "I'm feeling a little lonely today," and selects "Conversation mode," the generative AI model generates conversation content such as "What did you do today?" and provides it to User B.

[1173] In this way, the system of the present invention provides a concrete means for elderly users and single-person household users to lead richer daily lives.

[1174] The processing flow will be explained below.

[1175] Process to receive consultation details from the user

[1176] Step 1:

[1177] The user launches the app on their smartphone and inputs the details of their consultation either by voice or text.

[1178] Step 2:

[1179] The terminal converts the input consultation content into JSON format and sends an HTTP POST request to the cloud server.

[1180] AI responds to inquiries

[1181] Step 3:

[1182] The server parses the received JSON data of the consultation content and converts it into a specified format.

[1183] Step 4:

[1184] The server passes the consultation content after format conversion as input to the generative AI model.

[1185] Step 5:

[1186] A generative AI model generates advice and suggestions based on the consultation content.

[1187] Step 6:

[1188] The server converts the generated advice into JSON format and sends it to the user's device.

[1189] Step 7:

[1190] The device parses the received advice and displays it to the user within the application, and may also use speech synthesis to communicate it aloud if necessary.

[1191] Providing daily conversation partners and entertainment

[1192] Step 1:

[1193] The user selects either "talkie mode" or "game mode" in the application.

[1194] Step 2:

[1195] The device converts the selected mode information into JSON format and sends it to the cloud server via an HTTP POST request.

[1196] Step 3:

[1197] The server invokes a generative AI model or a game content generation system based on the received mode information.

[1198] Step 4:

[1199] In the conversation partner mode, the server uses the generative AI model to generate appropriate conversation content and send it to the user's device.In the game mode, the server generates game content and sends it to the user's device.

[1200] Step 5:

[1201] In the conversation partner mode, the terminal displays the generated conversation content, and in the game mode, the terminal starts a game so that the user can play together.

[1202] AI training function

[1203] Step 1:

[1204] Users set the AI's characteristics and direction through the application's development screen.

[1205] Step 2:

[1206] The device converts the development setting information into JSON format and sends it to the cloud server via an HTTP POST request.

[1207] Step 3:

[1208] The server automatically adjusts the response and proposal content of the generated AI model based on the received training setting information.

[1209] Revenue model and billing process

[1210] Step 1:

[1211] The user selects additional features from the application's billing page.

[1212] Step 2:

[1213] The device converts the purchase information into JSON format and sends it to the cloud server via an HTTP POST request.

[1214] Step 3:

[1215] The server calls a payment gateway based on the received purchase information and executes the payment process.

[1216] Step 4:

[1217] If the payment is successful, the server applies the additional function to the user's account and notifies the user's terminal of that information.

[1218] Step 5:

[1219] Based on the received information about the additional function, the terminal performs settings so that the user can use the function within the application.

[1220] Example 1

[1221] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1222] In modern life, people are looking for fast and effective solutions to their health and daily worries. Furthermore, with the increase in elderly people and single-person households, there is a need for mental health care and entertainment for those living alone. Systems that provide fast and personalized responses to these issues are lacking, and technological solutions are needed to meet user needs.

[1223] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1224] In this invention, the server includes means for inputting consultation details from a user, means for transmitting the input consultation details to a cloud server, means for the cloud server to analyze the received data, convert it into an appropriate format, and pass it to the generative AI model, means for the generative AI model to generate advice and suggestions based on the consultation details, and means for transmitting the generated advice again from the cloud server to the user's terminal and providing it to the user through an application. This enables users to receive quick and personalized advice and suggestions in real time, providing a powerful means for elderly people and single-person households to live richer daily lives.

[1225] "User" refers to a person who uses this system to input consultation details, select modes, and set up AI training.

[1226] "Consultation content" refers to information entered by the user regarding their health condition or concerns about their daily life.

[1227] A "cloud server" refers to a server that performs data analysis, sends data to generative AI models, and receives responses via the Internet.

[1228] A "generative AI model" refers to an artificial intelligence model that generates advice and suggestions based on the input consultation content.

[1229] "Application" refers to software that runs on the user's smartphone or tablet and allows them to input consultation details, select modes, and set up AI training.

[1230] "Conversation partner mode" refers to a mode in which the cloud server uses a generated AI model to provide the content of the conversation to the user.

[1231] "Game mode" refers to a mode in which the cloud server uses a generated AI model to provide users with entertainment content such as games and quizzes.

[1232] "Setting the AI's characteristics and direction" refers to the user making settings through the application to customize the responses and suggestions of the generated AI model.

[1233] "Training setting information" refers to information about the characteristics and direction of the AI ​​set by the user.

[1234] "Payment gateway" refers to an Internet service that allows a server to process payments based on billing information from users.

[1235] This invention provides a system that allows users to use an application linked to a cloud server to seek advice about health conditions or lifestyle concerns or to enjoy entertainment. Specific embodiments of this system will be described below.

[1236] Required Hardware and Software

[1237] User device: A computing device such as a smartphone or tablet.

[1238] Application: Software installed on a device that provides a user interface.

[1239] Cloud server: A server that performs data analysis and runs generative AI models via the internet.

[1240] Generative AI model: An artificial intelligence model that generates advice and suggestions based on the content of a consultation.

[1241] System Operation

[1242] First, the user launches the application on their smartphone or tablet and inputs the content of their consultation by voice or text. At this time, the voice data input by the user is converted into text data within the application. The application then sends the input content of the consultation to a cloud server. The cloud server analyzes the received data and converts it into a format that is easy for the generative AI model to understand.

[1243] The generative AI model on the cloud server analyzes the consultation content received from the user and generates appropriate advice and suggestions. The generated advice is sent from the cloud server to the user's device and provided to the user through an application. This process is carried out in real time, allowing users to receive advice quickly.

[1244] For example, if a user enters the text "My lower back hurts" and taps the send button, the device sends the input data to the cloud server. The cloud server analyzes the received data and passes it to the generative AI model. The generative AI model generates advice such as "Try some moderate stretching. Specifically, try these movements," and the cloud server sends the generated advice to the user's device. The device decodes the received advice and displays it on the screen. The user then checks the advice displayed on the screen.

[1245] Furthermore, when the user selects "conversation partner mode" or "game mode," the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and sends it to the user's device. In "conversation partner mode," the generative AI model generates conversation content with the user and displays it in the application. In "game mode," game content that can be enjoyed together with the user is provided.

[1246] Furthermore, users can set the AI's characteristics and direction through the application's training screen. The training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. This allows users to receive personalized responses tailored to their preferences.

[1247] As a revenue model, a method is also provided in which the user selects additional features and the cloud server processes billing based on that information. The user selects the additional features on the billing page within the application and the information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[1248] Examples of prompt statements

[1249] Consultation prompt:

[1250] Tell the AI ​​the following questions to generate appropriate advice:

[1251] Consultation: "My lower back hurts"

[1252] Interactive mode input prompt:

[1253] Continue the conversation with the user as follows:

[1254] Conversation topic: "I'm feeling a little lonely today"

[1255] Game mode input prompt:

[1256] Give your users a fun and easy quiz.

[1257] Theme: "General Knowledge"

[1258] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1259] Step 1:

[1260] The user launches the application on their smartphone or tablet and inputs the content of their consultation by voice or text. The input content is converted into text data by the application. After the user's voice input is converted into text, text data is generated. This is the output of Step 1.

[1261] Step 2:

[1262] The device transmits the text data, which is the output of step 1, to the cloud server. The transmitted text data is received by the cloud server. The data received by the cloud server is the input of step 2.

[1263] Step 3:

[1264] The server analyzes the received text data (input in step 2) and converts it into a format that is easy for the generative AI model to understand. For example, it converts it into JSON format. This format conversion is performed as data processing, and as a result, prepared data is generated to be passed to the generative AI model. This is the output of step 3.

[1265] Step 4:

[1266] The server passes the format-converted data, which is the output of step 3, to the generative AI model. The passed data becomes the input for the generative AI model, and at this stage the generative AI model begins analysis.

[1267] Step 5:

[1268] The generative AI model analyzes the format-converted data, which is the input in step 4, and generates appropriate advice and suggestions. Here, advice generated based on the prompt sentence is output. For example, for the input "My lower back hurts," specific advice such as "Try some moderate stretching" is generated.

[1269] Step 6:

[1270] The server sends the generated advice, which is the output of step 5, to the user's device. At this stage, the data is again encoded and transmitted securely using the appropriate communication protocol. The transmitted data is the input here.

[1271] Step 7:

[1272] The terminal receives and decodes the generated advice, which is the input of step 6. It displays the received advice to the user. This decoded advice is the output of step 7, and the user sees the information on the terminal screen.

[1273] Step 8:

[1274] The user selects "conversation mode" or "game mode" within the application. The selected mode information is input.

[1275] Step 9:

[1276] The terminal transmits mode information to the cloud server, which is the input of step 8. The transmitted mode information is received by the cloud server. The received mode information is the output of step 9.

[1277] Step 10:

[1278] The server generates appropriate content using the generative AI model based on the received mode information, which is the output of step 9. For example, in "conversation partner mode," it generates appropriate conversation content for the user, and in "game mode," it generates games and quizzes that can be enjoyed together with the user. The generated content is the output of step 10.

[1279] Step 11:

[1280] The server transmits the generated content, which is the output of step 10, to the user's terminal. The transmitted content becomes the input of step 11.

[1281] Step 12:

[1282] The terminal receives the generated content, which is the input of step 11, and displays it on the screen. This displayed content is the output of step 12, and the user can view, manipulate, and enjoy it.

[1283] Step 13:

[1284] Users can set the AI's characteristics and direction through the application's training screen, and this information is used as input.

[1285] Step 14:

[1286] The terminal transmits the development setting information, which is the input of step 13, to the cloud server. The transmitted development setting information is the output of step 14.

[1287] Step 15:

[1288] The server automatically adjusts the response and proposal content of the generated AI model based on the training setting information output in step 14. These adjusted response and proposal content are the output of step 15.

[1289] Step 16:

[1290] The user selects an additional feature on the billing page within the application, and the selected billing information is input.

[1291] Step 17:

[1292] The terminal transmits the billing information, which is the input of step 16, to the cloud server. The transmitted billing information is the output of step 17.

[1293] Step 18:

[1294] The server receives the billing information output from step 17 and passes it to the payment gateway for payment processing. If the payment is successful, it applies additional features to the user account. The applied additional features are output from step 18.

[1295] Step 19:

[1296] The terminal displays the applied additional function, which is the output of step 18, to the user and makes it available for use. The user can use the new function through the terminal application.

[1297] (Application example 1)

[1298] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1299] Maintaining a healthy diet and managing your health can be challenging in today's hectic lifestyles. Getting accurate advice based on your health status and individual dietary preferences is particularly challenging, and there are no systems that can directly order meals. Conventional food delivery systems simply deliver the meals selected by the user, but are unable to provide suggestions or advice based on the user's health status. Furthermore, there is a lack of systems that can quickly provide appropriate advice and even order meals on the spot.

[1300] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1301] In this invention, the server includes means for inputting consultation details from a user, means for transmitting the input consultation details to a cloud server, means for analyzing the consultation details on the cloud server and passing them as input to a generative AI model, means for the generative AI model to generate advice and suggestions based on the consultation details, means for providing the generated advice to the user, means for generating meal suggestions based on health status, and means for ordering the suggested meals from a food delivery system. This enables the user to quickly receive appropriate advice based on their health status and then order the suggested meals from a food delivery system.

[1302] A "user" is someone who uses this system to input their consultation details and receive health advice and dietary suggestions.

[1303] The "cloud server" is a remote server that receives and analyzes the consultation content sent by users via the Internet and generates advice and suggestions using a generative AI model.

[1304] "Consultation content" refers to worries, questions, and requests about health and diet entered by the user.

[1305] A "generative AI model" is an artificial intelligence (AI) model that runs on a cloud server and generates advice and suggestions based on the input consultation content.

[1306] "Advice and suggestions" are specific recommendations and instructions generated by the generative AI model based on the user's consultation.

[1307] "Health status" refers to information about the user's physical condition, food preferences, and nutritional balance.

[1308] "Meal suggestions" are specific meal menu suggestions recommended by the generative AI model based on the user's health status.

[1309] A "food delivery system" is an online service that allows users to order and have food delivered from a specified menu.

[1310] This system allows users to consult about their health and diet using devices such as smartphones and tablets, and receives appropriate advice and suggestions in cooperation with a cloud server. Furthermore, based on the suggestions, users can order meals via a food delivery system.

[1311] Overall system configuration

[1312] The system mainly consists of the following components:

[1313] 1. User Device

[1314] 2. Cloud Server

[1315] 3. Generative AI Models

[1316] 4. Food delivery system

[1317] Operation on the user device

[1318] Users start up a dedicated application on their smartphone or tablet and first input their health and dietary concerns by voice or text. This input is then sent from the device to a cloud server.

[1319] Analysis on a cloud server

[1320] The cloud server analyzes the consultation content received from the user, converts it into an appropriate format, and passes it to a generative AI model. For example, OpenAI's GPT-3 (generative AI model) is used as the generative AI model. This model generates advice and suggestions based on the user's consultation content.

[1321] Generate advice and suggestions

[1322] The generative AI model analyzes the consultation content sent from the cloud server and generates appropriate advice and suggestions. For example, it may suggest an appropriate diet based on the user's health condition. This generated advice and suggestions are then sent back to the user's device via the cloud server.

[1323] Meal suggestions based on health status

[1324] The cloud server generates meal suggestions appropriate for the user's health condition based on the advice generated by the generative AI model. For example, the advice generated might be, "Try to eat a balanced meal centered around vegetables today."

[1325] Ordering from a food delivery system

[1326] The user terminal transmits a meal order to the food delivery system based on the suggestions, which includes the meal menu, delivery address, payment method, etc. specified by the user.

[1327] Specific examples

[1328] For example, if a user types into their device, "What is the best meal for my health today?", the cloud server passes this information to the generative AI model. The generative AI model generates advice such as, "Try to eat a balanced meal with a focus on vegetables today." The cloud server then uses this advice to suggest a "healthy plate of grilled vegetables and chicken" and orders this meal through a food delivery system.

[1329] Prompt Sentence Examples

[1330] "Provide appropriate meal suggestions based on the user's health status. Advice: 'Aim for a balanced meal with a focus on vegetables today.'"

[1331] As described above, the present invention is a system that supports the user's health management and promptly suggests and delivers appropriate meals.

[1332] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1333] Step 1:

[1334] The user launches a dedicated application on their smartphone or tablet and inputs the details of their consultation by voice or text. The inputted details are sent from the device to the cloud server. The inputted data is sent in the form of text data.

[1335] Step 2:

[1336] The cloud server analyzes the received consultation content and converts it into an appropriate format. Specifically, it uses natural language processing to analyze the text data and converts it into a format that can be understood by a generative AI model (e.g., OpenAI's GPT-3). This process involves extracting the intent and keywords of the consultation content. The input is the text data of the user's consultation content, and the output is data formatted for the generative AI model.

[1337] Step 3:

[1338] A generative AI model on a cloud server generates advice and suggestions based on the formatted consultation content. The generative AI model uses an internal algorithm to analyze the user's consultation content and construct optimal advice and suggestions. In this process, the advice is generated using prompt text. The input is the formatted consultation content data, and the output is the generated advice or suggestion text.

[1339] Step 4:

[1340] The generated advice and suggestions are then sent to the user's device via the cloud server. The cloud server converts the output data from the generative AI model into a format that is easy for the user to understand and sends it. The input is the advice and suggestion text generated by the generative AI model, and the output is the advice text sent to the user's device.

[1341] Step 5:

[1342] The user device displays the advice and suggestions received from the cloud server. The user can check this through the application. Specifically, the received text data is displayed on the screen. The input is the advice text from the cloud server, and the output is the advice content displayed on the device.

[1343] Step 6:

[1344] The cloud server generates meal suggestions suitable for the user's health condition based on the advice created by the generative AI model. Specifically, it generates prompt sentences based on the generated advice and makes food suggestions. The input is text data of the advice content, and the output is text data of the meal suggestions.

[1345] Step 7:

[1346] The user terminal receives the meal suggestions sent from the cloud server and sends the order directly to the food delivery system. Specifically, it generates order information including the suggested meal menu, delivery address, and payment method and sends it to the food delivery system. The input is text data of the meal suggestions, and the output is order data sent to the food delivery system.

[1347] Through the above steps, the user can receive appropriate advice for health management and order meals based on that advice.

[1348] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1349] This invention is a system that combines a generative AI model and an emotion engine to support users' daily lives, provide appropriate advice for consultations, and respond based on emotion recognition. Specific embodiments of this system are described below.

[1350] Process to receive consultation details from the user

[1351] The user launches the application on their smartphone or tablet and inputs their consultation details by voice or text. The device then sends the input consultation details and voice data to a cloud server. The cloud server analyzes the received data, converts it into an appropriate format, and passes it to the generative AI model and emotion engine.

[1352] Emotion recognition and advice generation using an emotion engine

[1353] On the cloud server, the emotion engine recognizes emotions from the consultation content and voice data entered by the user. The recognized emotion information is reflected in the generative AI model, which then generates advice and suggestions based on the user's emotions. The generated advice is then sent back from the cloud server to the user's device and provided to the user via an application.

[1354] Providing daily conversation partners and entertainment

[1355] When a user selects "conversation partner mode" or "game mode" in the application, the application sends the selection information to the cloud server. The cloud server generates appropriate content based on the user's selection and emotions and sends it to the user's device. In "conversation partner mode," the emotion engine generates and provides conversation content based on the emotions recognized by the emotion engine. In "game mode," the difficulty and content of the game are adjusted according to the user's emotional state.

[1356] AI training function

[1357] Users can set the AI's characteristics and direction through the application's training screen. Training setting information is sent to a cloud server, and the generated AI model's responses and suggestions are automatically adjusted. By combining emotion engines, the user's emotional state is also taken into account, allowing for the provision of optimal services tailored to each individual user.

[1358] Revenue model and billing process

[1359] This system also provides a means for users to select additional features and for the cloud server to process billing based on that information. The user selects the additional features on a billing page within the application, and that information is sent to the cloud server. The cloud server processes the payment via a payment gateway, and if successful, the additional features are applied to the user's account. This allows the user to use the new features.

[1360] Specific examples

[1361] For example, if User A complains of a "pain in the lower back," the cloud server passes the information to the emotion engine. The emotion engine recognizes "anxiety" from User A's voice data and provides this information to the generative AI model. The generative AI model then generates a message that takes User A's emotions into consideration, such as "Try some moderate stretching. Specifically, try these movements," along with advice such as "Try it slowly, without overdoing it," and sends it to User A's device.

[1362] If User B thinks, "I'm feeling a little lonely today," and selects "Conversational Partner Mode," the emotion engine will recognize this feeling. The generative AI model will start the conversation with something like, "What did you do today?", and then generate content that takes emotion into consideration, such as, "Shall we have a fun conversation?", and provide it to User B.

[1363] In this way, the system of the present invention provides a concrete means for elderly users and single-person household users to lead richer daily lives.

[1364] The processing flow will be explained below.

[1365] Process to receive consultation details from the user

[1366] Step 1:

[1367] The user starts the smartphone app and inputs the details of their consultation. The user can input the details of their consultation by voice or text.

[1368] Step 2:

[1369] The device converts the input consultation content and voice data into JSON format and sends it to the cloud server via an HTTP POST request.

[1370] AI-based consultation processing and emotion recognition

[1371] Step 3:

[1372] The server parses the JSON data of the received consultation content and voice data and converts it into a specified format.

[1373] Step 4:

[1374] The server passes the consultation content after format conversion to the generative AI model and emotion engine.

[1375] Step 5:

[1376] The emotion engine recognizes emotions from the user's voice data and text. For example, emotions such as "anxiety" and "loneliness" can be recognized through voice analysis.

[1377] Step 6:

[1378] The generative AI model generates advice and suggestions based on the recognized emotional information and the content of the consultation. Specifically, in addition to advice such as "Try some moderate stretching. Specifically, try these movements," it generates messages that take the user's emotions into consideration, such as "Try it little by little without overdoing it."

[1379] Step 7:

[1380] The server converts the generated advice and emotion-sensitive messages into JSON format and sends them to the user's device.

[1381] Step 8:

[1382] The device parses the received advice and messages and displays them to the user within the application, and may also use speech synthesis to communicate them aloud if necessary.

[1383] Providing daily conversation partners and entertainment

[1384] Step 1:

[1385] The user selects either "talkie mode" or "game mode" in the application.

[1386] Step 2:

[1387] The device converts the selected mode information into JSON format and sends it to the cloud server via an HTTP POST request.

[1388] Step 3:

[1389] The server invokes a generative AI model and an emotion engine based on the received mode information.

[1390] Step 4:

[1391] The emotion engine recognizes emotions from the user's voice data and text and provides that information to the generative AI model.

[1392] Step 5:

[1393] The generative AI model generates appropriate conversation content in conversation partner mode and appropriate game content in game mode based on emotion information and mode information from the emotion engine.

[1394] Step 6:

[1395] The server converts the generated conversation content or game content into JSON format and sends it to the user's device.

[1396] Step 7:

[1397] In the conversation partner mode, the terminal displays the generated conversation content, and in the game mode, the terminal starts the generated game so that the user can play together with the user.

[1398] AI training function

[1399] Step 1:

[1400] Users set the AI's characteristics and direction through the application's development screen.

[1401] Step 2:

[1402] The device converts the development setting information into JSON format and sends it to the cloud server via an HTTP POST request.

[1403] Step 3:

[1404] The server automatically adjusts the responses and suggestions of the generative AI model based on the received training setting information, taking into account information from the emotion engine.

[1405] Revenue model and billing process

[1406] Step 1:

[1407] The user selects additional features from the application's billing page.

[1408] Step 2:

[1409] The device converts the purchase information into JSON format and sends it to the cloud server via an HTTP POST request.

[1410] Step 3:

[1411] The server calls a payment gateway based on the received purchase information and executes the payment process.

[1412] Step 4:

[1413] If the payment is successful, the server applies the additional features to the user's account and sends the information to the user's device via an HTTP POST request.

[1414] Step 5:

[1415] Based on the received additional function information, the terminal performs settings so that the user can use the function within the application.

[1416] Example 2

[1417] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1418] In modern society, users face various worries and problems in their daily lives, but it is difficult to quickly obtain appropriate advice. Furthermore, there is a lack of appropriate responses that take emotions into consideration, and people with whom people can easily talk, leading to feelings of loneliness. To address these issues, there is a need for a system that can understand users' emotions and provide appropriate advice and entertainment.

[1419] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1420] In this invention, the server includes means for inputting consultation content from a user, means for transmitting the input consultation content to a cloud server, means for analyzing the consultation content on the cloud server, converting it into an appropriate format, and passing it as input to the generative AI model and emotion identification device, means for the emotion identification device to recognize emotions from the consultation content and voice data, means for the generative AI model to generate advice and suggestions based on the recognized emotion information, and means for providing the generated advice to the user from the cloud server. This allows users to receive appropriate advice and entertainment according to their emotions at any given time, thereby improving the quality of their daily lives.

[1421] "User" refers to an individual who uses the system to input their inquiry and receive advice and suggestions.

[1422] "Consultation content" refers to the worries, questions, and other inquiries that the user inputs into the system by voice or text.

[1423] A "cloud server" is a server that can be accessed remotely via the Internet and is a central element of a system that analyzes and processes data.

[1424] A "generative AI model" is a generative model created by artificial intelligence, and refers to a program that creates advice and suggestions based on user input and emotional information.

[1425] An "emotion identification device" refers to a device or program that recognizes emotions from user input or voice data and reflects them in the analysis results.

[1426] "Advice and suggestions" refers to specific advice and recommendations generated by the generative AI model based on the user's consultation content and emotional information.

[1427] The "conversation partner mode" refers to a mode in which the user selects how the system will respond as a conversation partner, and conversation content based on emotions is provided.

[1428] "Play mode" refers to a mode in which the user selects a mode that provides games and entertainment content to the system, and the difficulty and content of the game are adjusted according to the user's emotional state.

[1429] "Setting the characteristics and direction of artificial intelligence" refers to the operation in which the user sets specific parameters through a settings screen within the system to adjust the response and proposal content of the generated AI model.

[1430] "Payment gateway" refers to an interface through which the cloud server processes the user's payment information and performs billing processing.

[1431] This invention is a system that allows users to input their consultation details using an application on a smartphone or tablet and receive appropriate advice and suggestions in response. The system operates on a cloud server using a generative AI model and an emotion recognition device. A specific embodiment of this system is described below.

[1432] The user launches an application installed on their smartphone or tablet and inputs the consultation details by voice or text. The device then sends the input consultation details to a cloud server. This transmission uses a communication protocol (e.g., HTTP, HTTPS) via an internet connection.

[1433] The cloud server parses the received data and converts it into an appropriate format (e.g., JSON format) using, for example, a natural language processing library or speech recognition technology (e.g., Google Speech-to-Text API).The converted data is then passed to a generative AI model and an emotion recognition device.

[1434] The emotion recognition device recognizes emotions from the consultation content and voice data entered by the user. The technology used can be, for example, IBM Watson's Natural Language Understanding. The emotion recognition device provides the recognized emotional information to the generative AI model.

[1435] The generative AI model generates advice and suggestions for the user based on this emotional information. For example, OpenAI's GPT model is used for generation. The generative AI model generates specific suggestions and response messages that reflect the user's emotional state.

[1436] The generated advice and suggestions are sent from the cloud server to the user's device. The device then provides the received advice to the user through an application. The advice is displayed using a visually easy-to-understand interface (e.g., text message, pop-up message).

[1437] For example, if a user complains of a "pain in the lower back," the cloud server passes the information to an emotion recognition device. The emotion recognition device recognizes "anxiety" from the voice data. This information is provided to a generative AI model, which then generates advice such as "Try some moderate stretching. Specifically, try the following movements," along with a message that takes the user's emotions into consideration, such as "Try doing it gradually, without overdoing it." The message is then sent to the user's device.

[1438] Furthermore, if the user selects "conversation partner mode," the emotion recognition device recognizes "loneliness." In this case, the generative AI model generates emotion-sensitive content, such as "What did you do today?" and "Shall we have a little fun conversation?", and provides these to the user.

[1439] An example of a specific prompt is as follows: "If a user complains of lower back pain, what advice would the emotion recognizer generate for the generative AI model by recognizing anxiety from the voice data?"

[1440] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1441] Step 1:

[1442] The user launches the application on their smartphone or tablet. Here, the user inputs the details of their consultation by voice or text. The input data is temporarily stored within the device. An example of input data is the voice input of "My lower back hurts."

[1443] Step 2:

[1444] The device converts the voice data entered by the user into text data. This conversion uses voice recognition technology (e.g., Google Speech-to-Text API). The converted text data becomes "My lower back hurts." This text data is then sent to a cloud server. HTTP or HTTPS is used as the transmission protocol.

[1445] Step 3:

[1446] The server parses the received text data using a natural language processing library (e.g., NLTK). The parsed data is then converted into an appropriate format (e.g., JSON). This formatted data is then passed as input to the generative AI model and emotion recognition device.

[1447] Step 4:

[1448] The emotion recognition device recognizes emotions from the user's text data. The technology used is, for example, IBM Watson's Natural Language Understanding. From the input data "My back hurts," the emotion "anxiety" is recognized. This recognized emotion data is provided to the generative AI model.

[1449] Step 5:

[1450] The generative AI model generates advice and suggestions for the user based on the emotional information received from the emotion recognition device. The generative AI model uses OpenAI's GPT model. Based on the input data "My lower back hurts" and the emotional information "anxiety," the advice message generated is "Try some moderate stretching. Specifically, do the following movements." This generated advice is sent back to the cloud server.

[1451] Step 6:

[1452] The server sends the advice received from the generative AI model to the user's device using HTTP or HTTPS as the transmission protocol. The input data includes the generated advice message.

[1453] Step 7:

[1454] The device provides the user with the advice received from the server. The advice is displayed as a text message on the application interface. Specifically, the message displayed is, "Try some moderate stretching. Don't push yourself too hard, and try it little by little."

[1455] Step 8 (optional):

[1456] To use additional features, the user selects them on the billing page within the application, and this selection information is sent from the terminal to the cloud server.

[1457] Step 9 (Optional):

[1458] The server processes the payment via a payment gateway (e.g., Stripe API). If the payment is successful, the add-on is applied to the user's account. Input data includes the user's payment information and the selected add-on.

[1459] (Application example 2)

[1460] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1461] In conventional store operations, in-store staff are responsible for all customer interactions and service provision, which results in a heavy workload. Furthermore, it is difficult to respond flexibly to customers' emotions and moods, creating a need for a system that can provide optimal service to individual customers. In particular, there is a growing need for a system that can recognize customer emotions and automatically generate appropriate advice and services based on them.

[1462] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1463] In this invention, the server includes means for inputting consultation content from a user, means for transmitting the input consultation content to a cloud server, means for analyzing the consultation content on the cloud server and passing it as input to a generative AI model, means for the generative AI model to generate advice or suggestions based on the consultation content, means for providing the generated advice to the user, means for recognizing the user's emotions using a camera, means for transmitting the emotion recognition result to the cloud server, means for the generative AI model to generate appropriate advice or suggestions for the user based on the emotion recognition result, and means for providing the advice or suggestions generated by an in-store robot. This makes it possible to efficiently handle customer service in the store and provide optimal services according to customer emotions.

[1464] A "user" is an individual or group that uses the system and is the entity that inputs the consultation content and voice data.

[1465] "Consultation content" refers to information such as questions, requests, and opinions that users provide to the system.

[1466] A "cloud server" is a computer system that provides computing resources that are remotely accessible via the Internet.

[1467] A "generative AI model" is an artificial intelligence algorithm that generates advice or suggestions based on specific input data (such as consultation content or emotion recognition results).

[1468] A "camera" is a device for capturing images and videos, and is used to recognize the user's emotions.

[1469] "Emotion recognition" is the process of analyzing information such as a user's facial expressions and voice to identify their emotional state (e.g., joy, sadness, anxiety, etc.).

[1470] A "robot" is a mechanical device that is programmed to work automatically or semi-automatically and perform specific tasks.

[1471] "Advice" is specific instructions or suggestions provided by the generative AI model based on the user's consultation content and emotions.

[1472] "Suggestions" are suggestions to the user for options or strategies to consider, generated based on the consultation and perceived emotions.

[1473] A "store" is a physical business location that offers goods and services to consumers.

[1474] "Staff" refers to the people who work in the store, dealing with customers and performing other tasks.

[1475] This invention is a system that combines generative AI models and emotion recognition technology to improve customer service and operational efficiency in brick-and-mortar stores. To realize this system, the following hardware and software are used to process and calculate data.

[1476] First, users (customers or staff) can input their concerns through a robot in a physical store. Input is done by voice or text using the robot's microphone or touch panel. The robot then sends the inputted concerns and voice data to a cloud server.

[1477] The cloud server analyzes the received data and passes it as input to the generative AI model. For emotion recognition, a camera mounted on the robot captures the user's facial expression and sends the image data to the cloud server. The cloud server then uses emotion recognition software (e.g., EmotionRecognition library) to recognize the user's emotion from the sent image data.

[1478] The recognized emotional information is fed into a generative AI model (e.g., GPT-2 model), which generates advice and suggestions based on the user's emotions and the content of the consultation. The generated advice and suggestions are then provided to the user from the cloud server via the robot's display and speakers.

[1479] For example, if a user says, "I'm not in the mood today," the robot will analyze their facial expressions and tone of voice and send them to the cloud server. If the cloud server recognizes the emotion and determines that the user is "tired," it will use the generative AI model to generate advice such as "I'll play some relaxing music," and provide it to the user via the robot.

[1480] Usage examples and prompt statements

[1481] Here is an example prompt:

[1482] The customer said: "I'm tired today." They seem to be feeling tired. What advice would be helpful for them?

[1483] In this way, the system of the present invention can improve the efficiency of customer service in stores and provide optimal services according to customer emotions.By using a generative AI model, it is possible to flexibly generate advice and suggestions customized for each individual user.

[1484] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1485] Step 1:

[1486] The user inputs the content of the consultation through the robot. The content of the consultation, input in voice or text format, is captured using the robot's microphone and touch panel.

[1487] Input: Consultation content (voice or text)

[1488] Output: The consultation content converted into data format by the robot

[1489] Step 2:

[1490] The robot sends the acquired consultation information to a cloud server using an internet connection.

[1491] Input: Consultation content converted into data format

[1492] Output: Data sent to the cloud server

[1493] Step 3:

[1494] The cloud server analyzes the received consultation content and starts emotion recognition.

[1495] Input: Consultation details sent

[1496] Output: Analysis results and data for emotion recognition

[1497] Step 4:

[1498] The cloud server uses the image data sent from the robot to recognize the user's emotions using emotion recognition software.

[1499] Input: Image data

[1500] Output: Recognized emotion information (e.g., joy, sadness, anxiety)

[1501] Step 5:

[1502] The cloud server inputs the consultation content and emotion recognition results into a generative AI model to generate appropriate advice and suggestions for the user.

[1503] Input: Consultation content, recognized emotion information

[1504] Output: Advice or suggestions generated by the generative AI model

[1505] Step 6:

[1506] The cloud server then sends the generated advice and suggestions back to the robot.

[1507] Input: Generated advice and suggestions

[1508] Output: Data sent to the robot

[1509] Step 7:

[1510] The robot provides the user with advice and suggestions received from the cloud server, which are communicated through the robot's display and speakers.

[1511] Input: Advice and suggestions received from the cloud server

[1512] Output: Advice and suggestions presented to the user

[1513] Step 8:

[1514] If desired, the user can interact with the robot again to provide additional consultation or feedback, and the process repeats.

[1515] Input: Additional consultation details, feedback

[1516] Output: Updated consultation details and feedback information

[1517] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1518] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1519] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1520] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1521] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1522] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1523] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1524] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1525] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1526] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1527] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1528] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1529] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1530] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1531] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1532] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1533] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1534] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1535] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1536] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1537] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1538] The following is further disclosed regarding the above embodiment.

[1539] (Claim 1)

[1540] A means for a user to input the content of the consultation;

[1541] means for transmitting the input consultation details to a cloud server;

[1542] A means to analyze the consultation content on a cloud server and pass it as input to the generative AI model,

[1543] A means for the generative AI model to generate advice and suggestions based on the consultation content;

[1544] means for providing the generated advice to a user;

[1545] A system including:

[1546] (Claim 2)

[1547] 10. The system according to claim 1, wherein the cloud server further comprises means for generating content for the conversation partner mode or the game mode based on a user selection and providing the content to the user.

[1548] (Claim 3)

[1549] The system of claim 1 further includes a means for a user to set the characteristics and direction of the AI, and for the cloud server to automatically adjust the response content and proposal content of the generated AI model based on the settings.

[1550] (Claim 4)

[1551] The system according to claim 1, further comprising: means for a user to select an additional function, a cloud server to process payment based on that information, and means for applying the additional function to the user account when payment is successful.

[1552] "Example 1"

[1553] (Claim 1)

[1554] A means for a user to input the content of the consultation;

[1555] means for transmitting the input consultation details to a cloud server;

[1556] A means for the cloud server to analyze the received data, convert it into an appropriate format, and pass it to the generative AI model;

[1557] A means for the generative AI model to generate advice and suggestions based on the consultation content;

[1558] a means for transmitting the generated advice from the cloud server to the user's device again and providing the advice to the user through an application;

[1559] A system including:

[1560] (Claim 2)

[1561] The system of claim 1, further comprising means for receiving selection information of a "conversation mode" or a "game mode" in an application from a cloud server, and generating and providing appropriate content to the user.

[1562] (Claim 3)

[1563] The system of claim 1 further includes a means for a user to set the characteristics and direction of the AI ​​within the application, and for the cloud server to automatically adjust the response content and proposal content of the generated AI model based on the training setting information.

[1564] "Application Example 1"

[1565] (Claim 1)

[1566] A means for a user to input the content of the consultation;

[1567] means for transmitting the input consultation details to a cloud server;

[1568] A means to analyze the consultation content on a cloud server and pass it as input to the generative AI model,

[1569] A means for the generative AI model to generate advice and suggestions based on the consultation content;

[1570] means for providing the generated advice to a user;

[1571] means for generating dietary suggestions based on health status;

[1572] a means for ordering the suggested meals from a food delivery system; and

[1573] A system including:

[1574] (Claim 2)

[1575] 10. The system according to claim 1, wherein the cloud server further comprises means for generating content for the conversation partner mode or the game mode based on a user selection and providing the content to the user.

[1576] (Claim 3)

[1577] The system of claim 1 further includes a means for a user to set the characteristics and direction of the AI, and for the cloud server to automatically adjust the response content and proposal content of the generated AI model based on the settings.

[1578] "Example 2: Combining Emotion Engines"

[1579] (Claim 1)

[1580] A means for a user to input the content of the consultation;

[1581] means for transmitting the input consultation details to a cloud server;

[1582] A means for analyzing the consultation content on a cloud server, converting it into an appropriate format, and passing it as input to the generation AI model and emotion recognition device;

[1583] means for an emotion identification device to recognize emotions from consultation content and voice data;

[1584] a means for the generative AI model to generate advice or suggestions based on the recognized emotion information; and

[1585] a means for providing the generated advice to a user from a cloud server;

[1586] A system including:

[1587] (Claim 2)

[1588] 2. The system according to claim 1, wherein the cloud server further comprises means for generating content for the conversation partner mode or the play mode based on a user's selection and providing the content to the user.

[1589] (Claim 3)

[1590] The system of claim 1 further includes a means for a user to set the characteristics and direction of the artificial intelligence, and for the cloud server to automatically adjust the response content and suggestions of the generated AI model based on the settings.

[1591] "Application example 2 when combining emotion engines"

[1592] (Claim 1)

[1593] A means for a user to input the content of the consultation;

[1594] means for transmitting the input consultation details to a cloud server;

[1595] A means to analyze the consultation content on a cloud server and pass it as input to the generative AI model,

[1596] A means for the generative AI model to generate advice and suggestions based on the consultation content;

[1597] means for providing the generated advice to a user;

[1598] means for recognizing a user's emotion using a camera;

[1599] means for transmitting emotion recognition results to a cloud server;

[1600] A means for the generative AI model to generate appropriate advice and suggestions for the user based on the emotion recognition results;

[1601] a means for providing in-store robot-generated advice and suggestions;

[1602] A system including:

[1603] (Claim 2)

[1604] 10. The system of claim 1, further comprising means for the generative AI model to generate and provide to the user daily store operation support or customer service content based on user selections.

[1605] (Claim 3)

[1606] The system of claim 1 further includes a means for a user to set the characteristics and direction of the AI, and for the cloud server to automatically adjust the response content and proposal content of the generated AI model based on the settings. [Explanation of symbols]

[1607] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for a user to input the content of the consultation; means for transmitting the input consultation details to a cloud server; A means to analyze the consultation content on a cloud server and pass it as input to the generative AI model, A means for the generative AI model to generate advice and suggestions based on the consultation content; means for providing the generated advice to a user; A system including:

2. The system according to claim 1 , further comprising means for the cloud server to generate content for the conversation partner mode or the game mode based on a user selection and provide the content to the user.

3. The system of claim 1 further includes a means for a user to set the characteristics and direction of the AI, and for the cloud server to automatically adjust the response content and proposal content of the generated AI model based on the settings.

4. The system according to claim 1, further comprising: means for a user to select an additional function, a cloud server to perform a payment process based on the information, and means for applying the additional function to the user account when the payment is successful.

Images