system

Abstract

We provide the system. [Solution] A screen display means for inputting user information, Means for sending the entered user information and requests to the server, A means for sending a request to a generative artificial intelligence based on user information and requests, A means of receiving a response from a generative artificial intelligence, A means for sending and displaying the received response on the user terminal, A means of receiving user feedback, A means of sending a new request to the generative artificial intelligence based on the feedback, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including: receiving a user utterance; adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot; encoding the prompt; and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Modern consumers use a large number of online services and applications, and the problem is how to incorporate these services into their lives and make the most of them. However, simply explaining the service content does not enable consumers to clearly understand how they can specifically apply it to their own life scenes. Therefore, it is difficult for service providers to effectively convey the convenience of the service, resulting in the problem that consumers cannot fully utilize the service.

Means for Solving the Problems

[0005] To solve the above problems, the present invention provides the following means: a screen display means for inputting user information, allowing the user to input basic information such as their age, gender, and hobbies. Next, a means for sending this user information and a request regarding the usage scenario of a specific service (e.g., an electronic payment service) to a server. Furthermore, the server sends a request to a generative artificial intelligence based on the received information, and this AI generates specific suggestions corresponding to the user's life scenario. The generated suggestions are formatted into an appropriate data format by the server and sent to the user terminal. The user terminal also has a function to display the suggestions to the user and receive feedback from the user. Finally, based on this feedback, a request is sent again to the generative artificial intelligence to generate more specific and customized suggestions. Through this series of means, the user can concretely understand how the provided service can be applied to their life scenario, and the effective use of the service is promoted.

[0006] "User information" refers to personally identifiable attribute information of the user, such as age, gender, and areas of interest.

[0007] "Screen display means" refers to the functions of a display device or software that provides an interface for the user to input, confirm, and select information.

[0008] A "request" refers to a specific request or question that a user asks about particular information or services.

[0009] A "server" refers to a central computing system that receives and processes requests from user terminals via a network.

[0010] "Generative artificial intelligence" refers to AI technology that uses natural language processing techniques to generate responses and suggestions based on user requests.

[0011] A "request" refers to the act of sending a request to a generative artificial intelligence system that seeks specific information processing, including user information and specific requests.

[0012] "Response" refers to information or suggestions generated by a generative artificial intelligence system based on a request.

[0013] "Feedback" refers to the act of a user responding to a suggestion with additional questions, requests, or reactions.

[0014] "User terminal" refers to electronic devices such as computers, smartphones, and tablets used by users.

[0015] "Data format" refers to the representation of data that has been appropriately organized and prepared in a form that can be communicated and processed. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] One embodiment of the present invention describes below a specific system for a user to incorporate a particular service into their life and make the most of it. This system provides the user with useful information through the input of user information, generation of suggestions by generative artificial intelligence, display of the suggested content, and processing of user feedback.

[0038] System-wide configuration

[0039] The system primarily consists of three parts: user terminals, servers, and generative artificial intelligence. These elements communicate and cooperate with each other via the internet to enable the overall system to function.

[0040] User terminal

[0041] 1. Enter basic information

[0042] The terminal first displays a screen for user information input. The user enters basic information such as their age, gender, and areas of interest. This information is later processed by generative artificial intelligence.

[0043] 2. Input of request

[0044] The terminal also provides an interface for receiving specific requests from the user. For example, the user can input a specific request such as, "Please tell me about situations in which PayPay can be used in daily life."

[0045] 3. Sending data to the server

[0046] The terminal sends user information and requests to the server. The transmitted data is used in the next processing step.

[0047] server

[0048] 4. Data processing and request generation

[0049] The server processes the received data, including user information and request details, and uses this information to generate appropriate requests for the generative artificial intelligence. For example, it might send a request to the generative AI such as, "Generate scenarios showing how a man in his 30s who enjoys outdoor activities can use PayPay in his daily life."

[0050] 5. Receiving a response from a generative artificial intelligence.

[0051] The server receives a response from the generative artificial intelligence. This response includes specific suggestions tailored to the user's life scenarios.

[0052] 6. Formatting and sending the results

[0053] The server formats the response from the generative artificial intelligence into an appropriate format and sends it to the user terminal.

[0054] Generative artificial intelligence

[0055] 7. Proposal generation

[0056] Generative artificial intelligence receives requests from a server and generates suggestions for specific life scenarios based on user information and requests. These suggestions show how the user can specifically utilize the service.

[0057] Specific example

[0058] 1. Enter user information

[0059] Example: The user enters information on their device such as "30s, male, enjoys outdoor activities."

[0060] 2. Input of request

[0061] Example: The user enters a request such as, "Tell me 10 situations in everyday life where PayPay can be used."

[0062] 3. Sending data to the server

[0063] The terminal sends user information and requests to the server.

[0064] 4. Data processing and request generation

[0065] Based on the data the server receives, it sends a request to the generative artificial intelligence saying, "Please generate scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[0066] 5. Receiving a response from a generative artificial intelligence.

[0067] The generative artificial intelligence generates the following suggestions and returns them to the server:

[0068] 1. Purchase camping equipment

[0069] 2. Payment of admission fees for outdoor events

[0070] 3. Paying for meals at restaurants

[0071] 4. Paying for fuel at a gas station

[0072] 5. Payment of public transport fares

[0073] 6. Payment of sports facility usage fees

[0074] 7. Outdoor-related subscription services

[0075] 8. Payment of travel expenses at the travel agency

[0076] 9. Booking and payment for accommodation

[0077] 10. Booking and Payment for Activities

[0078] 6. Formatting and sending the results

[0079] The server formats the generated proposal into the appropriate data format and sends it to the user's terminal.

[0080] 7. Displaying results and receiving feedback

[0081] The device displays the results to the user, who then provides feedback on the suggestions. For example, the user might input, "Which store is best for buying camping equipment?"

[0082] 8. Processing and resubmitting feedback

[0083] The server receives user feedback and sends a request to the generative artificial intelligence. The generative AI generates specific suggestions again, which the server receives, formats appropriately, and sends back to the user's terminal.

[0084] In this way, the system incorporates user feedback and has the function of continuously and concretely suggesting how users can utilize the services provided in their actual daily lives.

[0085] The following describes the processing flow.

[0086] Step 1:

[0087] The device displays a screen that prompts the user to enter basic information (age, gender, areas of interest, etc.).

[0088] Step 2:

[0089] The user enters information such as their age, gender, and areas of interest into the displayed input form.

[0090] Step 3:

[0091] The device sends basic information entered by the user to the server. This data includes the user's age, gender, and areas of interest.

[0092] Step 4:

[0093] The terminal displays an interface that prompts the user to input requests regarding specific service usage scenarios (e.g., "Please tell me about scenarios in which PayPay can be used in daily life").

[0094] Step 5:

[0095] The user enters their request into the terminal.

[0096] Step 6:

[0097] The terminal sends the user's request to the server. The transmitted data includes the details of the user's request.

[0098] Step 7:

[0099] The server converts the received user information and request into a request format appropriate for the generative artificial intelligence.

[0100] Step 8:

[0101] The server sends a request to the generative artificial intelligence. The request includes the user's age, gender, areas of interest, and the content of the request.

[0102] Step 9:

[0103] Generative artificial intelligence processes received requests and generates specific suggestions that correspond to the user's life scenarios.

[0104] Step 10:

[0105] The generative artificial intelligence sends the generated suggestions back to the server as a response.

[0106] Step 11:

[0107] The server receives the response from the generative artificial intelligence and formats it into an appropriate data format (e.g., JSON).

[0108] Step 12:

[0109] The server sends the formatted proposal to the terminal.

[0110] Step 13:

[0111] The device displays received suggestions to the user. These suggestions include specific examples of service usage tailored to the user's lifestyle.

[0112] Step 14:

[0113] Users provide feedback on the displayed suggestions (e.g., "Which store is best for purchasing camping equipment?").

[0114] Step 15:

[0115] The device sends user feedback to the server.

[0116] Step 16:

[0117] The server sends a new request to the generative artificial intelligence based on the user's feedback. This request includes additional information about the user.

[0118] Step 17:

[0119] The generative artificial intelligence receives a follow-up request and generates additional specific suggestions.

[0120] Step 18:

[0121] The generative artificial intelligence sends the generated additional suggestions back to the server as a response.

[0122] Step 19:

[0123] The server receives additional responses from the generative artificial intelligence, formats them into the appropriate data format, and sends them to the terminal.

[0124] Step 20:

[0125] The device displays the received additional suggestions to the user.

[0126] This series of steps allows the system to continuously provide specific suggestions that align with the user's requirements.

[0127] (Example 1)

[0128] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0129] Conventional systems struggled to generate personalized suggestions based on individual user requests, failing to improve user satisfaction. Furthermore, they lacked sufficient mechanisms for efficiently incorporating user feedback and updating suggestions.

[0130] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0131] In this invention, the server includes means for processing user information and requests, generating and sending prompt messages to a generative artificial intelligence; means for receiving responses from the generative artificial intelligence; and means for formatting the received responses into an appropriate data format and sending them to the user terminal. This makes it possible to generate personalized suggestions based on the user's individual requests and to efficiently incorporate feedback to update the suggestions.

[0132] "Means for inputting basic user information" refers to providing an interface for inputting basic user information such as age, gender, and areas of interest.

[0133] "Means for inputting user requests" refers to means of providing an interface for users to input specific requests (for example, questions about how to use a particular service).

[0134] "Means for sending entered user information and requests to the server" refers to means for sending the basic information and requests entered by the user to the server via the network.

[0135] "Means for processing user information and requests, and generating and sending prompt messages to a generative artificial intelligence" refers to means for creating and sending appropriate prompt messages to a generative artificial intelligence based on the received user information and requests.

[0136] "Means for receiving responses from generative artificial intelligence" refers to means for the server to receive proposals and responses generated by generative artificial intelligence.

[0137] "Means for formatting the received response into an appropriate data format and sending it to the user terminal" refers to means of converting the response received from the generative artificial intelligence into a format that is easy for the user to understand and sending it again to the user terminal via the network.

[0138] "Means for displaying responses to the user and receiving feedback from the user" refers to means that the user terminal provides an interface for displaying responses from generative artificial intelligence to the user and for receiving feedback from the user.

[0139] "Method for sending a new request to a generative artificial intelligence based on feedback" refers to a method for generating a new prompt message based on feedback received from the user and sending it to the generative artificial intelligence.

[0140] Overall system configuration

[0141] This invention is a system that enables users to incorporate specific services into their lives and make the most of them. This system provides users with useful information through the input of user information, generation of suggestions by generative artificial intelligence, display of the suggested content, and processing of user feedback.

[0142] Hardware and software to be used

[0143] The system primarily consists of three parts: user terminals, servers, and generative artificial intelligence. These elements communicate with each other via the internet and work together as a whole system. User terminals include smartphones, tablets, and personal computers. Servers include web servers and cloud servers. Generative artificial intelligence utilizes computing resources (e.g., cloud-based AI services) capable of executing appropriate AI models.

[0144] Program processing

[0145] The specific processing details of the system are as follows:

[0146] Entering user information

[0147] The user terminal displays a screen prompting the user to enter basic information such as age, gender, and areas of interest. The user enters information such as "30s, male, enjoys outdoor activities." This information is encoded in JSON format or similar and sent to the server.

[0148] Request Input

[0149] The user terminal provides an interface for the user to input specific requests. The user might input a specific request such as, "Please tell me about situations in daily life where PayPay can be used."

[0150] Sending data to the server

[0151] The user terminal sends the entered user information and requests to the server. The server receives this information and parses it.

[0152] Data processing and request generation

[0153] The server generates and sends a prompt message to the generative artificial intelligence based on user information and the request. For example, a request such as "Generate scenarios showing how a man in his 30s who enjoys outdoor activities can use PayPay in his daily life" might be generated.

[0154] Received a response from a generative artificial intelligence.

[0155] Generative artificial intelligence receives requests and generates suggestions for specific life scenarios based on user information. For example, it might generate suggestions such as "purchase camping equipment" or "pay admission fees for outdoor events." These responses are then returned to the server.

[0156] Formatting and sending the results

[0157] The server formats the response received from the generative artificial intelligence into an appropriate data format and sends it back to the user terminal. In this process, the data is formatted into a list and then encoded again in JSON format.

[0158] Displaying results and accepting feedback

[0159] The user terminal displays the received suggestions to the user. The user then reviews these suggestions and provides more specific feedback. For example, they might input feedback such as, "Which store is best for purchasing camping equipment?"

[0160] Processing and resubmitting feedback

[0161] Based on the feedback received from the user, the server generates a new prompt and sends it to the generative artificial intelligence. The generative AI then generates specific suggestions, which the server receives, formats appropriately, and sends back to the user's terminal.

[0162] Specific example

[0163] 1. Enter user information

[0164] The user enters information on their device such as "30s, male, enjoys outdoor activities."

[0165] 2. Input of request

[0166] The user enters a request asking for "10 ways to use PayPay in everyday life."

[0167] 3. Sending data to the server

[0168] The user terminal sends the entered data to the server.

[0169] 4. Data processing and request generation

[0170] The server parses the data and generates and sends a prompt message: "Generate scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[0171] 5. Receiving a response from a generative artificial intelligence.

[0172] Generative artificial intelligence generates suggestions and returns them to the server. Examples: "Purchase camping equipment," "Pay admission fee for outdoor event," etc.

[0173] 6. Formatting and sending the results

[0174] The server formats the proposal and sends it to the user's terminal.

[0175] 7. Displaying results and receiving feedback

[0176] The user's device displays suggestions to the user, who then provides more specific feedback.

[0177] 8. Processing and resubmitting feedback

[0178] The server processes the feedback and sends a new prompt to the generative artificial intelligence again, which generates a new suggestion, formats it again, and sends it to the user terminal.

[0179] The above describes a specific embodiment of the present invention, a system that efficiently provides personalized suggestions based on the individual needs of users and continuously improves while incorporating feedback.

[0180] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0181] Step 1: Enter user information

[0182] The user terminal displays a screen prompting the user to enter basic information such as age, gender, and areas of interest. The user enters information such as "30s, male, enjoys outdoor activities" into the input screen. The entered information is temporarily stored on the terminal. Input data formats such as JSON and XML are used.

[0183] Input: User's basic information (age, gender, areas of interest, etc.)

[0184] Output: Encoded user basic information data

[0185] Step 2: Enter the request

[0186] The user terminal provides an interface for entering specific requests. Users enter specific requests such as, "Please tell me about situations where PayPay can be used in daily life." The entered requests are then encoded into a data format.

[0187] Input: User request (e.g., "Please tell me about situations in daily life where PayPay can be used.")

[0188] Output: Encoded user request data

[0189] Step 3: Sending data to the server

[0190] The user terminal sends the entered user information and requests to the server. The HTTPS protocol is used for transmission, and the data is encoded in JSON format. Upon successful communication, the data is stored on the server.

[0191] Input: Encoded user basic information data and user request data

[0192] Output: User information and request data stored on the server

[0193] Step 4: Data processing and request generation

[0194] The server parses and analyzes the received user information and requests. For example, it extracts information such as age, gender, and areas of interest. Next, it generates a prompt message to send to the generative artificial intelligence based on that information. The prompt message is in a format such as, "Generate 10 scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[0195] Input: User information and request data stored on the server

[0196] Output: Generated prompt message

[0197] Step 5: Receiving a response from the generative artificial intelligence.

[0198] The server sends the generated prompt to the generative artificial intelligence (AI). The AI ​​receives the request, analyzes it based on the prompt, and generates specific suggestions to address the user's request. These suggestions are then returned to the server.

[0199] Input: Generated prompt message

[0200] Output: Response data from generative artificial intelligence (e.g., suggestions such as "purchase camping equipment" or "pay admission fee for an outdoor event")

[0201] Step 6: Format and submit the results

[0202] The server formats the response received from the generative artificial intelligence into an appropriate data format. The formatted data is then encoded again in JSON format and sent to the user's terminal.

[0203] Input: Response data from generative artificial intelligence

[0204] Output: Formatted proposal data

[0205] Step 7: Displaying results and receiving feedback

[0206] The user terminal displays a list of received suggestions to the user. The user reviews the suggestions and enters more specific feedback. For example, they might enter feedback such as, "Which store is best for purchasing camping equipment?" and click the submit button.

[0207] Input: Formatted proposal data

[0208] Output: User feedback (e.g., specific questions or requests)

[0209] Step 8: Processing feedback and resubmitting requests

[0210] The server parses and analyzes the feedback received from the user. Based on the feedback, it generates a new prompt and sends it to the generative artificial intelligence. The generative AI generates another suggestion, which the server receives, formats appropriately, and sends back to the user's terminal.

[0211] Input: User feedback data

[0212] Output: Newly generated prompt sentence and new response data from the generative AI.

[0213] Through the above processing steps, the system can efficiently provide personalized suggestions based on the user's individual requests and continuously improve by incorporating feedback.

[0214] (Application Example 1)

[0215] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0216] This invention relates to a system that generates suggestions tailored to the user's lifestyle and maximizes user convenience. In particular, to improve the shopping experience in physical stores, there is a need to provide optimized suggestions in real time using the user's current location information. However, conventional systems have the problem of not being able to improve user satisfaction because they do not provide insufficient suggestions based on user information and do not perform real-time optimization using the current location information.

[0217] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0218] In this invention, the server includes means for having a display device for inputting user information, means for transmitting the input user information and requests to a communication device, means for receiving responses from generative artificial intelligence, means for acquiring the user's current location information and optimizing response suggestions based on the location information, and means for displaying the generated suggestions in real time on a smartphone or smart glasses. This enables the user to fully enjoy the shopping experience in a physical store.

[0219] "User information" refers to basic data such as the user's age, gender, areas of interest, and current location.

[0220] A "display device" is an interface for users to input and confirm information, and includes devices such as smartphones and smart glasses.

[0221] A "communication device" is a device used to send user information and requests to a server.

[0222] "Generative artificial intelligence" refers to an artificial intelligence system that generates optimal suggestions based on user information and requests.

[0223] "Current location information" refers to the user's real-time geographical location data.

[0224] "Real-time" refers to a timeframe that responds immediately to the user's current actions and circumstances.

[0225] A "smartphone" is a type of mobile phone that is a device with multi-functional computer capabilities.

[0226] "Smart glasses" are wearable devices designed to enhance visual information.

[0227] The following describes specific embodiments for carrying out the present invention. The invention relates to a system for users to incorporate specific services into their lives and make the most of them. The specific configuration and operation will be described below.

[0228] System-wide configuration

[0229] The system primarily consists of the following elements: user terminals, servers, and generative artificial intelligence. These elements communicate with each other via the internet and cooperate to function as a whole system.

[0230] User terminal

[0231] A user terminal is equipped with a display device and provides an interface that allows the user to input information. Examples include smartphones and smart glasses.

[0232] 1. Enter basic information

[0233] The user enters basic information such as age, gender, and areas of interest on the display device of their user terminal.

[0234] 2. Input of request

[0235] Users utilize an interface to enter specific requests. For example, they might enter a request such as, "Please recommend some outdoor products."

[0236] 3. Sending data to the server

[0237] The user terminal transmits the entered user information and requests to the server via a communication device.

[0238] server

[0239] The server processes the data sent from the user terminal and generates appropriate requests for the generative artificial intelligence.

[0240] 4. Data processing and request generation

[0241] The server organizes the received user information and request details, and then sends the request to the generative artificial intelligence based on that information.

[0242] For example, generate a prompt such as, "Generate suggestions on what products a man in his 30s who enjoys outdoor activities should purchase at a physical store."

[0243] 5. Receiving a response from a generative artificial intelligence.

[0244] The server receives the response from the generative artificial intelligence and formats it into an appropriate format.

[0245] 6. Optimization and Display

[0246] The server obtains the user's current location information and optimizes response suggestions based on that location. The generated suggestions are then displayed in real time on the user's smartphone or smart glasses.

[0247] Generative artificial intelligence

[0248] Generative artificial intelligence receives requests from a server and generates specific lifestyle scenarios and product suggestions based on user information and requests.

[0249] Specific example

[0250] 1. Enter user information

[0251] Example: A user enters information such as "30s, male, enjoys outdoor activities" on their smartphone.

[0252] 2. Input of request

[0253] Example: A user voice-inputs, "Please recommend some outdoor products."

[0254] 3. Sending data to the server

[0255] The user terminal sends user information and requests to the server.

[0256] 4. Data processing and request generation

[0257] The server sends the prompt message, "A man in his 30s who enjoys the outdoors is seeking advice on recommended outdoor products available at physical stores," to the generative AI.

[0258] 5. Receiving a response from a generative artificial intelligence.

[0259] The generative artificial intelligence generates a response and returns it to the server. This response may include suggestions such as, "A new camping tent has recently arrived at a nearby sporting goods store."

[0260] 6. Optimization and Display

[0261] The server optimizes the suggested content based on the user's current location, sends it to the user's terminal, and displays it in real time.

[0262] In this way, the system incorporates user feedback and has the function of continuously and concretely suggesting how users can utilize the services provided in their actual daily lives.

[0263] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0264] Step 1:

[0265] The user enters basic information such as age, gender, and areas of interest using the display device on their terminal. For example, they might enter information such as "30s, male, enjoys outdoor activities." This input data is collected on the terminal.

[0266] Step 2:

[0267] The user terminal provides an interface for inputting specific requests from the user. Users input requests such as "Please recommend some outdoor products" via voice or text. This request data is also collected by the terminal.

[0268] Step 3:

[0269] The user terminal transmits the entered user information and requests to the server via a communication device. At this time, the input data and request data are combined into a single request packet and sent to the server.

[0270] Step 4:

[0271] The server processes the received user information and request details to generate an appropriate request for the generative artificial intelligence. For example, it might generate a prompt such as, "Generate suggestions on what products a man in his 30s who enjoys outdoor activities should purchase at a physical store." This prompt becomes the input data for the generative artificial intelligence.

[0272] Step 5:

[0273] Generative artificial intelligence receives prompt messages from a server and generates specific lifestyle scenarios and product suggestions based on user information and requests. For example, it might generate a suggestion such as, "A new camping tent has recently arrived at a nearby sports goods store." This suggestion is then sent back to the server.

[0274] Step 6:

[0275] The server receives a response from the generative artificial intelligence and obtains the user's current location information. Based on the obtained location information, it further optimizes the response. For example, it processes the data to provide information about the nearest store based on the user's location. This optimized suggestion is then sent to the user's terminal.

[0276] Step 7:

[0277] The user's device displays received suggestions in real time on their smartphone or smart glasses. This allows users to instantly see the best suggestions to enhance their in-store shopping experience.

[0278] Step 8:

[0279] Users provide feedback on the displayed suggestions. For example, they might enter feedback such as, "This information was helpful. I'd like to know more details." This user feedback is collected on the device.

[0280] Step 9:

[0281] The user terminal sends feedback from the user to the server. The feedback data arrives at the server.

[0282] Step 10:

[0283] The server sends a request again to the generative artificial intelligence based on the received feedback. For example, it generates a prompt sentence such as "A 30-year-old man who likes the outdoors is asking for more detailed information about outdoor products" and sends it to the generative artificial intelligence.

[0284] Step 11:

[0285] The generative artificial intelligence generates a response again and sends it to the server. This process is repeated according to the user's feedback, and the system can continue to provide the most optimal information for the user.

[0286] Furthermore, an emotion engine for estimating the user's emotions may be combined. That is, the specific processing unit 290 may estimate the user's emotions using the emotion recognition model 59 and perform specific processing using the user's emotions.

[0287] As one embodiment of the present invention, a specific embodiment of a system that recognizes the user's emotions and makes more appropriate service proposals based on them will be described below. This system provides the most optimal information for the user through the input of user information, the generation of proposals by the generative artificial intelligence, the recognition of emotions by the emotion engine, the display of proposal contents, and the processing of user feedback.

[0288] Overall configuration of the system

[0289] The system is mainly composed of the following four parts: user terminal, server, generative artificial intelligence, and emotion engine. These elements communicate with each other and cooperate via the Internet to realize the operation of the entire system.

[0290] User terminal

[0291] 1. Input of basic information

[0292] The device first displays a screen for user information input. The user enters basic information such as their age, gender, and areas of interest. This information is later processed by generative artificial intelligence and an emotion engine.

[0293] 2. Input of request

[0294] The terminal provides an interface for users to input specific requests (e.g., "Please tell me about situations in which PayPay can be used in daily life").

[0295] 3. Recognition of emotions

[0296] The device captures the user's facial expressions and voice as they enter their requests and sends them to the emotion engine.

[0297] 4. Sending data to the server

[0298] The device sends user information, requests, and sentiment analysis results from the sentiment engine to the server. The transmitted data includes the user's age, gender, areas of interest, request content, and sentiment analysis results.

[0299] server

[0300] 5. Data processing and request generation

[0301] The server receives data, including user information, request details, and sentiment analysis results, and then converts it into an appropriate request format for the generative artificial intelligence. For example, it might send a request to the generative AI such as, "Generate scenarios showing how a man in his 30s who enjoys the outdoors and has positive emotions can use PayPay in his daily life."

[0302] 6. Receiving a response from a generative artificial intelligence.

[0303] The server receives a response from the generative artificial intelligence. This response includes specific suggestions tailored to the user's life scenarios and is customized based on the sentiment analysis results.

[0304] 7. Formatting and Sending of Results

[0305] The server formats the response from the generative artificial intelligence into an appropriate format and sends it to the user terminal.

[0306] Generative Artificial Intelligence

[0307] 8. Generation of Proposals

[0308] The generative artificial intelligence receives a request from the server and generates proposals for specific life scenarios based on user information, requests, and sentiment analysis results. This proposal indicates how the user can specifically use the service and also corresponds to the user's emotional state.

[0309] Specific Example

[0310] 1. Input of User Information

[0311] Example: The user inputs information such as "in his 30s, male, with outdoor hobbies" on the terminal.

[0312] 2. Input of Requests

[0313] Example: The user inputs a request such as "Tell me 10 scenes where Paypay can be used in life."

[0314] 3. Recognition of Emotions

[0315] The terminal captures the user's expression with the camera, and the emotion engine analyzes it and determines it as "positive."

[0316] 4. Sending Data to the Server

[0317] The terminal sends the user information, request, and sentiment analysis results to the server.

[0318] 5. Data Processing and Request Generation

[0319] Based on the data the server receives, it sends a request to the generative artificial intelligence saying, "Please generate scenarios showing how a man in his 30s, who enjoys the outdoors and has positive emotions, can use PayPay in his daily life."

[0320] 6. Receiving a response from a generative artificial intelligence.

[0321] The generative artificial intelligence generates the following suggestions and returns them to the server:

[0322] 1. Purchase camping equipment

[0323] 2. Payment of admission fees for outdoor events

[0324] 3. Paying for meals at restaurants

[0325] 4. Paying for fuel at a gas station

[0326] 5. Payment of public transport fares

[0327] 6. Payment of sports facility usage fees

[0328] 7. Outdoor-related subscription services

[0329] 8. Payment of travel expenses at the travel agency

[0330] 9. Booking and payment for accommodation

[0331] 10. Booking and Payment for Activities

[0332] 7. Formatting and sending the results

[0333] The server formats the generated proposal into the appropriate data format and sends it to the user's terminal.

[0334] 8. Displaying results and receiving feedback

[0335] The device displays the results to the user, who then provides feedback on the suggestions. For example, the user might input, "Which store is best for purchasing camping equipment?"

[0336] 9. Processing and resubmitting feedback

[0337] The server receives user feedback and sends a request to the generative artificial intelligence. The generative AI generates specific suggestions again, which the server receives, formats appropriately, and sends back to the user's terminal.

[0338] Through these steps, the system takes user emotions into consideration and proposes services in a more appropriate way. This allows users to more concretely integrate the services offered into their own lives.

[0339] The following describes the processing flow.

[0340] Step 1:

[0341] The device displays a screen that prompts the user to enter basic information (age, gender, areas of interest, etc.).

[0342] Step 2:

[0343] The user enters information such as their age, gender, and areas of interest into the displayed input form.

[0344] Step 3:

[0345] The device sends basic information entered by the user to the server. This data includes the user's age, gender, and areas of interest.

[0346] Step 4:

[0347] The terminal displays an interface that prompts the user to input requests regarding specific service usage scenarios (e.g., "Please tell me about scenarios in which PayPay can be used in daily life").

[0348] Step 5:

[0349] The user enters their request into the terminal.

[0350] Step 6:

[0351] The terminal sends the user's request to the server. The transmitted data includes the details of the user's request.

[0352] Step 7:

[0353] The device captures the user's facial expressions and voice as they enter their requests and sends this information to the emotion engine.

[0354] Step 8:

[0355] The emotion engine analyzes captured facial and audio data to recognize the user's emotional state (e.g., positive, negative, neutral).

[0356] Step 9:

[0357] The emotion engine sends the analysis results to the server. The transmitted data includes the user's emotional state.

[0358] Step 10:

[0359] The server converts the received user information, request details, and sentiment analysis results into a request format appropriate for the generative artificial intelligence.

[0360] Step 11:

[0361] The server sends a request to the generative artificial intelligence. The request includes the user's age, gender, areas of interest, request content, and sentiment analysis results.

[0362] Step 12:

[0363] Generative artificial intelligence processes received requests and generates specific suggestions tailored to the user's life scenarios. These suggestions also reflect the user's emotional state.

[0364] Step 13:

[0365] The generative artificial intelligence sends the generated suggestions back to the server as a response.

[0366] Step 14:

[0367] The server receives the response from the generative artificial intelligence and formats it into an appropriate data format (e.g., JSON).

[0368] Step 15:

[0369] The server sends the formatted proposal to the terminal.

[0370] Step 16:

[0371] The device displays received suggestions to the user. These suggestions include specific examples of service usage tailored to the user's lifestyle.

[0372] Step 17:

[0373] Users provide feedback on the displayed suggestions (e.g., "Which store is best for purchasing camping equipment?").

[0374] Step 18:

[0375] The device sends user feedback to the server.

[0376] Step 19:

[0377] The server sends a new request to the generative artificial intelligence based on the user's feedback. This request includes additional information about the user.

[0378] Step 20:

[0379] The generative artificial intelligence receives a follow-up request and generates additional specific suggestions.

[0380] Step 21:

[0381] The generative artificial intelligence sends the generated additional suggestions back to the server as a response.

[0382] Step 22:

[0383] The server receives additional responses from the generative artificial intelligence, formats them into the appropriate data format, and sends them to the terminal.

[0384] Step 23:

[0385] The device displays the received additional suggestions to the user.

[0386] Through this series of steps, the system takes into account not only the user's requests but also their emotional state, enabling it to specifically suggest the optimal service usage scenario.

[0387] (Example 2)

[0388] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0389] Traditional systems often fail to consider user emotions when proposing services, resulting in the provision of information and service suggestions that are not optimal for user needs. This can lead to a diminished user experience and a decrease in the frequency of service usage. Furthermore, the lack of sufficient functionality to quickly revise suggestions based on feedback makes it difficult to promptly provide users with the specific information they require.

[0390] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting user information, means for transmitting the inputted user information and requests, means for capturing the facial expressions and voice of the user when they input a request and performing emotion analysis, means for analyzing the captured facial expressions and voice and transmitting the emotion analysis results, means for transmitting a request to a generative artificial intelligence based on the received user information, requests, and emotion analysis results, means for receiving a response from the generative artificial intelligence, means for formatting the received response into an appropriate data format and transmitting it to the user terminal for display, means for receiving feedback from the user, and means for transmitting a re-request to the generative artificial intelligence based on the feedback. This enables the realization of optimal service proposals that take into account the user's emotions and enables rapid re-proposals based on feedback.

[0391] "User information" refers to basic data such as the user's age, gender, and areas of interest.

[0392] A "request" refers to the content of a user's request to the system, such as their wishes, questions, or specific service usage scenarios.

[0393] A "terminal" is a device that functions as the user interface of a system, and is a device that performs tasks such as inputting user information, sending requests, and displaying results.

[0394] A "server" is a central processing unit that receives user information, requests, and sentiment analysis results, sends requests to generative artificial intelligence, and receives, processes, and transmits responses from generative artificial intelligence.

[0395] An "emotion engine" is software or a device that analyzes a user's emotional state by performing emotional analysis based on the user's facial expressions and voice data.

[0396] "Generative artificial intelligence" is an artificial intelligence system that generates optimal service suggestions based on received user information, requests, and sentiment analysis results.

[0397] "Feedback" refers to responses from users to proposed services or information, such as evaluations, additional requests, or questions.

[0398] "Formatting" is the process of converting responses and suggestions received from generative artificial intelligence into a format that is easy for the user to understand.

[0399] A "request" is an inquiry, such as instructions or questions, that a server sends to a generative artificial intelligence system based on user requests.

[0400] "Data format" refers to the format that represents the structure and layout of data, and specifically refers to formats such as HTML and JSON.

[0401] This invention is a system that analyzes a user's emotions and provides appropriate service suggestions based on the results. The system mainly consists of four elements: a user terminal, a server, a generative artificial intelligence system, and an emotion engine. These elements communicate with each other via the internet and operate in an integrated manner.

[0402] User terminal

[0403] The user terminal handles user information input, request input, emotion recognition, data transmission, and result display. The terminal performs the following specific actions:

[0404] 1. Enter user information

[0405] The device screen displays a screen where the user can enter basic information such as age, gender, and areas of interest.

[0406] Example: The user enters "30s, male, enjoys outdoor activities."

[0407] 2. Input of request

[0408] The user enters a specific request via their device. For example, they might type, "Please tell me how I can use PayPay in my daily life."

[0409] 3. Recognition of emotions

[0410] The device uses its camera and microphone to capture the user's facial expressions and voice, and sends them to the emotion engine. This emotion engine then uses specialized software to perform emotion analysis.

[0411] Specific example: The emotion engine analyzes the user's facial expression and determines it to be "positive."

[0412] 4. Sending data to the server

[0413] The terminal sends user information, requests, and sentiment analysis results to the server.

[0414] server

[0415] The server receives data sent from the user terminal, sends a request to the generative artificial intelligence, and receives a response. The specific operation of the server is as follows:

[0416] 5. Processing of received data

[0417] The server stores user information, request details, and sentiment analysis results in a database.

[0418] 6. Sending requests to generative artificial intelligence

[0419] The server generates and sends a prompt message to the generative artificial intelligence based on the received data.

[0420] Example of a prompt:

[0421] Please create scenarios showing how a man in his 30s, who enjoys the outdoors and has a positive outlook, can use PayPay in his daily life.

[0422] 7. Receiving a response from a generative artificial intelligence.

[0423] Generative artificial intelligence generates suggestions tailored to the user's daily life and sends them back to the server.

[0424] Specific example: The following suggestions may be generated.

[0425] 1. Purchase camping equipment

[0426] 2. Payment of admission fees for outdoor events

[0427] 3. Paying for meals at restaurants

[0428] 4. Paying for fuel at a gas station

[0429] 5. Payment of public transport fares

[0430] 6. Payment of sports facility usage fees

[0431] 7. Outdoor-related subscription services

[0432] 8. Payment of travel expenses at the travel agency

[0433] 9. Booking and payment for accommodation

[0434] 10. Booking and Payment for Activities

[0435] 8. Formatting and sending the results

[0436] The server formats the generated proposals into the appropriate data format and sends them to the user's terminal.

[0437] Feedback processing

[0438] Users can provide feedback on proposals and then make revised proposals based on that feedback. The feedback process is as follows:

[0439] 9. Receiving user feedback

[0440] Users provide feedback on the suggestions via their devices.

[0441] Specific example: A user provides feedback asking, "Which store is best for purchasing camping equipment?"

[0442] 10. Processing and resubmitting feedback

[0443] The server receives user feedback and, based on that, sends a new request to the generative artificial intelligence. This new request generates a new suggestion, which is then sent back to the user's terminal.

[0444] In this way, the present invention makes it possible to propose optimal services that take user emotions into consideration and improve the user experience.

[0445] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0446] Step 1:

[0447] Entering user information

[0448] The device displays a screen for the user to enter basic information such as age, gender, and areas of interest. The user then enters their information and presses the submit button.

[0449] Input: Information about the user's age, gender, and areas of interest.

[0450] Output: Input user information data.

[0451] Specific operation: The terminal converts user information data into JSON format and sends it to the server.

[0452] Step 2:

[0453] Request Input

[0454] The terminal provides an interface that allows the user to enter a specific request. The user enters the request and presses the submit button.

[0455] Input: The request entered by the user.

[0456] Output: The input request data.

[0457] Specific operation: The terminal converts the request data into JSON format and sends it to the server.

[0458] Step 3:

[0459] Recognition of emotions

[0460] The device uses its camera and microphone to capture the user's facial expressions and voice as they input requests. This data is then sent to an emotion engine for analysis.

[0461] Input: User's facial expression data and voice data.

[0462] Output: Emotion analysis results.

[0463] Specific operation: The device sends the captured data to the emotion engine in real time, and the emotion engine returns the emotion analysis result, such as "positive."

[0464] Step 4:

[0465] Sending data to the server

[0466] The device sends user information, requests, and sentiment analysis results to the server.

[0467] Inputs: User information data, request data, sentiment analysis results.

[0468] Output: Server received data (user information, requests, sentiment analysis results).

[0469] Specific operation: The terminal combines this data into one file and sends it to the server.

[0470] Step 5:

[0471] Processing of received data

[0472] The server stores the received user information, request details, and sentiment analysis results in its database. It then prepares to send the request to the generative artificial intelligence.

[0473] Input: Server received data (user information, requests, sentiment analysis results).

[0474] Output: Request data to be sent to the generative artificial intelligence.

[0475] Specific operation: The server analyzes and organizes the received data and generates a prompt message.

[0476] Step 6:

[0477] Sending a request to a generative artificial intelligence

[0478] The server sends a prompt message to the generative artificial intelligence.

[0479] Input: Request data (prompt text).

[0480] Output: Response from a generative artificial intelligence.

[0481] Specific operation: The server generates a prompt message like the following and sends it to the generative artificial intelligence:

[0482] Please create scenarios showing how a man in his 30s, who enjoys the outdoors and has a positive outlook, can use PayPay in his daily life.

[0483] Step 7:

[0484] Received a response from a generative artificial intelligence.

[0485] The server receives responses from the generative artificial intelligence. These responses include suggestions tailored to the user's daily life scenarios.

[0486] Input: Response data from a generative artificial intelligence.

[0487] Output: Proposed content.

[0488] Specific action: Save the received proposal content to the database.

[0489] Step 8:

[0490] Formatting and sending the results

[0491] The server formats the generated proposals into the appropriate data format and sends them to the user's terminal.

[0492] Input: Proposal content.

[0493] Output: Proposed data format for display.

[0494] Specific operation: Convert suggestions received from generative artificial intelligence (e.g., "Purchase camping equipment," "Pay admission fee for outdoor event," etc.) into HTML or JSON format and send them to the terminal.

[0495] Step 9:

[0496] Receiving user feedback

[0497] The device displays a form for the user to enter feedback on the proposal. The user enters their feedback and presses the submit button.

[0498] Input: User feedback.

[0499] Output: Feedback data.

[0500] Specific operation: The user enters "Which store is best for purchasing camping equipment?" and sends this to the server.

[0501] Step 10:

[0502] Processing and resubmitting feedback

[0503] Based on the feedback received from the user, the server sends a new request to the generative artificial intelligence.

[0504] Input: Feedback data.

[0505] Output: Response to the re-request.

[0506] Specific operation: The server uses feedback data to generate a new prompt and sends it to the generative artificial intelligence. Then, it receives a revised suggestion and sends it back to the user terminal.

[0507] In this way, the system can provide optimal service suggestions that take user emotions into consideration, thereby improving the user experience.

[0508] (Application Example 2)

[0509] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0510] Conventional real-time service suggestion systems lacked the ability to consider the user's emotional state. As a result, optimal suggestions were not made at the timing and with the nuances the user desired, leading to decreased service satisfaction. This invention aims to improve the user experience by recognizing the user's emotions and making suggestions based on those emotions.

[0511] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes a screen display means for inputting user information, a means for transmitting the input user information and requests to the server, a means for capturing the user's facial expressions and voice and transmitting them to the emotion engine, a means for transmitting the emotion analysis results analyzed by the emotion engine to the server, a means for sending a request to a generative artificial intelligence based on the received data, a means for receiving a response from the generative artificial intelligence, a means for transmitting the received response to the user terminal and displaying it, a means for receiving feedback from the user, and a means for sending a re-request to the generative artificial intelligence based on the feedback. This makes it possible to propose services optimized for the user's emotional state.

[0512] "User information input screen display means" refers to a device or program that provides an interface that allows a user to input their basic information (age, gender, areas of interest, etc.) and requests.

[0513] "Means for transmitting entered user information and requests to the server" refers to a device or program that transmits information and requests entered by a user to a server via the Internet.

[0514] "Means for capturing user facial expressions and voice and transmitting them to an emotion engine" refers to a device or program that acquires the user's face and voice using a camera and microphone and sends that data to an emotion analysis device.

[0515] "Means for sending emotion analysis results analyzed by the emotion engine to the server" refers to a device or program in which an emotion analysis device analyzes the user's emotional state and sends the analysis results to the server.

[0516] "Means for sending requests to generative artificial intelligence based on received data" refers to a device or program that sends appropriate requests to generative artificial intelligence based on user information, requests, and sentiment analysis results received by the server.

[0517] "Means for receiving responses from generative artificial intelligence" refers to a device or program that receives proposals and responses generated by generative artificial intelligence on a server.

[0518] "Means for sending and displaying received responses to a user terminal" refers to a device or program in which the server formats suggestions and responses obtained from a generative artificial intelligence into an appropriate format, and sends and displays them to the user's terminal.

[0519] "Means for receiving user feedback" refers to a device or program that collects opinions and requests from users regarding the proposed content.

[0520] "Means of sending a re-request to a generative artificial intelligence based on feedback" refers to a device or program that analyzes user feedback and sends a re-request to a generative artificial intelligence.

[0521] This invention is a system that recognizes user emotions and makes service suggestions that take them into consideration. The specific form of implementation is described below. The system uses a user terminal, a server, a generative artificial intelligence system, and an emotion engine.

[0522] User terminal

[0523] User terminals are devices such as smartphones and smart glasses. These terminals have the following main functions:

[0524] 1. Enter user information

[0525] The user terminal displays a screen for entering user information. Here, the user enters basic information such as their age, gender, and areas of interest, as well as specific requests.

[0526] 2. Emotion recognition

[0527] When a user enters information, the device's built-in camera and microphone are used to capture the user's facial expressions and voice. This data is then sent to the emotion engine for analysis.

[0528] server

[0529] The server performs the following roles:

[0530] 1. Data transmission

[0531] It receives user information, requests, and sentiment analysis results sent from the user's terminal, and converts them into a request format suitable for generative artificial intelligence.

[0532] 2. Requests to Generative Artificial Intelligence

[0533] A request is sent to a generative AI model (for example, GPT-3®) to generate specific proposals.

[0534] 3. Receiving and formatting the response

[0535] The system formats the response received from the generative artificial intelligence into an appropriate data format and sends it to the user's terminal.

[0536] Generative artificial intelligence

[0537] Generative artificial intelligence receives requests from a server and generates optimal suggestions. This generation process takes into account the user's basic information, requests, and emotional state.

[0538] Emotional Engine

[0539] The emotion engine analyzes the user's facial expressions and voice to determine their emotional state. The results are then sent to the server.

[0540] Hardware and software used

[0541] User devices: smartphones, smart glasses

[0542] Server: Cloud server or on-premises server

[0543] Emotion Engine: Emotion Recognition API

[0544] Generative artificial intelligence: Generative AI models (e.g., GPT-3)

[0545] Specific example of processing

[0546] The following is an example of how to specifically suggest the most suitable product when a user is feeling relaxed.

[0547] 1. Enter user information:

[0548] Example: The user enters information such as "30s, female, enjoys outdoor activities" on their smartphone.

[0549] The user enters a request such as, "I want to relax, I want to find a book."

[0550] 2. Recognition of emotions:

[0551] The user's device captures the user's facial expressions, and the emotion engine analyzes that data to determine that the user is "relaxed."

[0552] 3. Example of a prompt:

[0553] "Generate product suggestions suitable for a female user in her 30s who enjoys the outdoors and wants to relax. The user's basic information is as follows: Age: 30s, Gender: Female, Interests: Outdoors, Emotion: Want to relax. Please generate five suggestions."

[0554] Sending this prompt to the generating AI model yields suggestions such as:

[0555] 1. Aromatherapy Candles

[0556] 2. Relaxation chair

[0557] 3. Relaxation music subscription

[0558] 4. Herb tea set

[0559] 5. Essential oils

[0560] This suggestion is displayed on the user's terminal, providing the user with optimal product knowledge. In this way, a system can be built that enables service suggestions that take into account the user's emotional state.

[0561] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0562] Step 1:

[0563] The user enters basic information and requests using a device. The user enters basic information such as age, gender, and areas of interest, as well as specific requests (e.g., "I want to relax, I want to find a book") through an input form displayed on the screen of a smartphone or smart glasses.

[0564] Input: User's basic information and requests

[0565] Output: The information entered by the user is temporarily stored on the device.

[0566] Step 2:

[0567] The device captures the user's facial expressions and voice. It uses the device's built-in camera and microphone to collect user facial expression and voice data.

[0568] Input: User's facial expressions and voice

[0569] Output: Captured facial expression data and audio data

[0570] Step 3:

[0571] The device sends the captured data to an emotion engine for analysis. The device then sends the data to emotion analysis software such as the Emotion Recognition API to obtain the emotion analysis results.

[0572] Input: Facial expression data and audio data

[0573] Output: Emotion analysis results (e.g., "Relaxed state")

[0574] Step 4:

[0575] The sentiment analysis results are sent from the terminal to the server. Along with the analysis results, the terminal also sends the user's basic information and requests to the server.

[0576] Input: Sentiment analysis results, basic information, request

[0577] Output: Data sent to the server

[0578] Step 5:

[0579] The server sends a request to the generative artificial intelligence based on the data it receives. The server analyzes the received data, converts it into a request format suitable for the generative AI model, and sends it. For example,

[0580] "Generate product suggestions suitable for a female user in her 30s who enjoys the outdoors and wants to relax. The user's basic information is as follows: Age: 30s, Gender: Female, Interests: Outdoors, Emotion: Want to relax. Please generate five suggestions."

[0581] Input: User information, requests, sentiment analysis results

[0582] Output: Request sent to the generative artificial intelligence

[0583] Step 6:

[0584] The server receives the response from the generative artificial intelligence. The server receives the suggestions sent by the generative AI model and performs analysis.

[0585] Input: Response from generative artificial intelligence

[0586] Output: Proposed ideas (e.g., aromatherapy candles, relaxation chairs, etc.)

[0587] Step 7:

[0588] The server sends the received response to the user terminal and displays it on the terminal. The server formats the proposed content into an appropriate format and sends it to the user terminal for display.

[0589] Input: Proposal Content

[0590] Output: Data sent to the user terminal

[0591] Step 8:

[0592] The device sends user feedback to the server. The user enters feedback on the displayed suggestions, and the device sends that information to the server.

[0593] Input: User feedback

[0594] Output: Feedback sent to the server

[0595] Step 9:

[0596] The server sends a new request to the generative artificial intelligence based on the feedback. The feedback is analyzed, and another request is sent to the generative AI model to generate new suggestions.

[0597] Input: Feedback Information

[0598] Output: Re-request sent to the generative AI

[0599] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0600] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0601] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0602] [Second Embodiment]

[0603] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0604] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0605] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0606] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0607] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0608] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0609] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0610] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0611] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0612] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0613] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0614] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0615] One embodiment of the present invention describes below a specific system for a user to incorporate a particular service into their life and make the most of it. This system provides the user with useful information through the input of user information, generation of suggestions by generative artificial intelligence, display of the suggested content, and processing of user feedback.

[0616] System-wide configuration

[0617] The system primarily consists of three parts: user terminals, servers, and generative artificial intelligence. These elements communicate and cooperate with each other via the internet to enable the overall system to function.

[0618] User terminal

[0619] 1. Enter basic information

[0620] The terminal first displays a screen for user information input. The user enters basic information such as their age, gender, and areas of interest. This information is later processed by generative artificial intelligence.

[0621] 2. Input of request

[0622] The terminal also provides an interface for receiving specific requests from the user. For example, the user can input a specific request such as, "Please tell me about situations in which PayPay can be used in daily life."

[0623] 3. Sending data to the server

[0624] The terminal sends user information and requests to the server. The transmitted data is used in the next processing step.

[0625] server

[0626] 4. Data processing and request generation

[0627] The server processes the received data, including user information and request details, and uses this information to generate appropriate requests for the generative artificial intelligence. For example, it might send a request to the generative AI such as, "Generate scenarios showing how a man in his 30s who enjoys outdoor activities can use PayPay in his daily life."

[0628] 5. Receiving a response from a generative artificial intelligence.

[0629] The server receives a response from the generative artificial intelligence. This response includes specific suggestions tailored to the user's life scenarios.

[0630] 6. Formatting and sending the results

[0631] The server formats the response from the generative artificial intelligence into an appropriate format and sends it to the user terminal.

[0632] Generative artificial intelligence

[0633] 7. Proposal generation

[0634] Generative artificial intelligence receives requests from a server and generates suggestions for specific life scenarios based on user information and requests. These suggestions show how the user can specifically utilize the service.

[0635] Specific example

[0636] 1. Enter user information

[0637] Example: The user enters information on their device such as "30s, male, enjoys outdoor activities."

[0638] 2. Input of request

[0639] Example: The user enters a request such as, "Tell me 10 situations in everyday life where PayPay can be used."

[0640] 3. Sending data to the server

[0641] The terminal sends user information and requests to the server.

[0642] 4. Data processing and request generation

[0643] Based on the data the server receives, it sends a request to the generative artificial intelligence saying, "Please generate scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[0644] 5. Receiving a response from a generative artificial intelligence.

[0645] The generative artificial intelligence generates the following suggestions and returns them to the server:

[0646] 1. Purchase camping equipment

[0647] 2. Payment of admission fees for outdoor events

[0648] 3. Paying for meals at restaurants

[0649] 4. Paying for fuel at a gas station

[0650] 5. Payment of public transport fares

[0651] 6. Payment of sports facility usage fees

[0652] 7. Outdoor-related subscription services

[0653] 8. Payment of travel expenses at the travel agency

[0654] 9. Booking and payment for accommodation

[0655] 10. Booking and Payment for Activities

[0656] 6. Formatting and sending the results

[0657] The server formats the generated proposal into the appropriate data format and sends it to the user's terminal.

[0658] 7. Displaying results and receiving feedback

[0659] The device displays the results to the user, who then provides feedback on the suggestions. For example, the user might input, "Which store is best for buying camping equipment?"

[0660] 8. Processing and resubmitting feedback

[0661] The server receives user feedback and sends a request to the generative artificial intelligence. The generative AI generates specific suggestions again, which the server receives, formats appropriately, and sends back to the user's terminal.

[0662] In this way, the system incorporates user feedback and has the function of continuously and concretely suggesting how users can utilize the services provided in their actual daily lives.

[0663] The following describes the processing flow.

[0664] Step 1:

[0665] The device displays a screen that prompts the user to enter basic information (age, gender, areas of interest, etc.).

[0666] Step 2:

[0667] The user enters information such as their age, gender, and areas of interest into the displayed input form.

[0668] Step 3:

[0669] The device sends basic information entered by the user to the server. This data includes the user's age, gender, and areas of interest.

[0670] Step 4:

[0671] The terminal displays an interface that prompts the user to input requests regarding specific service usage scenarios (e.g., "Please tell me about scenarios in which PayPay can be used in daily life").

[0672] Step 5:

[0673] The user enters their request into the terminal.

[0674] Step 6:

[0675] The terminal sends the user's request to the server. The transmitted data includes the details of the user's request.

[0676] Step 7:

[0677] The server converts the received user information and request into a request format appropriate for the generative artificial intelligence.

[0678] Step 8:

[0679] The server sends a request to the generative artificial intelligence. The request includes the user's age, gender, areas of interest, and the content of the request.

[0680] Step 9:

[0681] Generative artificial intelligence processes received requests and generates specific suggestions that correspond to the user's life scenarios.

[0682] Step 10:

[0683] The generative artificial intelligence sends the generated suggestions back to the server as a response.

[0684] Step 11:

[0685] The server receives the response from the generative artificial intelligence and formats it into an appropriate data format (e.g., JSON).

[0686] Step 12:

[0687] The server sends the formatted proposal to the terminal.

[0688] Step 13:

[0689] The device displays received suggestions to the user. These suggestions include specific examples of service usage tailored to the user's lifestyle.

[0690] Step 14:

[0691] Users provide feedback on the displayed suggestions (e.g., "Which store is best for purchasing camping equipment?").

[0692] Step 15:

[0693] The device sends user feedback to the server.

[0694] Step 16:

[0695] The server sends a new request to the generative artificial intelligence based on the user's feedback. This request includes additional information about the user.

[0696] Step 17:

[0697] The generative artificial intelligence receives a follow-up request and generates additional specific suggestions.

[0698] Step 18:

[0699] The generative artificial intelligence sends the generated additional suggestions back to the server as a response.

[0700] Step 19:

[0701] The server receives additional responses from the generative artificial intelligence, formats them into the appropriate data format, and sends them to the terminal.

[0702] Step 20:

[0703] The device displays the received additional suggestions to the user.

[0704] This series of steps allows the system to continuously provide specific suggestions that align with the user's requirements.

[0705] (Example 1)

[0706] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0707] Conventional systems struggled to generate personalized suggestions based on individual user requests, failing to improve user satisfaction. Furthermore, they lacked sufficient mechanisms for efficiently incorporating user feedback and updating suggestions.

[0708] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0709] In this invention, the server includes means for processing user information and requests, generating and sending prompt messages to a generative artificial intelligence; means for receiving responses from the generative artificial intelligence; and means for formatting the received responses into an appropriate data format and sending them to the user terminal. This makes it possible to generate personalized suggestions based on the user's individual requests and to efficiently incorporate feedback to update the suggestions.

[0710] "Means for inputting basic user information" refers to providing an interface for inputting basic user information such as age, gender, and areas of interest.

[0711] "Means for inputting user requests" refers to means of providing an interface for users to input specific requests (for example, questions about how to use a particular service).

[0712] "Means for sending entered user information and requests to the server" refers to means for sending the basic information and requests entered by the user to the server via the network.

[0713] "Means for processing user information and requests, and generating and sending prompt messages to a generative artificial intelligence" refers to means for creating and sending appropriate prompt messages to a generative artificial intelligence based on the received user information and requests.

[0714] "Means for receiving responses from generative artificial intelligence" refers to means for the server to receive proposals and responses generated by generative artificial intelligence.

[0715] "Means for formatting the received response into an appropriate data format and sending it to the user terminal" refers to means of converting the response received from the generative artificial intelligence into a format that is easy for the user to understand and sending it again to the user terminal via the network.

[0716] "Means for displaying responses to the user and receiving feedback from the user" refers to means that the user terminal provides an interface for displaying responses from generative artificial intelligence to the user and for receiving feedback from the user.

[0717] "Method for sending a new request to a generative artificial intelligence based on feedback" refers to a method for generating a new prompt message based on feedback received from the user and sending it to the generative artificial intelligence.

[0718] Overall system configuration

[0719] This invention is a system that enables users to incorporate specific services into their lives and make the most of them. This system provides users with useful information through the input of user information, generation of suggestions by generative artificial intelligence, display of the suggested content, and processing of user feedback.

[0720] Hardware and software to be used

[0721] The system primarily consists of three parts: user terminals, servers, and generative artificial intelligence. These elements communicate with each other via the internet and work together as a whole system. User terminals include smartphones, tablets, and personal computers. Servers include web servers and cloud servers. Generative artificial intelligence utilizes computing resources (e.g., cloud-based AI services) capable of executing appropriate AI models.

[0722] Program processing

[0723] The specific processing details of the system are as follows:

[0724] Entering user information

[0725] The user terminal displays a screen prompting the user to enter basic information such as age, gender, and areas of interest. The user enters information such as "30s, male, enjoys outdoor activities." This information is encoded in JSON format or similar and sent to the server.

[0726] Request Input

[0727] The user terminal provides an interface for the user to input specific requests. The user might input a specific request such as, "Please tell me about situations in daily life where PayPay can be used."

[0728] Sending data to the server

[0729] The user terminal sends the entered user information and requests to the server. The server receives this information and parses it.

[0730] Data processing and request generation

[0731] The server generates and sends a prompt message to the generative artificial intelligence based on user information and the request. For example, a request such as "Generate scenarios showing how a man in his 30s who enjoys outdoor activities can use PayPay in his daily life" might be generated.

[0732] Received a response from a generative artificial intelligence.

[0733] Generative artificial intelligence receives requests and generates suggestions for specific life scenarios based on user information. For example, it might generate suggestions such as "purchase camping equipment" or "pay admission fees for outdoor events." These responses are then returned to the server.

[0734] Formatting and sending the results

[0735] The server formats the response received from the generative artificial intelligence into an appropriate data format and sends it back to the user terminal. In this process, the data is formatted into a list and then encoded again in JSON format.

[0736] Displaying results and accepting feedback

[0737] The user terminal displays the received suggestions to the user. The user then reviews these suggestions and provides more specific feedback. For example, they might input feedback such as, "Which store is best for purchasing camping equipment?"

[0738] Processing and resubmitting feedback

[0739] Based on the feedback received from the user, the server generates a new prompt and sends it to the generative artificial intelligence. The generative AI then generates specific suggestions, which the server receives, formats appropriately, and sends back to the user's terminal.

[0740] Specific example

[0741] 1. Enter user information

[0742] The user enters information on their device such as "30s, male, enjoys outdoor activities."

[0743] 2. Input of request

[0744] The user enters a request asking for "10 ways to use PayPay in everyday life."

[0745] 3. Sending data to the server

[0746] The user terminal sends the entered data to the server.

[0747] 4. Data processing and request generation

[0748] The server parses the data and generates and sends a prompt message: "Generate scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[0749] 5. Receiving a response from a generative artificial intelligence.

[0750] Generative artificial intelligence generates suggestions and returns them to the server. Examples: "Purchase camping equipment," "Pay admission fee for outdoor event," etc.

[0751] 6. Formatting and sending the results

[0752] The server formats the proposal and sends it to the user's terminal.

[0753] 7. Displaying results and receiving feedback

[0754] The user's device displays suggestions to the user, who then provides more specific feedback.

[0755] 8. Processing and resubmitting feedback

[0756] The server processes the feedback and sends a new prompt to the generative artificial intelligence again, which generates a new suggestion, formats it again, and sends it to the user terminal.

[0757] The above describes a specific embodiment of the present invention, a system that efficiently provides personalized suggestions based on the individual needs of users and continuously improves while incorporating feedback.

[0758] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0759] Step 1: Enter user information

[0760] The user terminal displays a screen prompting the user to enter basic information such as age, gender, and areas of interest. The user enters information such as "30s, male, enjoys outdoor activities" into the input screen. The entered information is temporarily stored on the terminal. Input data formats such as JSON and XML are used.

[0761] Input: User's basic information (age, gender, areas of interest, etc.)

[0762] Output: Encoded user basic information data

[0763] Step 2: Enter the request

[0764] The user terminal provides an interface for entering specific requests. Users enter specific requests such as, "Please tell me about situations where PayPay can be used in daily life." The entered requests are then encoded into a data format.

[0765] Input: User request (e.g., "Please tell me about situations in daily life where PayPay can be used.")

[0766] Output: Encoded user request data

[0767] Step 3: Sending data to the server

[0768] The user terminal sends the entered user information and requests to the server. The HTTPS protocol is used for transmission, and the data is encoded in JSON format. Upon successful communication, the data is stored on the server.

[0769] Input: Encoded user basic information data and user request data

[0770] Output: User information and request data stored on the server

[0771] Step 4: Data processing and request generation

[0772] The server parses and analyzes the received user information and requests. For example, it extracts information such as age, gender, and areas of interest. Next, it generates a prompt message to send to the generative artificial intelligence based on that information. The prompt message is in a format such as, "Generate 10 scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[0773] Input: User information and request data stored on the server

[0774] Output: Generated prompt message

[0775] Step 5: Receiving a response from the generative artificial intelligence.

[0776] The server sends the generated prompt to the generative artificial intelligence (AI). The AI ​​receives the request, analyzes it based on the prompt, and generates specific suggestions to address the user's request. These suggestions are then returned to the server.

[0777] Input: Generated prompt message

[0778] Output: Response data from generative artificial intelligence (e.g., suggestions such as "purchase camping equipment" or "pay admission fee for an outdoor event")

[0779] Step 6: Format and submit the results

[0780] The server formats the response received from the generative artificial intelligence into an appropriate data format. The formatted data is then encoded again in JSON format and sent to the user's terminal.

[0781] Input: Response data from generative artificial intelligence

[0782] Output: Formatted proposal data

[0783] Step 7: Displaying results and receiving feedback

[0784] The user terminal displays a list of received suggestions to the user. The user reviews the suggestions and enters more specific feedback. For example, they might enter feedback such as, "Which store is best for purchasing camping equipment?" and click the submit button.

[0785] Input: Formatted proposal data

[0786] Output: User feedback (e.g., specific questions or requests)

[0787] Step 8: Processing feedback and resubmitting requests

[0788] The server parses and analyzes the feedback received from the user. Based on the feedback, it generates a new prompt and sends it to the generative artificial intelligence. The generative AI generates another suggestion, which the server receives, formats appropriately, and sends back to the user's terminal.

[0789] Input: User feedback data

[0790] Output: Newly generated prompt sentence and new response data from the generative AI.

[0791] Through the above processing steps, the system can efficiently provide personalized suggestions based on the user's individual requests and continuously improve by incorporating feedback.

[0792] (Application Example 1)

[0793] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0794] This invention relates to a system that generates suggestions tailored to the user's lifestyle and maximizes user convenience. In particular, to improve the shopping experience in physical stores, there is a need to provide optimized suggestions in real time using the user's current location information. However, conventional systems have the problem of not being able to improve user satisfaction because they do not provide insufficient suggestions based on user information and do not perform real-time optimization using the current location information.

[0795] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0796] In this invention, the server includes means for having a display device for inputting user information, means for transmitting the input user information and requests to a communication device, means for receiving responses from generative artificial intelligence, means for acquiring the user's current location information and optimizing response suggestions based on the location information, and means for displaying the generated suggestions in real time on a smartphone or smart glasses. This enables the user to fully enjoy the shopping experience in a physical store.

[0797] "User information" refers to basic data such as the user's age, gender, areas of interest, and current location.

[0798] A "display device" is an interface for users to input and confirm information, and includes devices such as smartphones and smart glasses.

[0799] A "communication device" is a device used to send user information and requests to a server.

[0800] "Generative artificial intelligence" refers to an artificial intelligence system that generates optimal suggestions based on user information and requests.

[0801] "Current location information" refers to the user's real-time geographical location data.

[0802] "Real-time" refers to a timeframe that responds immediately to the user's current actions and circumstances.

[0803] A "smartphone" is a type of mobile phone that is a device with multi-functional computer capabilities.

[0804] "Smart glasses" are wearable devices designed to enhance visual information.

[0805] The following describes specific embodiments for carrying out the present invention. The invention relates to a system for users to incorporate specific services into their lives and make the most of them. The specific configuration and operation will be described below.

[0806] System-wide configuration

[0807] The system primarily consists of the following elements: user terminals, servers, and generative artificial intelligence. These elements communicate with each other via the internet and cooperate to function as a whole system.

[0808] User terminal

[0809] A user terminal is equipped with a display device and provides an interface that allows the user to input information. Examples include smartphones and smart glasses.

[0810] 1. Enter basic information

[0811] The user enters basic information such as age, gender, and areas of interest on the display device of their user terminal.

[0812] 2. Input of request

[0813] Users utilize an interface to enter specific requests. For example, they might enter a request such as, "Please recommend some outdoor products."

[0814] 3. Sending data to the server

[0815] The user terminal transmits the entered user information and requests to the server via a communication device.

[0816] server

[0817] The server processes the data sent from the user terminal and generates appropriate requests for the generative artificial intelligence.

[0818] 4. Data processing and request generation

[0819] The server organizes the received user information and request details, and then sends the request to the generative artificial intelligence based on that information.

[0820] For example, generate a prompt such as, "Generate suggestions on what products a man in his 30s who enjoys outdoor activities should purchase at a physical store."

[0821] 5. Receiving a response from a generative artificial intelligence.

[0822] The server receives the response from the generative artificial intelligence and formats it into an appropriate format.

[0823] 6. Optimization and Display

[0824] The server obtains the user's current location information and optimizes response suggestions based on that location. The generated suggestions are then displayed in real time on the user's smartphone or smart glasses.

[0825] Generative artificial intelligence

[0826] Generative artificial intelligence receives requests from a server and generates specific lifestyle scenarios and product suggestions based on user information and requests.

[0827] Specific example

[0828] 1. Enter user information

[0829] Example: A user enters information such as "30s, male, enjoys outdoor activities" on their smartphone.

[0830] 2. Input of request

[0831] Example: A user voice-inputs, "Please recommend some outdoor products."

[0832] 3. Sending data to the server

[0833] The user terminal sends user information and requests to the server.

[0834] 4. Data processing and request generation

[0835] The server sends the prompt message, "A man in his 30s who enjoys the outdoors is seeking advice on recommended outdoor products available at physical stores," to the generative AI.

[0836] 5. Receiving a response from a generative artificial intelligence.

[0837] The generative artificial intelligence generates a response and returns it to the server. This response may include suggestions such as, "A new camping tent has recently arrived at a nearby sporting goods store."

[0838] 6. Optimization and Display

[0839] The server optimizes the suggested content based on the user's current location, sends it to the user's terminal, and displays it in real time.

[0840] In this way, the system incorporates user feedback and has the function of continuously and concretely suggesting how users can utilize the services provided in their actual daily lives.

[0841] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0842] Step 1:

[0843] The user enters basic information such as age, gender, and areas of interest using the display device on their terminal. For example, they might enter information such as "30s, male, enjoys outdoor activities." This input data is collected on the terminal.

[0844] Step 2:

[0845] The user terminal provides an interface for inputting specific requests from the user. Users input requests such as "Please recommend some outdoor products" via voice or text. This request data is also collected by the terminal.

[0846] Step 3:

[0847] The user terminal transmits the entered user information and requests to the server via a communication device. At this time, the input data and request data are combined into a single request packet and sent to the server.

[0848] Step 4:

[0849] The server processes the received user information and request details to generate an appropriate request for the generative artificial intelligence. For example, it might generate a prompt such as, "Generate suggestions on what products a man in his 30s who enjoys outdoor activities should purchase at a physical store." This prompt becomes the input data for the generative artificial intelligence.

[0850] Step 5:

[0851] Generative artificial intelligence receives prompt messages from a server and generates specific lifestyle scenarios and product suggestions based on user information and requests. For example, it might generate a suggestion such as, "A new camping tent has recently arrived at a nearby sports goods store." This suggestion is then sent back to the server.

[0852] Step 6:

[0853] The server receives a response from the generative artificial intelligence and obtains the user's current location information. Based on the obtained location information, it further optimizes the response. For example, it processes the data to provide information about the nearest store based on the user's location. This optimized suggestion is then sent to the user's terminal.

[0854] Step 7:

[0855] The user's device displays received suggestions in real time on their smartphone or smart glasses. This allows users to instantly see the best suggestions to enhance their in-store shopping experience.

[0856] Step 8:

[0857] Users provide feedback on the displayed suggestions. For example, they might enter feedback such as, "This information was helpful. I'd like to know more details." This user feedback is collected on the device.

[0858] Step 9:

[0859] The user terminal sends feedback from the user to the server. The feedback data arrives at the server.

[0860] Step 10:

[0861] Based on the feedback received, the server sends another request to the generative artificial intelligence. For example, it might generate a prompt such as, "A man in his 30s who enjoys the outdoors is looking for more detailed information on outdoor products," and send it to the generative AI.

[0862] Step 11:

[0863] The generative artificial intelligence generates a response again and sends it to the server. This process is repeated in accordance with user feedback, allowing the system to continuously provide the most relevant information to the user.

[0864] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0865] As one embodiment of the present invention, a specific embodiment of a system that recognizes a user's emotions and makes more appropriate service suggestions based on those emotions is described below. This system provides the user with optimal information through the input of user information, suggestion generation by generative artificial intelligence, emotion recognition by an emotion engine, display of the suggested content, and processing of user feedback.

[0866] System-wide configuration

[0867] The system primarily consists of the following four parts: user terminal, server, generative artificial intelligence, and emotion engine. These elements communicate with each other via the internet and cooperate to enable the overall system to function.

[0868] User terminal

[0869] 1. Enter basic information

[0870] The device first displays a screen for user information input. The user enters basic information such as their age, gender, and areas of interest. This information is later processed by generative artificial intelligence and an emotion engine.

[0871] 2. Input of request

[0872] The terminal provides an interface for users to input specific requests (e.g., "Please tell me about situations in which PayPay can be used in daily life").

[0873] 3. Recognition of emotions

[0874] The device captures the user's facial expressions and voice as they enter their requests and sends them to the emotion engine.

[0875] 4. Sending data to the server

[0876] The device sends user information, requests, and sentiment analysis results from the sentiment engine to the server. The transmitted data includes the user's age, gender, areas of interest, request content, and sentiment analysis results.

[0877] server

[0878] 5. Data processing and request generation

[0879] The server receives data, including user information, request details, and sentiment analysis results, and then converts it into an appropriate request format for the generative artificial intelligence. For example, it might send a request to the generative AI such as, "Generate scenarios showing how a man in his 30s who enjoys the outdoors and has positive emotions can use PayPay in his daily life."

[0880] 6. Receiving a response from a generative artificial intelligence.

[0881] The server receives a response from the generative artificial intelligence. This response includes specific suggestions tailored to the user's life scenarios and is customized based on the sentiment analysis results.

[0882] 7. Formatting and sending the results

[0883] The server formats the response from the generative artificial intelligence into an appropriate format and sends it to the user terminal.

[0884] Generative artificial intelligence

[0885] 8. Proposal generation

[0886] Generative artificial intelligence receives requests from the server and generates specific life scenario suggestions based on user information, requests, and sentiment analysis results. These suggestions show how the user can specifically use the service and also take into account the user's emotional state.

[0887] Specific example

[0888] 1. Enter user information

[0889] Example: The user enters information on their device such as "30s, male, enjoys outdoor activities."

[0890] 2. Input of request

[0891] Example: The user enters a request such as, "Tell me 10 situations in everyday life where PayPay can be used."

[0892] 3. Recognition of emotions

[0893] The device captures the user's facial expressions with its camera, and an emotion engine analyzes them to determine if they are "positive."

[0894] 4. Sending data to the server

[0895] The device sends user information, requests, and sentiment analysis results to the server.

[0896] 5. Data processing and request generation

[0897] Based on the data the server receives, it sends a request to the generative artificial intelligence saying, "Please generate scenarios showing how a man in his 30s, who enjoys the outdoors and has positive emotions, can use PayPay in his daily life."

[0898] 6. Receiving a response from a generative artificial intelligence.

[0899] The generative artificial intelligence generates the following suggestions and returns them to the server:

[0900] 1. Purchase camping equipment

[0901] 2. Payment of admission fees for outdoor events

[0902] 3. Paying for meals at restaurants

[0903] 4. Paying for fuel at a gas station

[0904] 5. Payment of public transport fares

[0905] 6. Payment of sports facility usage fees

[0906] 7. Outdoor-related subscription services

[0907] 8. Payment of travel expenses at the travel agency

[0908] 9. Booking and payment for accommodation

[0909] 10. Booking and Payment for Activities

[0910] 7. Formatting and sending the results

[0911] The server formats the generated proposal into the appropriate data format and sends it to the user's terminal.

[0912] 8. Displaying results and receiving feedback

[0913] The device displays the results to the user, who then provides feedback on the suggestions. For example, the user might input, "Which store is best for purchasing camping equipment?"

[0914] 9. Processing and resubmitting feedback

[0915] The server receives user feedback and sends a request to the generative artificial intelligence. The generative AI generates specific suggestions again, which the server receives, formats appropriately, and sends back to the user's terminal.

[0916] Through these steps, the system takes user emotions into consideration and proposes services in a more appropriate way. This allows users to more concretely integrate the services offered into their own lives.

[0917] The following describes the processing flow.

[0918] Step 1:

[0919] The device displays a screen that prompts the user to enter basic information (age, gender, areas of interest, etc.).

[0920] Step 2:

[0921] The user enters information such as their age, gender, and areas of interest into the displayed input form.

[0922] Step 3:

[0923] The device sends basic information entered by the user to the server. This data includes the user's age, gender, and areas of interest.

[0924] Step 4:

[0925] The terminal displays an interface that prompts the user to input requests regarding specific service usage scenarios (e.g., "Please tell me about scenarios in which PayPay can be used in daily life").

[0926] Step 5:

[0927] The user enters their request into the terminal.

[0928] Step 6:

[0929] The terminal sends the user's request to the server. The transmitted data includes the details of the user's request.

[0930] Step 7:

[0931] The device captures the user's facial expressions and voice as they enter their requests and sends this information to the emotion engine.

[0932] Step 8:

[0933] The emotion engine analyzes captured facial and audio data to recognize the user's emotional state (e.g., positive, negative, neutral).

[0934] Step 9:

[0935] The emotion engine sends the analysis results to the server. The transmitted data includes the user's emotional state.

[0936] Step 10:

[0937] The server converts the received user information, request details, and sentiment analysis results into a request format appropriate for the generative artificial intelligence.

[0938] Step 11:

[0939] The server sends a request to the generative artificial intelligence. The request includes the user's age, gender, areas of interest, request content, and sentiment analysis results.

[0940] Step 12:

[0941] Generative artificial intelligence processes received requests and generates specific suggestions tailored to the user's life scenarios. These suggestions also reflect the user's emotional state.

[0942] Step 13:

[0943] The generative artificial intelligence sends the generated suggestions back to the server as a response.

[0944] Step 14:

[0945] The server receives the response from the generative artificial intelligence and formats it into an appropriate data format (e.g., JSON).

[0946] Step 15:

[0947] The server sends the formatted proposal to the terminal.

[0948] Step 16:

[0949] The device displays received suggestions to the user. These suggestions include specific examples of service usage tailored to the user's lifestyle.

[0950] Step 17:

[0951] Users provide feedback on the displayed suggestions (e.g., "Which store is best for purchasing camping equipment?").

[0952] Step 18:

[0953] The device sends user feedback to the server.

[0954] Step 19:

[0955] The server sends a new request to the generative artificial intelligence based on the user's feedback. This request includes additional information about the user.

[0956] Step 20:

[0957] The generative artificial intelligence receives a follow-up request and generates additional specific suggestions.

[0958] Step 21:

[0959] The generative artificial intelligence sends the generated additional suggestions back to the server as a response.

[0960] Step 22:

[0961] The server receives additional responses from the generative artificial intelligence, formats them into the appropriate data format, and sends them to the terminal.

[0962] Step 23:

[0963] The device displays the received additional suggestions to the user.

[0964] Through this series of steps, the system takes into account not only the user's requests but also their emotional state, enabling it to specifically suggest the optimal service usage scenario.

[0965] (Example 2)

[0966] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0967] Traditional systems often fail to consider user emotions when proposing services, resulting in the provision of information and service suggestions that are not optimal for user needs. This can lead to a diminished user experience and a decrease in the frequency of service usage. Furthermore, the lack of sufficient functionality to quickly revise suggestions based on feedback makes it difficult to promptly provide users with the specific information they require.

[0968] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting user information, means for transmitting the inputted user information and requests, means for capturing the facial expressions and voice of the user when they input a request and performing emotion analysis, means for analyzing the captured facial expressions and voice and transmitting the emotion analysis results, means for transmitting a request to a generative artificial intelligence based on the received user information, requests, and emotion analysis results, means for receiving a response from the generative artificial intelligence, means for formatting the received response into an appropriate data format and transmitting it to the user terminal for display, means for receiving feedback from the user, and means for transmitting a re-request to the generative artificial intelligence based on the feedback. This enables the realization of optimal service proposals that take into account the user's emotions and enables rapid re-proposals based on feedback.

[0969] "User information" refers to basic data such as the user's age, gender, and areas of interest.

[0970] A "request" refers to the content of a user's request to the system, such as their wishes, questions, or specific service usage scenarios.

[0971] A "terminal" is a device that functions as the user interface of a system, and is a device that performs tasks such as inputting user information, sending requests, and displaying results.

[0972] A "server" is a central processing unit that receives user information, requests, and sentiment analysis results, sends requests to generative artificial intelligence, and receives, processes, and transmits responses from generative artificial intelligence.

[0973] An "emotion engine" is software or a device that analyzes a user's emotional state by performing emotional analysis based on the user's facial expressions and voice data.

[0974] "Generative artificial intelligence" is an artificial intelligence system that generates optimal service suggestions based on received user information, requests, and sentiment analysis results.

[0975] "Feedback" refers to responses from users to proposed services or information, such as evaluations, additional requests, or questions.

[0976] "Formatting" is the process of converting responses and suggestions received from generative artificial intelligence into a format that is easy for the user to understand.

[0977] A "request" is an inquiry, such as instructions or questions, that a server sends to a generative artificial intelligence system based on user requests.

[0978] "Data format" refers to the format that represents the structure and layout of data, and specifically refers to formats such as HTML and JSON.

[0979] This invention is a system that analyzes a user's emotions and provides appropriate service suggestions based on the results. The system mainly consists of four elements: a user terminal, a server, a generative artificial intelligence system, and an emotion engine. These elements communicate with each other via the internet and operate in an integrated manner.

[0980] User terminal

[0981] The user terminal handles user information input, request input, emotion recognition, data transmission, and result display. The terminal performs the following specific actions:

[0982] 1. Enter user information

[0983] The device screen displays a screen where the user can enter basic information such as age, gender, and areas of interest.

[0984] Example: The user enters "30s, male, enjoys outdoor activities."

[0985] 2. Input of request

[0986] The user enters a specific request via their device. For example, they might type, "Please tell me how I can use PayPay in my daily life."

[0987] 3. Recognition of emotions

[0988] The device uses its camera and microphone to capture the user's facial expressions and voice, and sends them to the emotion engine. This emotion engine then uses specialized software to perform emotion analysis.

[0989] Specific example: The emotion engine analyzes the user's facial expression and determines it to be "positive."

[0990] 4. Sending data to the server

[0991] The terminal sends user information, requests, and sentiment analysis results to the server.

[0992] server

[0993] The server receives data sent from the user terminal, sends a request to the generative artificial intelligence, and receives a response. The specific operation of the server is as follows:

[0994] 5. Processing of received data

[0995] The server stores user information, request details, and sentiment analysis results in a database.

[0996] 6. Sending requests to generative artificial intelligence

[0997] The server generates and sends a prompt message to the generative artificial intelligence based on the received data.

[0998] Example of a prompt:

[0999] Please create scenarios showing how a man in his 30s, who enjoys the outdoors and has a positive outlook, can use PayPay in his daily life.

[1000] 7. Receiving a response from a generative artificial intelligence.

[1001] Generative artificial intelligence generates suggestions tailored to the user's daily life and sends them back to the server.

[1002] Specific example: The following suggestions may be generated.

[1003] 1. Purchase camping equipment

[1004] 2. Payment of admission fees for outdoor events

[1005] 3. Paying for meals at restaurants

[1006] 4. Paying for fuel at a gas station

[1007] 5. Payment of public transport fares

[1008] 6. Payment of sports facility usage fees

[1009] 7. Outdoor-related subscription services

[1010] 8. Payment of travel expenses at the travel agency

[1011] 9. Booking and payment for accommodation

[1012] 10. Booking and Payment for Activities

[1013] 8. Formatting and sending the results

[1014] The server formats the generated proposals into the appropriate data format and sends them to the user's terminal.

[1015] Feedback processing

[1016] Users can provide feedback on proposals and then make revised proposals based on that feedback. The feedback process is as follows:

[1017] 9. Receiving user feedback

[1018] Users provide feedback on the suggestions via their devices.

[1019] Specific example: A user provides feedback asking, "Which store is best for purchasing camping equipment?"

[1020] 10. Processing and resubmitting feedback

[1021] The server receives user feedback and, based on that, sends a new request to the generative artificial intelligence. This new request generates a new suggestion, which is then sent back to the user's terminal.

[1022] In this way, the present invention makes it possible to propose optimal services that take user emotions into consideration and improve the user experience.

[1023] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1024] Step 1:

[1025] Entering user information

[1026] The device displays a screen for the user to enter basic information such as age, gender, and areas of interest. The user then enters their information and presses the submit button.

[1027] Input: Information about the user's age, gender, and areas of interest.

[1028] Output: Input user information data.

[1029] Specific operation: The terminal converts user information data into JSON format and sends it to the server.

[1030] Step 2:

[1031] Request Input

[1032] The terminal provides an interface that allows the user to enter a specific request. The user enters the request and presses the submit button.

[1033] Input: The request entered by the user.

[1034] Output: The input request data.

[1035] Specific operation: The terminal converts the request data into JSON format and sends it to the server.

[1036] Step 3:

[1037] Recognition of emotions

[1038] The device uses its camera and microphone to capture the user's facial expressions and voice as they input requests. This data is then sent to an emotion engine for analysis.

[1039] Input: User's facial expression data and voice data.

[1040] Output: Emotion analysis results.

[1041] Specific operation: The device sends the captured data to the emotion engine in real time, and the emotion engine returns the emotion analysis result, such as "positive."

[1042] Step 4:

[1043] Sending data to the server

[1044] The device sends user information, requests, and sentiment analysis results to the server.

[1045] Inputs: User information data, request data, sentiment analysis results.

[1046] Output: Server received data (user information, requests, sentiment analysis results).

[1047] Specific operation: The terminal combines this data into one file and sends it to the server.

[1048] Step 5:

[1049] Processing of received data

[1050] The server stores the received user information, request details, and sentiment analysis results in its database. It then prepares to send the request to the generative artificial intelligence.

[1051] Input: Server received data (user information, requests, sentiment analysis results).

[1052] Output: Request data to be sent to the generative artificial intelligence.

[1053] Specific operation: The server analyzes and organizes the received data and generates a prompt message.

[1054] Step 6:

[1055] Sending a request to a generative artificial intelligence

[1056] The server sends a prompt message to the generative artificial intelligence.

[1057] Input: Request data (prompt text).

[1058] Output: Response from a generative artificial intelligence.

[1059] Specific operation: The server generates a prompt message like the following and sends it to the generative artificial intelligence:

[1060] Please create scenarios showing how a man in his 30s, who enjoys the outdoors and has a positive outlook, can use PayPay in his daily life.

[1061] Step 7:

[1062] Received a response from a generative artificial intelligence.

[1063] The server receives responses from the generative artificial intelligence. These responses include suggestions tailored to the user's daily life scenarios.

[1064] Input: Response data from a generative artificial intelligence.

[1065] Output: Proposed content.

[1066] Specific action: Save the received proposal content to the database.

[1067] Step 8:

[1068] Formatting and sending the results

[1069] The server formats the generated proposals into the appropriate data format and sends them to the user's terminal.

[1070] Input: Proposal content.

[1071] Output: Proposed data format for display.

[1072] Specific operation: Convert suggestions received from generative artificial intelligence (e.g., "Purchase camping equipment," "Pay admission fee for outdoor event," etc.) into HTML or JSON format and send them to the terminal.

[1073] Step 9:

[1074] Receiving user feedback

[1075] The device displays a form for the user to enter feedback on the proposal. The user enters their feedback and presses the submit button.

[1076] Input: User feedback.

[1077] Output: Feedback data.

[1078] Specific operation: The user enters "Which store is best for purchasing camping equipment?" and sends this to the server.

[1079] Step 10:

[1080] Processing and resubmitting feedback

[1081] Based on the feedback received from the user, the server sends a new request to the generative artificial intelligence.

[1082] Input: Feedback data.

[1083] Output: Response to the re-request.

[1084] Specific operation: The server uses feedback data to generate a new prompt and sends it to the generative artificial intelligence. Then, it receives a revised suggestion and sends it back to the user terminal.

[1085] In this way, the system can provide optimal service suggestions that take user emotions into consideration, thereby improving the user experience.

[1086] (Application Example 2)

[1087] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1088] Conventional real-time service suggestion systems lacked the ability to consider the user's emotional state. As a result, optimal suggestions were not made at the timing and with the nuances the user desired, leading to decreased service satisfaction. This invention aims to improve the user experience by recognizing the user's emotions and making suggestions based on those emotions.

[1089] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes a screen display means for inputting user information, a means for transmitting the input user information and requests to the server, a means for capturing the user's facial expressions and voice and transmitting them to the emotion engine, a means for transmitting the emotion analysis results analyzed by the emotion engine to the server, a means for sending a request to a generative artificial intelligence based on the received data, a means for receiving a response from the generative artificial intelligence, a means for transmitting the received response to the user terminal and displaying it, a means for receiving feedback from the user, and a means for sending a re-request to the generative artificial intelligence based on the feedback. This makes it possible to propose services optimized for the user's emotional state.

[1090] "User information input screen display means" refers to a device or program that provides an interface that allows a user to input their basic information (age, gender, areas of interest, etc.) and requests.

[1091] "Means for transmitting entered user information and requests to the server" refers to a device or program that transmits information and requests entered by a user to a server via the Internet.

[1092] "Means for capturing user facial expressions and voice and transmitting them to an emotion engine" refers to a device or program that acquires the user's face and voice using a camera and microphone and sends that data to an emotion analysis device.

[1093] "Means for sending emotion analysis results analyzed by the emotion engine to the server" refers to a device or program in which an emotion analysis device analyzes the user's emotional state and sends the analysis results to the server.

[1094] "Means for sending requests to generative artificial intelligence based on received data" refers to a device or program that sends appropriate requests to generative artificial intelligence based on user information, requests, and sentiment analysis results received by the server.

[1095] "Means for receiving responses from generative artificial intelligence" refers to a device or program that receives proposals and responses generated by generative artificial intelligence on a server.

[1096] "Means for sending and displaying received responses to a user terminal" refers to a device or program in which the server formats suggestions and responses obtained from a generative artificial intelligence into an appropriate format, and sends and displays them to the user's terminal.

[1097] "Means for receiving user feedback" refers to a device or program that collects opinions and requests from users regarding the proposed content.

[1098] "Means of sending a re-request to a generative artificial intelligence based on feedback" refers to a device or program that analyzes user feedback and sends a re-request to a generative artificial intelligence.

[1099] This invention is a system that recognizes user emotions and makes service suggestions that take them into consideration. The specific form of implementation is described below. The system uses a user terminal, a server, a generative artificial intelligence system, and an emotion engine.

[1100] User terminal

[1101] User terminals are devices such as smartphones and smart glasses. These terminals have the following main functions:

[1102] 1. Enter user information

[1103] The user terminal displays a screen for entering user information. Here, the user enters basic information such as their age, gender, and areas of interest, as well as specific requests.

[1104] 2. Emotion recognition

[1105] When a user enters information, the device's built-in camera and microphone are used to capture the user's facial expressions and voice. This data is then sent to the emotion engine for analysis.

[1106] server

[1107] The server performs the following roles:

[1108] 1. Data transmission

[1109] It receives user information, requests, and sentiment analysis results sent from the user's terminal, and converts them into a request format suitable for generative artificial intelligence.

[1110] 2. Requests to Generative Artificial Intelligence

[1111] A request is sent to a generative AI model (e.g., GPT-3) to generate specific suggestions.

[1112] 3. Receiving and formatting the response

[1113] The system formats the response received from the generative artificial intelligence into an appropriate data format and sends it to the user's terminal.

[1114] Generative artificial intelligence

[1115] Generative artificial intelligence receives requests from a server and generates optimal suggestions. This generation process takes into account the user's basic information, requests, and emotional state.

[1116] Emotional Engine

[1117] The emotion engine analyzes the user's facial expressions and voice to determine their emotional state. The results are then sent to the server.

[1118] Hardware and software used

[1119] User devices: smartphones, smart glasses

[1120] Server: Cloud server or on-premises server

[1121] Emotion Engine: Emotion Recognition API

[1122] Generative artificial intelligence: Generative AI models (e.g., GPT-3)

[1123] Specific example of processing

[1124] The following is an example of how to specifically suggest the most suitable product when a user is feeling relaxed.

[1125] 1. Enter user information:

[1126] Example: The user enters information such as "30s, female, enjoys outdoor activities" on their smartphone.

[1127] The user enters a request such as, "I want to relax, I want to find a book."

[1128] 2. Recognition of emotions:

[1129] The user's device captures the user's facial expressions, and the emotion engine analyzes that data to determine that the user is "relaxed."

[1130] 3. Example of a prompt:

[1131] "Generate product suggestions suitable for a female user in her 30s who enjoys the outdoors and wants to relax. The user's basic information is as follows: Age: 30s, Gender: Female, Interests: Outdoors, Emotion: Want to relax. Please generate five suggestions."

[1132] Sending this prompt to the generating AI model yields suggestions such as:

[1133] 1. Aromatherapy Candles

[1134] 2. Relaxation chair

[1135] 3. Relaxation music subscription

[1136] 4. Herb tea set

[1137] 5. Essential oils

[1138] This suggestion is displayed on the user's terminal, providing the user with optimal product knowledge. In this way, a system can be built that enables service suggestions that take into account the user's emotional state.

[1139] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1140] Step 1:

[1141] The user enters basic information and requests using a device. The user enters basic information such as age, gender, and areas of interest, as well as specific requests (e.g., "I want to relax, I want to find a book") through an input form displayed on the screen of a smartphone or smart glasses.

[1142] Input: User's basic information and requests

[1143] Output: The information entered by the user is temporarily stored on the device.

[1144] Step 2:

[1145] The device captures the user's facial expressions and voice. It uses the device's built-in camera and microphone to collect user facial expression and voice data.

[1146] Input: User's facial expressions and voice

[1147] Output: Captured facial expression data and audio data

[1148] Step 3:

[1149] The device sends the captured data to an emotion engine for analysis. The device then sends the data to emotion analysis software such as the Emotion Recognition API to obtain the emotion analysis results.

[1150] Input: Facial expression data and audio data

[1151] Output: Emotion analysis results (e.g., "Relaxed state")

[1152] Step 4:

[1153] The sentiment analysis results are sent from the terminal to the server. Along with the analysis results, the terminal also sends the user's basic information and requests to the server.

[1154] Input: Sentiment analysis results, basic information, request

[1155] Output: Data sent to the server

[1156] Step 5:

[1157] The server sends a request to the generative artificial intelligence based on the data it receives. The server analyzes the received data, converts it into a request format suitable for the generative AI model, and sends it. For example,

[1158] "Generate product suggestions suitable for a female user in her 30s who enjoys the outdoors and wants to relax. The user's basic information is as follows: Age: 30s, Gender: Female, Interests: Outdoors, Emotion: Want to relax. Please generate five suggestions."

[1159] Input: User information, requests, sentiment analysis results

[1160] Output: Request sent to the generative artificial intelligence

[1161] Step 6:

[1162] The server receives the response from the generative artificial intelligence. The server receives the suggestions sent by the generative AI model and performs analysis.

[1163] Input: Response from generative artificial intelligence

[1164] Output: Proposed ideas (e.g., aromatherapy candles, relaxation chairs, etc.)

[1165] Step 7:

[1166] The server sends the received response to the user terminal and displays it on the terminal. The server formats the proposed content into an appropriate format and sends it to the user terminal for display.

[1167] Input: Proposal Content

[1168] Output: Data sent to the user terminal

[1169] Step 8:

[1170] The device sends user feedback to the server. The user enters feedback on the displayed suggestions, and the device sends that information to the server.

[1171] Input: User feedback

[1172] Output: Feedback sent to the server

[1173] Step 9:

[1174] The server sends a new request to the generative artificial intelligence based on the feedback. The feedback is analyzed, and another request is sent to the generative AI model to generate new suggestions.

[1175] Input: Feedback Information

[1176] Output: Re-request sent to the generative AI

[1177] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1178] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1179] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1180] [Third Embodiment]

[1181] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1182] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1183] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1184] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1185] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1186] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1187] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1188] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1189] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1190] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1191] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1192] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1193] One embodiment of the present invention describes below a specific system for a user to incorporate a particular service into their life and make the most of it. This system provides the user with useful information through the input of user information, generation of suggestions by generative artificial intelligence, display of the suggested content, and processing of user feedback.

[1194] System-wide configuration

[1195] The system primarily consists of three parts: user terminals, servers, and generative artificial intelligence. These elements communicate and cooperate with each other via the internet to enable the overall system to function.

[1196] User terminal

[1197] 1. Enter basic information

[1198] The terminal first displays a screen for user information input. The user enters basic information such as their age, gender, and areas of interest. This information is later processed by generative artificial intelligence.

[1199] 2. Input of request

[1200] The terminal also provides an interface for receiving specific requests from the user. For example, the user can input a specific request such as, "Please tell me about situations in which PayPay can be used in daily life."

[1201] 3. Sending data to the server

[1202] The terminal sends user information and requests to the server. The transmitted data is used in the next processing step.

[1203] server

[1204] 4. Data processing and request generation

[1205] The server processes the received data, including user information and request details, and uses this information to generate appropriate requests for the generative artificial intelligence. For example, it might send a request to the generative AI such as, "Generate scenarios showing how a man in his 30s who enjoys outdoor activities can use PayPay in his daily life."

[1206] 5. Receiving a response from a generative artificial intelligence.

[1207] The server receives a response from the generative artificial intelligence. This response includes specific suggestions tailored to the user's life scenarios.

[1208] 6. Formatting and sending the results

[1209] The server formats the response from the generative artificial intelligence into an appropriate format and sends it to the user terminal.

[1210] Generative artificial intelligence

[1211] 7. Proposal generation

[1212] Generative artificial intelligence receives requests from a server and generates suggestions for specific life scenarios based on user information and requests. These suggestions show how the user can specifically utilize the service.

[1213] Specific example

[1214] 1. Enter user information

[1215] Example: The user enters information on their device such as "30s, male, enjoys outdoor activities."

[1216] 2. Input of request

[1217] Example: The user enters a request such as, "Tell me 10 situations in everyday life where PayPay can be used."

[1218] 3. Sending data to the server

[1219] The terminal sends user information and requests to the server.

[1220] 4. Data processing and request generation

[1221] Based on the data the server receives, it sends a request to the generative artificial intelligence saying, "Please generate scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[1222] 5. Receiving a response from a generative artificial intelligence.

[1223] The generative artificial intelligence generates the following suggestions and returns them to the server:

[1224] 1. Purchase camping equipment

[1225] 2. Payment of admission fees for outdoor events

[1226] 3. Paying for meals at restaurants

[1227] 4. Paying for fuel at a gas station

[1228] 5. Payment of public transport fares

[1229] 6. Payment of sports facility usage fees

[1230] 7. Outdoor-related subscription services

[1231] 8. Payment of travel expenses at the travel agency

[1232] 9. Booking and payment for accommodation

[1233] 10. Booking and Payment for Activities

[1234] 6. Formatting and sending the results

[1235] The server formats the generated proposal into the appropriate data format and sends it to the user's terminal.

[1236] 7. Displaying results and receiving feedback

[1237] The device displays the results to the user, who then provides feedback on the suggestions. For example, the user might input, "Which store is best for buying camping equipment?"

[1238] 8. Processing and resubmitting feedback

[1239] The server receives user feedback and sends a request to the generative artificial intelligence. The generative AI generates specific suggestions again, which the server receives, formats appropriately, and sends back to the user's terminal.

[1240] In this way, the system incorporates user feedback and has the function of continuously and concretely suggesting how users can utilize the services provided in their actual daily lives.

[1241] The following describes the processing flow.

[1242] Step 1:

[1243] The device displays a screen that prompts the user to enter basic information (age, gender, areas of interest, etc.).

[1244] Step 2:

[1245] The user enters information such as their age, gender, and areas of interest into the displayed input form.

[1246] Step 3:

[1247] The device sends basic information entered by the user to the server. This data includes the user's age, gender, and areas of interest.

[1248] Step 4:

[1249] The terminal displays an interface that prompts the user to input requests regarding specific service usage scenarios (e.g., "Please tell me about scenarios in which PayPay can be used in daily life").

[1250] Step 5:

[1251] The user enters their request into the terminal.

[1252] Step 6:

[1253] The terminal sends the user's request to the server. The transmitted data includes the details of the user's request.

[1254] Step 7:

[1255] The server converts the received user information and request into a request format appropriate for the generative artificial intelligence.

[1256] Step 8:

[1257] The server sends a request to the generative artificial intelligence. The request includes the user's age, gender, areas of interest, and the content of the request.

[1258] Step 9:

[1259] Generative artificial intelligence processes received requests and generates specific suggestions that correspond to the user's life scenarios.

[1260] Step 10:

[1261] The generative artificial intelligence sends the generated suggestions back to the server as a response.

[1262] Step 11:

[1263] The server receives the response from the generative artificial intelligence and formats it into an appropriate data format (e.g., JSON).

[1264] Step 12:

[1265] The server sends the formatted proposal to the terminal.

[1266] Step 13:

[1267] The device displays received suggestions to the user. These suggestions include specific examples of service usage tailored to the user's lifestyle.

[1268] Step 14:

[1269] Users provide feedback on the displayed suggestions (e.g., "Which store is best for purchasing camping equipment?").

[1270] Step 15:

[1271] The device sends user feedback to the server.

[1272] Step 16:

[1273] The server sends a new request to the generative artificial intelligence based on the user's feedback. This request includes additional information about the user.

[1274] Step 17:

[1275] The generative artificial intelligence receives a follow-up request and generates additional specific suggestions.

[1276] Step 18:

[1277] The generative artificial intelligence sends the generated additional suggestions back to the server as a response.

[1278] Step 19:

[1279] The server receives additional responses from the generative artificial intelligence, formats them into the appropriate data format, and sends them to the terminal.

[1280] Step 20:

[1281] The device displays the received additional suggestions to the user.

[1282] This series of steps allows the system to continuously provide specific suggestions that align with the user's requirements.

[1283] (Example 1)

[1284] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1285] Conventional systems struggled to generate personalized suggestions based on individual user requests, failing to improve user satisfaction. Furthermore, they lacked sufficient mechanisms for efficiently incorporating user feedback and updating suggestions.

[1286] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1287] In this invention, the server includes means for processing user information and requests, generating and sending prompt messages to a generative artificial intelligence; means for receiving responses from the generative artificial intelligence; and means for formatting the received responses into an appropriate data format and sending them to the user terminal. This makes it possible to generate personalized suggestions based on the user's individual requests and to efficiently incorporate feedback to update the suggestions.

[1288] "Means for inputting basic user information" refers to providing an interface for inputting basic user information such as age, gender, and areas of interest.

[1289] "Means for inputting user requests" refers to means of providing an interface for users to input specific requests (for example, questions about how to use a particular service).

[1290] "Means for sending entered user information and requests to the server" refers to means for sending the basic information and requests entered by the user to the server via the network.

[1291] "Means for processing user information and requests, and generating and sending prompt messages to a generative artificial intelligence" refers to means for creating and sending appropriate prompt messages to a generative artificial intelligence based on the received user information and requests.

[1292] "Means for receiving responses from generative artificial intelligence" refers to means for the server to receive proposals and responses generated by generative artificial intelligence.

[1293] "Means for formatting the received response into an appropriate data format and sending it to the user terminal" refers to means of converting the response received from the generative artificial intelligence into a format that is easy for the user to understand and sending it again to the user terminal via the network.

[1294] "Means for displaying responses to the user and receiving feedback from the user" refers to means that the user terminal provides an interface for displaying responses from generative artificial intelligence to the user and for receiving feedback from the user.

[1295] "Method for sending a new request to a generative artificial intelligence based on feedback" refers to a method for generating a new prompt message based on feedback received from the user and sending it to the generative artificial intelligence.

[1296] Overall system configuration

[1297] This invention is a system that enables users to incorporate specific services into their lives and make the most of them. This system provides users with useful information through the input of user information, generation of suggestions by generative artificial intelligence, display of the suggested content, and processing of user feedback.

[1298] Hardware and software to be used

[1299] The system primarily consists of three parts: user terminals, servers, and generative artificial intelligence. These elements communicate with each other via the internet and work together as a whole system. User terminals include smartphones, tablets, and personal computers. Servers include web servers and cloud servers. Generative artificial intelligence utilizes computing resources (e.g., cloud-based AI services) capable of executing appropriate AI models.

[1300] Program processing

[1301] The specific processing details of the system are as follows:

[1302] Entering user information

[1303] The user terminal displays a screen prompting the user to enter basic information such as age, gender, and areas of interest. The user enters information such as "30s, male, enjoys outdoor activities." This information is encoded in JSON format or similar and sent to the server.

[1304] Request Input

[1305] The user terminal provides an interface for the user to input specific requests. The user might input a specific request such as, "Please tell me about situations in daily life where PayPay can be used."

[1306] Sending data to the server

[1307] The user terminal sends the entered user information and requests to the server. The server receives this information and parses it.

[1308] Data processing and request generation

[1309] The server generates and sends a prompt message to the generative artificial intelligence based on user information and the request. For example, a request such as "Generate scenarios showing how a man in his 30s who enjoys outdoor activities can use PayPay in his daily life" might be generated.

[1310] Received a response from a generative artificial intelligence.

[1311] Generative artificial intelligence receives requests and generates suggestions for specific life scenarios based on user information. For example, it might generate suggestions such as "purchase camping equipment" or "pay admission fees for outdoor events." These responses are then returned to the server.

[1312] Formatting and sending the results

[1313] The server formats the response received from the generative artificial intelligence into an appropriate data format and sends it back to the user terminal. In this process, the data is formatted into a list and then encoded again in JSON format.

[1314] Displaying results and accepting feedback

[1315] The user terminal displays the received suggestions to the user. The user then reviews these suggestions and provides more specific feedback. For example, they might input feedback such as, "Which store is best for purchasing camping equipment?"

[1316] Processing and resubmitting feedback

[1317] Based on the feedback received from the user, the server generates a new prompt and sends it to the generative artificial intelligence. The generative AI then generates specific suggestions, which the server receives, formats appropriately, and sends back to the user's terminal.

[1318] Specific example

[1319] 1. Enter user information

[1320] The user enters information on their device such as "30s, male, enjoys outdoor activities."

[1321] 2. Input of request

[1322] The user enters a request asking for "10 ways to use PayPay in everyday life."

[1323] 3. Sending data to the server

[1324] The user terminal sends the entered data to the server.

[1325] 4. Data processing and request generation

[1326] The server parses the data and generates and sends a prompt message: "Generate scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[1327] 5. Receiving a response from a generative artificial intelligence.

[1328] Generative artificial intelligence generates suggestions and returns them to the server. Examples: "Purchase camping equipment," "Pay admission fee for outdoor event," etc.

[1329] 6. Formatting and sending the results

[1330] The server formats the proposal and sends it to the user's terminal.

[1331] 7. Displaying results and receiving feedback

[1332] The user's device displays suggestions to the user, who then provides more specific feedback.

[1333] 8. Processing and resubmitting feedback

[1334] The server processes the feedback and sends a new prompt to the generative artificial intelligence again, which generates a new suggestion, formats it again, and sends it to the user terminal.

[1335] The above describes a specific embodiment of the present invention, a system that efficiently provides personalized suggestions based on the individual needs of users and continuously improves while incorporating feedback.

[1336] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1337] Step 1: Enter user information

[1338] The user terminal displays a screen prompting the user to enter basic information such as age, gender, and areas of interest. The user enters information such as "30s, male, enjoys outdoor activities" into the input screen. The entered information is temporarily stored on the terminal. Input data formats such as JSON and XML are used.

[1339] Input: User's basic information (age, gender, areas of interest, etc.)

[1340] Output: Encoded user basic information data

[1341] Step 2: Enter the request

[1342] The user terminal provides an interface for entering specific requests. Users enter specific requests such as, "Please tell me about situations where PayPay can be used in daily life." The entered requests are then encoded into a data format.

[1343] Input: User request (e.g., "Please tell me about situations in daily life where PayPay can be used.")

[1344] Output: Encoded user request data

[1345] Step 3: Sending data to the server

[1346] The user terminal sends the entered user information and requests to the server. The HTTPS protocol is used for transmission, and the data is encoded in JSON format. Upon successful communication, the data is stored on the server.

[1347] Input: Encoded user basic information data and user request data

[1348] Output: User information and request data stored on the server

[1349] Step 4: Data processing and request generation

[1350] The server parses and analyzes the received user information and requests. For example, it extracts information such as age, gender, and areas of interest. Next, it generates a prompt message to send to the generative artificial intelligence based on that information. The prompt message is in a format such as, "Generate 10 scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[1351] Input: User information and request data stored on the server

[1352] Output: Generated prompt message

[1353] Step 5: Receiving a response from the generative artificial intelligence.

[1354] The server sends the generated prompt to the generative artificial intelligence (AI). The AI ​​receives the request, analyzes it based on the prompt, and generates specific suggestions to address the user's request. These suggestions are then returned to the server.

[1355] Input: Generated prompt message

[1356] Output: Response data from generative artificial intelligence (e.g., suggestions such as "purchase camping equipment" or "pay admission fee for an outdoor event")

[1357] Step 6: Format and submit the results

[1358] The server formats the response received from the generative artificial intelligence into an appropriate data format. The formatted data is then encoded again in JSON format and sent to the user's terminal.

[1359] Input: Response data from generative artificial intelligence

[1360] Output: Formatted proposal data

[1361] Step 7: Displaying results and receiving feedback

[1362] The user terminal displays a list of received suggestions to the user. The user reviews the suggestions and enters more specific feedback. For example, they might enter feedback such as, "Which store is best for purchasing camping equipment?" and click the submit button.

[1363] Input: Formatted proposal data

[1364] Output: User feedback (e.g., specific questions or requests)

[1365] Step 8: Processing feedback and resubmitting requests

[1366] The server parses and analyzes the feedback received from the user. Based on the feedback, it generates a new prompt and sends it to the generative artificial intelligence. The generative AI generates another suggestion, which the server receives, formats appropriately, and sends back to the user's terminal.

[1367] Input: User feedback data

[1368] Output: Newly generated prompt sentence and new response data from the generative AI.

[1369] Through the above processing steps, the system can efficiently provide personalized suggestions based on the user's individual requests and continuously improve by incorporating feedback.

[1370] (Application Example 1)

[1371] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1372] This invention relates to a system that generates suggestions tailored to the user's lifestyle and maximizes user convenience. In particular, to improve the shopping experience in physical stores, there is a need to provide optimized suggestions in real time using the user's current location information. However, conventional systems have the problem of not being able to improve user satisfaction because they do not provide insufficient suggestions based on user information and do not perform real-time optimization using the current location information.

[1373] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1374] In this invention, the server includes means for having a display device for inputting user information, means for transmitting the input user information and requests to a communication device, means for receiving responses from generative artificial intelligence, means for acquiring the user's current location information and optimizing response suggestions based on the location information, and means for displaying the generated suggestions in real time on a smartphone or smart glasses. This enables the user to fully enjoy the shopping experience in a physical store.

[1375] "User information" refers to basic data such as the user's age, gender, areas of interest, and current location.

[1376] A "display device" is an interface for users to input and confirm information, and includes devices such as smartphones and smart glasses.

[1377] A "communication device" is a device used to send user information and requests to a server.

[1378] "Generative artificial intelligence" refers to an artificial intelligence system that generates optimal suggestions based on user information and requests.

[1379] "Current location information" refers to the user's real-time geographical location data.

[1380] "Real-time" refers to a timeframe that responds immediately to the user's current actions and circumstances.

[1381] A "smartphone" is a type of mobile phone that is a device with multi-functional computer capabilities.

[1382] "Smart glasses" are wearable devices designed to enhance visual information.

[1383] The following describes specific embodiments for carrying out the present invention. The invention relates to a system for users to incorporate specific services into their lives and make the most of them. The specific configuration and operation will be described below.

[1384] System-wide configuration

[1385] The system primarily consists of the following elements: user terminals, servers, and generative artificial intelligence. These elements communicate with each other via the internet and cooperate to function as a whole system.

[1386] User terminal

[1387] A user terminal is equipped with a display device and provides an interface that allows the user to input information. Examples include smartphones and smart glasses.

[1388] 1. Enter basic information

[1389] The user enters basic information such as age, gender, and areas of interest on the display device of their user terminal.

[1390] 2. Input of request

[1391] Users utilize an interface to enter specific requests. For example, they might enter a request such as, "Please recommend some outdoor products."

[1392] 3. Sending data to the server

[1393] The user terminal transmits the entered user information and requests to the server via a communication device.

[1394] server

[1395] The server processes the data sent from the user terminal and generates appropriate requests for the generative artificial intelligence.

[1396] 4. Data processing and request generation

[1397] The server organizes the received user information and request details, and then sends the request to the generative artificial intelligence based on that information.

[1398] For example, generate a prompt such as, "Generate suggestions on what products a man in his 30s who enjoys outdoor activities should purchase at a physical store."

[1399] 5. Receiving a response from a generative artificial intelligence.

[1400] The server receives the response from the generative artificial intelligence and formats it into an appropriate format.

[1401] 6. Optimization and Display

[1402] The server obtains the user's current location information and optimizes response suggestions based on that location. The generated suggestions are then displayed in real time on the user's smartphone or smart glasses.

[1403] Generative artificial intelligence

[1404] Generative artificial intelligence receives requests from a server and generates specific lifestyle scenarios and product suggestions based on user information and requests.

[1405] Specific example

[1406] 1. Enter user information

[1407] Example: A user enters information such as "30s, male, enjoys outdoor activities" on their smartphone.

[1408] 2. Input of request

[1409] Example: A user voice-inputs, "Please recommend some outdoor products."

[1410] 3. Sending data to the server

[1411] The user terminal sends user information and requests to the server.

[1412] 4. Data processing and request generation

[1413] The server sends the prompt message, "A man in his 30s who enjoys the outdoors is seeking advice on recommended outdoor products available at physical stores," to the generative AI.

[1414] 5. Receiving a response from a generative artificial intelligence.

[1415] The generative artificial intelligence generates a response and returns it to the server. This response may include suggestions such as, "A new camping tent has recently arrived at a nearby sporting goods store."

[1416] 6. Optimization and Display

[1417] The server optimizes the suggested content based on the user's current location, sends it to the user's terminal, and displays it in real time.

[1418] In this way, the system incorporates user feedback and has the function of continuously and concretely suggesting how users can utilize the services provided in their actual daily lives.

[1419] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1420] Step 1:

[1421] The user enters basic information such as age, gender, and areas of interest using the display device on their terminal. For example, they might enter information such as "30s, male, enjoys outdoor activities." This input data is collected on the terminal.

[1422] Step 2:

[1423] The user terminal provides an interface for inputting specific requests from the user. Users input requests such as "Please recommend some outdoor products" via voice or text. This request data is also collected by the terminal.

[1424] Step 3:

[1425] The user terminal transmits the entered user information and requests to the server via a communication device. At this time, the input data and request data are combined into a single request packet and sent to the server.

[1426] Step 4:

[1427] The server processes the received user information and request details to generate an appropriate request for the generative artificial intelligence. For example, it might generate a prompt such as, "Generate suggestions on what products a man in his 30s who enjoys outdoor activities should purchase at a physical store." This prompt becomes the input data for the generative artificial intelligence.

[1428] Step 5:

[1429] Generative artificial intelligence receives prompt messages from a server and generates specific lifestyle scenarios and product suggestions based on user information and requests. For example, it might generate a suggestion such as, "A new camping tent has recently arrived at a nearby sports goods store." This suggestion is then sent back to the server.

[1430] Step 6:

[1431] The server receives a response from the generative artificial intelligence and obtains the user's current location information. Based on the obtained location information, it further optimizes the response. For example, it processes the data to provide information about the nearest store based on the user's location. This optimized suggestion is then sent to the user's terminal.

[1432] Step 7:

[1433] The user's device displays received suggestions in real time on their smartphone or smart glasses. This allows users to instantly see the best suggestions to enhance their in-store shopping experience.

[1434] Step 8:

[1435] Users provide feedback on the displayed suggestions. For example, they might enter feedback such as, "This information was helpful. I'd like to know more details." This user feedback is collected on the device.

[1436] Step 9:

[1437] The user terminal sends feedback from the user to the server. The feedback data arrives at the server.

[1438] Step 10:

[1439] Based on the feedback received, the server sends another request to the generative artificial intelligence. For example, it might generate a prompt such as, "A man in his 30s who enjoys the outdoors is looking for more detailed information on outdoor products," and send it to the generative AI.

[1440] Step 11:

[1441] The generative artificial intelligence generates a response again and sends it to the server. This process is repeated in accordance with user feedback, allowing the system to continuously provide the most relevant information to the user.

[1442] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1443] As one embodiment of the present invention, a specific embodiment of a system that recognizes a user's emotions and makes more appropriate service suggestions based on those emotions is described below. This system provides the user with optimal information through the input of user information, suggestion generation by generative artificial intelligence, emotion recognition by an emotion engine, display of the suggested content, and processing of user feedback.

[1444] System-wide configuration

[1445] The system primarily consists of the following four parts: user terminal, server, generative artificial intelligence, and emotion engine. These elements communicate with each other via the internet and cooperate to enable the overall system to function.

[1446] User terminal

[1447] 1. Enter basic information

[1448] The device first displays a screen for user information input. The user enters basic information such as their age, gender, and areas of interest. This information is later processed by generative artificial intelligence and an emotion engine.

[1449] 2. Input of request

[1450] The terminal provides an interface for users to input specific requests (e.g., "Please tell me about situations in which PayPay can be used in daily life").

[1451] 3. Recognition of emotions

[1452] The device captures the user's facial expressions and voice as they enter their requests and sends them to the emotion engine.

[1453] 4. Sending data to the server

[1454] The device sends user information, requests, and sentiment analysis results from the sentiment engine to the server. The transmitted data includes the user's age, gender, areas of interest, request content, and sentiment analysis results.

[1455] server

[1456] 5. Data processing and request generation

[1457] The server receives data, including user information, request details, and sentiment analysis results, and then converts it into an appropriate request format for the generative artificial intelligence. For example, it might send a request to the generative AI such as, "Generate scenarios showing how a man in his 30s who enjoys the outdoors and has positive emotions can use PayPay in his daily life."

[1458] 6. Receiving a response from a generative artificial intelligence.

[1459] The server receives a response from the generative artificial intelligence. This response includes specific suggestions tailored to the user's life scenarios and is customized based on the sentiment analysis results.

[1460] 7. Formatting and sending the results

[1461] The server formats the response from the generative artificial intelligence into an appropriate format and sends it to the user terminal.

[1462] Generative artificial intelligence

[1463] 8. Proposal generation

[1464] Generative artificial intelligence receives requests from the server and generates specific life scenario suggestions based on user information, requests, and sentiment analysis results. These suggestions show how the user can specifically use the service and also take into account the user's emotional state.

[1465] Specific example

[1466] 1. Enter user information

[1467] Example: The user enters information on their device such as "30s, male, enjoys outdoor activities."

[1468] 2. Input of request

[1469] Example: The user enters a request such as, "Tell me 10 situations in everyday life where PayPay can be used."

[1470] 3. Recognition of emotions

[1471] The device captures the user's facial expressions with its camera, and an emotion engine analyzes them to determine if they are "positive."

[1472] 4. Sending data to the server

[1473] The device sends user information, requests, and sentiment analysis results to the server.

[1474] 5. Data processing and request generation

[1475] Based on the data the server receives, it sends a request to the generative artificial intelligence saying, "Please generate scenarios showing how a man in his 30s, who enjoys the outdoors and has positive emotions, can use PayPay in his daily life."

[1476] 6. Receiving a response from a generative artificial intelligence.

[1477] The generative artificial intelligence generates the following suggestions and returns them to the server:

[1478] 1. Purchase camping equipment

[1479] 2. Payment of admission fees for outdoor events

[1480] 3. Paying for meals at restaurants

[1481] 4. Paying for fuel at a gas station

[1482] 5. Payment of public transport fares

[1483] 6. Payment of sports facility usage fees

[1484] 7. Outdoor-related subscription services

[1485] 8. Payment of travel expenses at the travel agency

[1486] 9. Booking and payment for accommodation

[1487] 10. Booking and Payment for Activities

[1488] 7. Formatting and sending the results

[1489] The server formats the generated proposal into the appropriate data format and sends it to the user's terminal.

[1490] 8. Displaying results and receiving feedback

[1491] The device displays the results to the user, who then provides feedback on the suggestions. For example, the user might input, "Which store is best for purchasing camping equipment?"

[1492] 9. Processing and resubmitting feedback

[1493] The server receives user feedback and sends a request to the generative artificial intelligence. The generative AI generates specific suggestions again, which the server receives, formats appropriately, and sends back to the user's terminal.

[1494] Through these steps, the system takes user emotions into consideration and proposes services in a more appropriate way. This allows users to more concretely integrate the services offered into their own lives.

[1495] The following describes the processing flow.

[1496] Step 1:

[1497] The device displays a screen that prompts the user to enter basic information (age, gender, areas of interest, etc.).

[1498] Step 2:

[1499] The user enters information such as their age, gender, and areas of interest into the displayed input form.

[1500] Step 3:

[1501] The device sends basic information entered by the user to the server. This data includes the user's age, gender, and areas of interest.

[1502] Step 4:

[1503] The terminal displays an interface that prompts the user to input requests regarding specific service usage scenarios (e.g., "Please tell me about scenarios in which PayPay can be used in daily life").

[1504] Step 5:

[1505] The user enters their request into the terminal.

[1506] Step 6:

[1507] The terminal sends the user's request to the server. The transmitted data includes the details of the user's request.

[1508] Step 7:

[1509] The device captures the user's facial expressions and voice as they enter their requests and sends this information to the emotion engine.

[1510] Step 8:

[1511] The emotion engine analyzes captured facial and audio data to recognize the user's emotional state (e.g., positive, negative, neutral).

[1512] Step 9:

[1513] The emotion engine sends the analysis results to the server. The transmitted data includes the user's emotional state.

[1514] Step 10:

[1515] The server converts the received user information, request details, and sentiment analysis results into a request format appropriate for the generative artificial intelligence.

[1516] Step 11:

[1517] The server sends a request to the generative artificial intelligence. The request includes the user's age, gender, areas of interest, request content, and sentiment analysis results.

[1518] Step 12:

[1519] Generative artificial intelligence processes received requests and generates specific suggestions tailored to the user's life scenarios. These suggestions also reflect the user's emotional state.

[1520] Step 13:

[1521] The generative artificial intelligence sends the generated suggestions back to the server as a response.

[1522] Step 14:

[1523] The server receives the response from the generative artificial intelligence and formats it into an appropriate data format (e.g., JSON).

[1524] Step 15:

[1525] The server sends the formatted proposal to the terminal.

[1526] Step 16:

[1527] The device displays received suggestions to the user. These suggestions include specific examples of service usage tailored to the user's lifestyle.

[1528] Step 17:

[1529] Users provide feedback on the displayed suggestions (e.g., "Which store is best for purchasing camping equipment?").

[1530] Step 18:

[1531] The device sends user feedback to the server.

[1532] Step 19:

[1533] The server sends a new request to the generative artificial intelligence based on the user's feedback. This request includes additional information about the user.

[1534] Step 20:

[1535] The generative artificial intelligence receives a follow-up request and generates additional specific suggestions.

[1536] Step 21:

[1537] The generative artificial intelligence sends the generated additional suggestions back to the server as a response.

[1538] Step 22:

[1539] The server receives additional responses from the generative artificial intelligence, formats them into the appropriate data format, and sends them to the terminal.

[1540] Step 23:

[1541] The device displays the received additional suggestions to the user.

[1542] Through this series of steps, the system takes into account not only the user's requests but also their emotional state, enabling it to specifically suggest the optimal service usage scenario.

[1543] (Example 2)

[1544] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1545] Traditional systems often fail to consider user emotions when proposing services, resulting in the provision of information and service suggestions that are not optimal for user needs. This can lead to a diminished user experience and a decrease in the frequency of service usage. Furthermore, the lack of sufficient functionality to quickly revise suggestions based on feedback makes it difficult to promptly provide users with the specific information they require.

[1546] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting user information, means for transmitting the inputted user information and requests, means for capturing the facial expressions and voice of the user when they input a request and performing emotion analysis, means for analyzing the captured facial expressions and voice and transmitting the emotion analysis results, means for transmitting a request to a generative artificial intelligence based on the received user information, requests, and emotion analysis results, means for receiving a response from the generative artificial intelligence, means for formatting the received response into an appropriate data format and transmitting it to the user terminal for display, means for receiving feedback from the user, and means for transmitting a re-request to the generative artificial intelligence based on the feedback. This enables the realization of optimal service proposals that take into account the user's emotions and enables rapid re-proposals based on feedback.

[1547] "User information" refers to basic data such as the user's age, gender, and areas of interest.

[1548] A "request" refers to the content of a user's request to the system, such as their wishes, questions, or specific service usage scenarios.

[1549] A "terminal" is a device that functions as the user interface of a system, and is a device that performs tasks such as inputting user information, sending requests, and displaying results.

[1550] A "server" is a central processing unit that receives user information, requests, and sentiment analysis results, sends requests to generative artificial intelligence, and receives, processes, and transmits responses from generative artificial intelligence.

[1551] An "emotion engine" is software or a device that analyzes a user's emotional state by performing emotional analysis based on the user's facial expressions and voice data.

[1552] "Generative artificial intelligence" is an artificial intelligence system that generates optimal service suggestions based on received user information, requests, and sentiment analysis results.

[1553] "Feedback" refers to responses from users to proposed services or information, such as evaluations, additional requests, or questions.

[1554] "Formatting" is the process of converting responses and suggestions received from generative artificial intelligence into a format that is easy for the user to understand.

[1555] A "request" is an inquiry, such as instructions or questions, that a server sends to a generative artificial intelligence system based on user requests.

[1556] "Data format" refers to the format that represents the structure and layout of data, and specifically refers to formats such as HTML and JSON.

[1557] This invention is a system that analyzes a user's emotions and provides appropriate service suggestions based on the results. The system mainly consists of four elements: a user terminal, a server, a generative artificial intelligence system, and an emotion engine. These elements communicate with each other via the internet and operate in an integrated manner.

[1558] User terminal

[1559] The user terminal handles user information input, request input, emotion recognition, data transmission, and result display. The terminal performs the following specific actions:

[1560] 1. Enter user information

[1561] The device screen displays a screen where the user can enter basic information such as age, gender, and areas of interest.

[1562] Example: The user enters "30s, male, enjoys outdoor activities."

[1563] 2. Input of request

[1564] The user enters a specific request via their device. For example, they might type, "Please tell me how I can use PayPay in my daily life."

[1565] 3. Recognition of emotions

[1566] The device uses its camera and microphone to capture the user's facial expressions and voice, and sends them to the emotion engine. This emotion engine then uses specialized software to perform emotion analysis.

[1567] Specific example: The emotion engine analyzes the user's facial expression and determines it to be "positive."

[1568] 4. Sending data to the server

[1569] The terminal sends user information, requests, and sentiment analysis results to the server.

[1570] server

[1571] The server receives data sent from the user terminal, sends a request to the generative artificial intelligence, and receives a response. The specific operation of the server is as follows:

[1572] 5. Processing of received data

[1573] The server stores user information, request details, and sentiment analysis results in a database.

[1574] 6. Sending requests to generative artificial intelligence

[1575] The server generates and sends a prompt message to the generative artificial intelligence based on the received data.

[1576] Example of a prompt:

[1577] Please create scenarios showing how a man in his 30s, who enjoys the outdoors and has a positive outlook, can use PayPay in his daily life.

[1578] 7. Receiving a response from a generative artificial intelligence.

[1579] Generative artificial intelligence generates suggestions tailored to the user's daily life and sends them back to the server.

[1580] Specific example: The following suggestions may be generated.

[1581] 1. Purchase camping equipment

[1582] 2. Payment of admission fees for outdoor events

[1583] 3. Paying for meals at restaurants

[1584] 4. Paying for fuel at a gas station

[1585] 5. Payment of public transport fares

[1586] 6. Payment of sports facility usage fees

[1587] 7. Outdoor-related subscription services

[1588] 8. Payment of travel expenses at the travel agency

[1589] 9. Booking and payment for accommodation

[1590] 10. Booking and Payment for Activities

[1591] 8. Formatting and sending the results

[1592] The server formats the generated proposals into the appropriate data format and sends them to the user's terminal.

[1593] Feedback processing

[1594] Users can provide feedback on proposals and then make revised proposals based on that feedback. The feedback process is as follows:

[1595] 9. Receiving user feedback

[1596] Users provide feedback on the suggestions via their devices.

[1597] Specific example: A user provides feedback asking, "Which store is best for purchasing camping equipment?"

[1598] 10. Processing and resubmitting feedback

[1599] The server receives user feedback and, based on that, sends a new request to the generative artificial intelligence. This new request generates a new suggestion, which is then sent back to the user's terminal.

[1600] In this way, the present invention makes it possible to propose optimal services that take user emotions into consideration and improve the user experience.

[1601] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1602] Step 1:

[1603] Entering user information

[1604] The device displays a screen for the user to enter basic information such as age, gender, and areas of interest. The user then enters their information and presses the submit button.

[1605] Input: Information about the user's age, gender, and areas of interest.

[1606] Output: Input user information data.

[1607] Specific operation: The terminal converts user information data into JSON format and sends it to the server.

[1608] Step 2:

[1609] Request Input

[1610] The terminal provides an interface that allows the user to enter a specific request. The user enters the request and presses the submit button.

[1611] Input: The request entered by the user.

[1612] Output: The input request data.

[1613] Specific operation: The terminal converts the request data into JSON format and sends it to the server.

[1614] Step 3:

[1615] Recognition of emotions

[1616] The device uses its camera and microphone to capture the user's facial expressions and voice as they input requests. This data is then sent to an emotion engine for analysis.

[1617] Input: User's facial expression data and voice data.

[1618] Output: Emotion analysis results.

[1619] Specific operation: The device sends the captured data to the emotion engine in real time, and the emotion engine returns the emotion analysis result, such as "positive."

[1620] Step 4:

[1621] Sending data to the server

[1622] The device sends user information, requests, and sentiment analysis results to the server.

[1623] Inputs: User information data, request data, sentiment analysis results.

[1624] Output: Server received data (user information, requests, sentiment analysis results).

[1625] Specific operation: The terminal combines this data into one file and sends it to the server.

[1626] Step 5:

[1627] Processing of received data

[1628] The server stores the received user information, request details, and sentiment analysis results in its database. It then prepares to send the request to the generative artificial intelligence.

[1629] Input: Server received data (user information, requests, sentiment analysis results).

[1630] Output: Request data to be sent to the generative artificial intelligence.

[1631] Specific operation: The server analyzes and organizes the received data and generates a prompt message.

[1632] Step 6:

[1633] Sending a request to a generative artificial intelligence

[1634] The server sends a prompt message to the generative artificial intelligence.

[1635] Input: Request data (prompt text).

[1636] Output: Response from a generative artificial intelligence.

[1637] Specific operation: The server generates a prompt message like the following and sends it to the generative artificial intelligence:

[1638] Please create scenarios showing how a man in his 30s, who enjoys the outdoors and has a positive outlook, can use PayPay in his daily life.

[1639] Step 7:

[1640] Received a response from a generative artificial intelligence.

[1641] The server receives responses from the generative artificial intelligence. These responses include suggestions tailored to the user's daily life scenarios.

[1642] Input: Response data from a generative artificial intelligence.

[1643] Output: Proposed content.

[1644] Specific action: Save the received proposal content to the database.

[1645] Step 8:

[1646] Formatting and sending the results

[1647] The server formats the generated proposals into the appropriate data format and sends them to the user's terminal.

[1648] Input: Proposal content.

[1649] Output: Proposed data format for display.

[1650] Specific operation: Convert suggestions received from generative artificial intelligence (e.g., "Purchase camping equipment," "Pay admission fee for outdoor event," etc.) into HTML or JSON format and send them to the terminal.

[1651] Step 9:

[1652] Receiving user feedback

[1653] The device displays a form for the user to enter feedback on the proposal. The user enters their feedback and presses the submit button.

[1654] Input: User feedback.

[1655] Output: Feedback data.

[1656] Specific operation: The user enters "Which store is best for purchasing camping equipment?" and sends this to the server.

[1657] Step 10:

[1658] Processing and resubmitting feedback

[1659] Based on the feedback received from the user, the server sends a new request to the generative artificial intelligence.

[1660] Input: Feedback data.

[1661] Output: Response to the re-request.

[1662] Specific operation: The server uses feedback data to generate a new prompt and sends it to the generative artificial intelligence. Then, it receives a revised suggestion and sends it back to the user terminal.

[1663] In this way, the system can provide optimal service suggestions that take user emotions into consideration, thereby improving the user experience.

[1664] (Application Example 2)

[1665] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1666] Conventional real-time service suggestion systems lacked the ability to consider the user's emotional state. As a result, optimal suggestions were not made at the timing and with the nuances the user desired, leading to decreased service satisfaction. This invention aims to improve the user experience by recognizing the user's emotions and making suggestions based on those emotions.

[1667] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes a screen display means for inputting user information, a means for transmitting the input user information and requests to the server, a means for capturing the user's facial expressions and voice and transmitting them to the emotion engine, a means for transmitting the emotion analysis results analyzed by the emotion engine to the server, a means for sending a request to a generative artificial intelligence based on the received data, a means for receiving a response from the generative artificial intelligence, a means for transmitting the received response to the user terminal and displaying it, a means for receiving feedback from the user, and a means for sending a re-request to the generative artificial intelligence based on the feedback. This makes it possible to propose services optimized for the user's emotional state.

[1668] "User information input screen display means" refers to a device or program that provides an interface that allows a user to input their basic information (age, gender, areas of interest, etc.) and requests.

[1669] "Means for transmitting entered user information and requests to the server" refers to a device or program that transmits information and requests entered by a user to a server via the Internet.

[1670] "Means for capturing user facial expressions and voice and transmitting them to an emotion engine" refers to a device or program that acquires the user's face and voice using a camera and microphone and sends that data to an emotion analysis device.

[1671] "Means for sending emotion analysis results analyzed by the emotion engine to the server" refers to a device or program in which an emotion analysis device analyzes the user's emotional state and sends the analysis results to the server.

[1672] "Means for sending requests to generative artificial intelligence based on received data" refers to a device or program that sends appropriate requests to generative artificial intelligence based on user information, requests, and sentiment analysis results received by the server.

[1673] "Means for receiving responses from generative artificial intelligence" refers to a device or program that receives proposals and responses generated by generative artificial intelligence on a server.

[1674] "Means for sending and displaying received responses to a user terminal" refers to a device or program in which the server formats suggestions and responses obtained from a generative artificial intelligence into an appropriate format, and sends and displays them to the user's terminal.

[1675] "Means for receiving user feedback" refers to a device or program that collects opinions and requests from users regarding the proposed content.

[1676] "Means of sending a re-request to a generative artificial intelligence based on feedback" refers to a device or program that analyzes user feedback and sends a re-request to a generative artificial intelligence.

[1677] This invention is a system that recognizes user emotions and makes service suggestions that take them into consideration. The specific form of implementation is described below. The system uses a user terminal, a server, a generative artificial intelligence system, and an emotion engine.

[1678] User terminal

[1679] User terminals are devices such as smartphones and smart glasses. These terminals have the following main functions:

[1680] 1. Enter user information

[1681] The user terminal displays a screen for entering user information. Here, the user enters basic information such as their age, gender, and areas of interest, as well as specific requests.

[1682] 2. Emotion recognition

[1683] When a user enters information, the device's built-in camera and microphone are used to capture the user's facial expressions and voice. This data is then sent to the emotion engine for analysis.

[1684] server

[1685] The server performs the following roles:

[1686] 1. Data transmission

[1687] It receives user information, requests, and sentiment analysis results sent from the user's terminal, and converts them into a request format suitable for generative artificial intelligence.

[1688] 2. Requests to Generative Artificial Intelligence

[1689] A request is sent to a generative AI model (e.g., GPT-3) to generate specific suggestions.

[1690] 3. Receiving and formatting the response

[1691] The system formats the response received from the generative artificial intelligence into an appropriate data format and sends it to the user's terminal.

[1692] Generative artificial intelligence

[1693] Generative artificial intelligence receives requests from a server and generates optimal suggestions. This generation process takes into account the user's basic information, requests, and emotional state.

[1694] Emotional Engine

[1695] The emotion engine analyzes the user's facial expressions and voice to determine their emotional state. The results are then sent to the server.

[1696] Hardware and software used

[1697] User devices: smartphones, smart glasses

[1698] Server: Cloud server or on-premises server

[1699] Emotion Engine: Emotion Recognition API

[1700] Generative artificial intelligence: Generative AI models (e.g., GPT-3)

[1701] Specific example of processing

[1702] The following is an example of how to specifically suggest the most suitable product when a user is feeling relaxed.

[1703] 1. Enter user information:

[1704] Example: The user enters information such as "30s, female, enjoys outdoor activities" on their smartphone.

[1705] The user enters a request such as, "I want to relax, I want to find a book."

[1706] 2. Recognition of emotions:

[1707] The user's device captures the user's facial expressions, and the emotion engine analyzes that data to determine that the user is "relaxed."

[1708] 3. Example of a prompt:

[1709] "Generate product suggestions suitable for a female user in her 30s who enjoys the outdoors and wants to relax. The user's basic information is as follows: Age: 30s, Gender: Female, Interests: Outdoors, Emotion: Want to relax. Please generate five suggestions."

[1710] Sending this prompt to the generating AI model yields suggestions such as:

[1711] 1. Aromatherapy Candles

[1712] 2. Relaxation chair

[1713] 3. Relaxation music subscription

[1714] 4. Herb tea set

[1715] 5. Essential oils

[1716] This suggestion is displayed on the user's terminal, providing the user with optimal product knowledge. In this way, a system can be built that enables service suggestions that take into account the user's emotional state.

[1717] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1718] Step 1:

[1719] The user enters basic information and requests using a device. The user enters basic information such as age, gender, and areas of interest, as well as specific requests (e.g., "I want to relax, I want to find a book") through an input form displayed on the screen of a smartphone or smart glasses.

[1720] Input: User's basic information and requests

[1721] Output: The information entered by the user is temporarily stored on the device.

[1722] Step 2:

[1723] The device captures the user's facial expressions and voice. It uses the device's built-in camera and microphone to collect user facial expression and voice data.

[1724] Input: User's facial expressions and voice

[1725] Output: Captured facial expression data and audio data

[1726] Step 3:

[1727] The device sends the captured data to an emotion engine for analysis. The device then sends the data to emotion analysis software such as the Emotion Recognition API to obtain the emotion analysis results.

[1728] Input: Facial expression data and audio data

[1729] Output: Emotion analysis results (e.g., "Relaxed state")

[1730] Step 4:

[1731] The sentiment analysis results are sent from the terminal to the server. Along with the analysis results, the terminal also sends the user's basic information and requests to the server.

[1732] Input: Sentiment analysis results, basic information, request

[1733] Output: Data sent to the server

[1734] Step 5:

[1735] The server sends a request to the generative artificial intelligence based on the data it receives. The server analyzes the received data, converts it into a request format suitable for the generative AI model, and sends it. For example,

[1736] "Generate product suggestions suitable for a female user in her 30s who enjoys the outdoors and wants to relax. The user's basic information is as follows: Age: 30s, Gender: Female, Interests: Outdoors, Emotion: Want to relax. Please generate five suggestions."

[1737] Input: User information, requests, sentiment analysis results

[1738] Output: Request sent to the generative artificial intelligence

[1739] Step 6:

[1740] The server receives the response from the generative artificial intelligence. The server receives the suggestions sent by the generative AI model and performs analysis.

[1741] Input: Response from generative artificial intelligence

[1742] Output: Proposed ideas (e.g., aromatherapy candles, relaxation chairs, etc.)

[1743] Step 7:

[1744] The server sends the received response to the user terminal and displays it on the terminal. The server formats the proposed content into an appropriate format and sends it to the user terminal for display.

[1745] Input: Proposal Content

[1746] Output: Data sent to the user terminal

[1747] Step 8:

[1748] The device sends user feedback to the server. The user enters feedback on the displayed suggestions, and the device sends that information to the server.

[1749] Input: User feedback

[1750] Output: Feedback sent to the server

[1751] Step 9:

[1752] The server sends a new request to the generative artificial intelligence based on the feedback. The feedback is analyzed, and another request is sent to the generative AI model to generate new suggestions.

[1753] Input: Feedback Information

[1754] Output: Re-request sent to the generative AI

[1755] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1756] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1757] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1758] [Fourth Embodiment]

[1759] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1760] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1761] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1762] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1763] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1764] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1765] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1766] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1767] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1768] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1769] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1770] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1771] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1772] One embodiment of the present invention describes below a specific system for a user to incorporate a particular service into their life and make the most of it. This system provides the user with useful information through the input of user information, generation of suggestions by generative artificial intelligence, display of the suggested content, and processing of user feedback.

[1773] System-wide configuration

[1774] The system primarily consists of three parts: user terminals, servers, and generative artificial intelligence. These elements communicate and cooperate with each other via the internet to enable the overall system to function.

[1775] User terminal

[1776] 1. Enter basic information

[1777] The terminal first displays a screen for user information input. The user enters basic information such as their age, gender, and areas of interest. This information is later processed by generative artificial intelligence.

[1778] 2. Input of request

[1779] The terminal also provides an interface for receiving specific requests from the user. For example, the user can input a specific request such as, "Please tell me about situations in which PayPay can be used in daily life."

[1780] 3. Sending data to the server

[1781] The terminal sends user information and requests to the server. The transmitted data is used in the next processing step.

[1782] server

[1783] 4. Data processing and request generation

[1784] The server processes the received data, including user information and request details, and uses this information to generate appropriate requests for the generative artificial intelligence. For example, it might send a request to the generative AI such as, "Generate scenarios showing how a man in his 30s who enjoys outdoor activities can use PayPay in his daily life."

[1785] 5. Receiving a response from a generative artificial intelligence.

[1786] The server receives a response from the generative artificial intelligence. This response includes specific suggestions tailored to the user's life scenarios.

[1787] 6. Formatting and sending the results

[1788] The server formats the response from the generative artificial intelligence into an appropriate format and sends it to the user terminal.

[1789] Generative artificial intelligence

[1790] 7. Proposal generation

[1791] Generative artificial intelligence receives requests from a server and generates suggestions for specific life scenarios based on user information and requests. These suggestions show how the user can specifically utilize the service.

[1792] Specific example

[1793] 1. Enter user information

[1794] Example: The user enters information on their device such as "30s, male, enjoys outdoor activities."

[1795] 2. Input of request

[1796] Example: The user enters a request such as, "Tell me 10 situations in everyday life where PayPay can be used."

[1797] 3. Sending data to the server

[1798] The terminal sends user information and requests to the server.

[1799] 4. Data processing and request generation

[1800] Based on the data the server receives, it sends a request to the generative artificial intelligence saying, "Please generate scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[1801] 5. Receiving a response from a generative artificial intelligence.

[1802] The generative artificial intelligence generates the following suggestions and returns them to the server:

[1803] 1. Purchase camping equipment

[1804] 2. Payment of admission fees for outdoor events

[1805] 3. Paying for meals at restaurants

[1806] 4. Paying for fuel at a gas station

[1807] 5. Payment of public transport fares

[1808] 6. Payment of sports facility usage fees

[1809] 7. Outdoor-related subscription services

[1810] 8. Payment of travel expenses at the travel agency

[1811] 9. Booking and payment for accommodation

[1812] 10. Booking and Payment for Activities

[1813] 6. Formatting and sending the results

[1814] The server formats the generated proposal into the appropriate data format and sends it to the user's terminal.

[1815] 7. Displaying results and receiving feedback

[1816] The device displays the results to the user, who then provides feedback on the suggestions. For example, the user might input, "Which store is best for buying camping equipment?"

[1817] 8. Processing and resubmitting feedback

[1818] The server receives user feedback and sends a request to the generative artificial intelligence. The generative AI generates specific suggestions again, which the server receives, formats appropriately, and sends back to the user's terminal.

[1819] In this way, the system incorporates user feedback and has the function of continuously and concretely suggesting how users can utilize the services provided in their actual daily lives.

[1820] The following describes the processing flow.

[1821] Step 1:

[1822] The device displays a screen that prompts the user to enter basic information (age, gender, areas of interest, etc.).

[1823] Step 2:

[1824] The user enters information such as their age, gender, and areas of interest into the displayed input form.

[1825] Step 3:

[1826] The device sends basic information entered by the user to the server. This data includes the user's age, gender, and areas of interest.

[1827] Step 4:

[1828] The terminal displays an interface that prompts the user to input requests regarding specific service usage scenarios (e.g., "Please tell me about scenarios in which PayPay can be used in daily life").

[1829] Step 5:

[1830] The user enters their request into the terminal.

[1831] Step 6:

[1832] The terminal sends the user's request to the server. The transmitted data includes the details of the user's request.

[1833] Step 7:

[1834] The server converts the received user information and request into a request format appropriate for the generative artificial intelligence.

[1835] Step 8:

[1836] The server sends a request to the generative artificial intelligence. The request includes the user's age, gender, areas of interest, and the content of the request.

[1837] Step 9:

[1838] Generative artificial intelligence processes received requests and generates specific suggestions that correspond to the user's life scenarios.

[1839] Step 10:

[1840] The generative artificial intelligence sends the generated suggestions back to the server as a response.

[1841] Step 11:

[1842] The server receives the response from the generative artificial intelligence and formats it into an appropriate data format (e.g., JSON).

[1843] Step 12:

[1844] The server sends the formatted proposal to the terminal.

[1845] Step 13:

[1846] The device displays received suggestions to the user. These suggestions include specific examples of service usage tailored to the user's lifestyle.

[1847] Step 14:

[1848] Users provide feedback on the displayed suggestions (e.g., "Which store is best for purchasing camping equipment?").

[1849] Step 15:

[1850] The device sends user feedback to the server.

[1851] Step 16:

[1852] The server sends a new request to the generative artificial intelligence based on the user's feedback. This request includes additional information about the user.

[1853] Step 17:

[1854] The generative artificial intelligence receives a follow-up request and generates additional specific suggestions.

[1855] Step 18:

[1856] The generative artificial intelligence sends the generated additional suggestions back to the server as a response.

[1857] Step 19:

[1858] The server receives additional responses from the generative artificial intelligence, formats them into the appropriate data format, and sends them to the terminal.

[1859] Step 20:

[1860] The device displays the received additional suggestions to the user.

[1861] This series of steps allows the system to continuously provide specific suggestions that align with the user's requirements.

[1862] (Example 1)

[1863] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1864] Conventional systems struggled to generate personalized suggestions based on individual user requests, failing to improve user satisfaction. Furthermore, they lacked sufficient mechanisms for efficiently incorporating user feedback and updating suggestions.

[1865] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1866] In this invention, the server includes means for processing user information and requests, generating and sending prompt messages to a generative artificial intelligence; means for receiving responses from the generative artificial intelligence; and means for formatting the received responses into an appropriate data format and sending them to the user terminal. This makes it possible to generate personalized suggestions based on the user's individual requests and to efficiently incorporate feedback to update the suggestions.

[1867] "Means for inputting basic user information" refers to providing an interface for inputting basic user information such as age, gender, and areas of interest.

[1868] "Means for inputting user requests" refers to means of providing an interface for users to input specific requests (for example, questions about how to use a particular service).

[1869] "Means for sending entered user information and requests to the server" refers to means for sending the basic information and requests entered by the user to the server via the network.

[1870] "Means for processing user information and requests, and generating and sending prompt messages to a generative artificial intelligence" refers to means for creating and sending appropriate prompt messages to a generative artificial intelligence based on the received user information and requests.

[1871] "Means for receiving responses from generative artificial intelligence" refers to means for the server to receive proposals and responses generated by generative artificial intelligence.

[1872] "Means for formatting the received response into an appropriate data format and sending it to the user terminal" refers to means of converting the response received from the generative artificial intelligence into a format that is easy for the user to understand and sending it again to the user terminal via the network.

[1873] "Means for displaying responses to the user and receiving feedback from the user" refers to means that the user terminal provides an interface for displaying responses from generative artificial intelligence to the user and for receiving feedback from the user.

[1874] "Method for sending a new request to a generative artificial intelligence based on feedback" refers to a method for generating a new prompt message based on feedback received from the user and sending it to the generative artificial intelligence.

[1875] Overall system configuration

[1876] This invention is a system that enables users to incorporate specific services into their lives and make the most of them. This system provides users with useful information through the input of user information, generation of suggestions by generative artificial intelligence, display of the suggested content, and processing of user feedback.

[1877] Hardware and software to be used

[1878] The system primarily consists of three parts: user terminals, servers, and generative artificial intelligence. These elements communicate with each other via the internet and work together as a whole system. User terminals include smartphones, tablets, and personal computers. Servers include web servers and cloud servers. Generative artificial intelligence utilizes computing resources (e.g., cloud-based AI services) capable of executing appropriate AI models.

[1879] Program processing

[1880] The specific processing details of the system are as follows:

[1881] Entering user information

[1882] The user terminal displays a screen prompting the user to enter basic information such as age, gender, and areas of interest. The user enters information such as "30s, male, enjoys outdoor activities." This information is encoded in JSON format or similar and sent to the server.

[1883] Request Input

[1884] The user terminal provides an interface for the user to input specific requests. The user might input a specific request such as, "Please tell me about situations in daily life where PayPay can be used."

[1885] Sending data to the server

[1886] The user terminal sends the entered user information and requests to the server. The server receives this information and parses it.

[1887] Data processing and request generation

[1888] The server generates and sends a prompt message to the generative artificial intelligence based on user information and the request. For example, a request such as "Generate scenarios showing how a man in his 30s who enjoys outdoor activities can use PayPay in his daily life" might be generated.

[1889] Received a response from a generative artificial intelligence.

[1890] Generative artificial intelligence receives requests and generates suggestions for specific life scenarios based on user information. For example, it might generate suggestions such as "purchase camping equipment" or "pay admission fees for outdoor events." These responses are then returned to the server.

[1891] Formatting and sending the results

[1892] The server formats the response received from the generative artificial intelligence into an appropriate data format and sends it back to the user terminal. In this process, the data is formatted into a list and then encoded again in JSON format.

[1893] Displaying results and accepting feedback

[1894] The user terminal displays the received suggestions to the user. The user then reviews these suggestions and provides more specific feedback. For example, they might input feedback such as, "Which store is best for purchasing camping equipment?"

[1895] Processing and resubmitting feedback

[1896] Based on the feedback received from the user, the server generates a new prompt and sends it to the generative artificial intelligence. The generative AI then generates specific suggestions, which the server receives, formats appropriately, and sends back to the user's terminal.

[1897] Specific example

[1898] 1. Enter user information

[1899] The user enters information on their device such as "30s, male, enjoys outdoor activities."

[1900] 2. Input of request

[1901] The user enters a request asking for "10 ways to use PayPay in everyday life."

[1902] 3. Sending data to the server

[1903] The user terminal sends the entered data to the server.

[1904] 4. Data processing and request generation

[1905] The server parses the data and generates and sends a prompt message: "Generate scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[1906] 5. Receiving a response from a generative artificial intelligence.

[1907] Generative artificial intelligence generates suggestions and returns them to the server. Examples: "Purchase camping equipment," "Pay admission fee for outdoor event," etc.

[1908] 6. Formatting and sending the results

[1909] The server formats the proposal and sends it to the user's terminal.

[1910] 7. Displaying results and receiving feedback

[1911] The user's device displays suggestions to the user, who then provides more specific feedback.

[1912] 8. Processing and resubmitting feedback

[1913] The server processes the feedback and sends a new prompt to the generative artificial intelligence again, which generates a new suggestion, formats it again, and sends it to the user terminal.

[1914] The above describes a specific embodiment of the present invention, a system that efficiently provides personalized suggestions based on the individual needs of users and continuously improves while incorporating feedback.

[1915] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1916] Step 1: Enter user information

[1917] The user terminal displays a screen prompting the user to enter basic information such as age, gender, and areas of interest. The user enters information such as "30s, male, enjoys outdoor activities" into the input screen. The entered information is temporarily stored on the terminal. Input data formats such as JSON and XML are used.

[1918] Input: User's basic information (age, gender, areas of interest, etc.)

[1919] Output: Encoded user basic information data

[1920] Step 2: Enter the request

[1921] The user terminal provides an interface for entering specific requests. Users enter specific requests such as, "Please tell me about situations where PayPay can be used in daily life." The entered requests are then encoded into a data format.

[1922] Input: User request (e.g., "Please tell me about situations in daily life where PayPay can be used.")

[1923] Output: Encoded user request data

[1924] Step 3: Sending data to the server

[1925] The user terminal sends the entered user information and requests to the server. The HTTPS protocol is used for transmission, and the data is encoded in JSON format. Upon successful communication, the data is stored on the server.

[1926] Input: Encoded user basic information data and user request data

[1927] Output: User information and request data stored on the server

[1928] Step 4: Data processing and request generation

[1929] The server parses and analyzes the received user information and requests. For example, it extracts information such as age, gender, and areas of interest. Next, it generates a prompt message to send to the generative artificial intelligence based on that information. The prompt message is in a format such as, "Generate 10 scenarios showing how a man in his 30s who enjoys the outdoors can use PayPay in his daily life."

[1930] Input: User information and request data stored on the server

[1931] Output: Generated prompt message

[1932] Step 5: Receiving a response from the generative artificial intelligence.

[1933] The server sends the generated prompt to the generative artificial intelligence (AI). The AI ​​receives the request, analyzes it based on the prompt, and generates specific suggestions to address the user's request. These suggestions are then returned to the server.

[1934] Input: Generated prompt message

[1935] Output: Response data from generative artificial intelligence (e.g., suggestions such as "purchase camping equipment" or "pay admission fee for an outdoor event")

[1936] Step 6: Format and submit the results

[1937] The server formats the response received from the generative artificial intelligence into an appropriate data format. The formatted data is then encoded again in JSON format and sent to the user's terminal.

[1938] Input: Response data from generative artificial intelligence

[1939] Output: Formatted proposal data

[1940] Step 7: Displaying results and receiving feedback

[1941] The user terminal displays a list of received suggestions to the user. The user reviews the suggestions and enters more specific feedback. For example, they might enter feedback such as, "Which store is best for purchasing camping equipment?" and click the submit button.

[1942] Input: Formatted proposal data

[1943] Output: User feedback (e.g., specific questions or requests)

[1944] Step 8: Processing feedback and resubmitting requests

[1945] The server parses and analyzes the feedback received from the user. Based on the feedback, it generates a new prompt and sends it to the generative artificial intelligence. The generative AI generates another suggestion, which the server receives, formats appropriately, and sends back to the user's terminal.

[1946] Input: User feedback data

[1947] Output: Newly generated prompt sentence and new response data from the generative AI.

[1948] Through the above processing steps, the system can efficiently provide personalized suggestions based on the user's individual requests and continuously improve by incorporating feedback.

[1949] (Application Example 1)

[1950] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1951] This invention relates to a system that generates suggestions tailored to the user's lifestyle and maximizes user convenience. In particular, to improve the shopping experience in physical stores, there is a need to provide optimized suggestions in real time using the user's current location information. However, conventional systems have the problem of not being able to improve user satisfaction because they do not provide insufficient suggestions based on user information and do not perform real-time optimization using the current location information.

[1952] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1953] In this invention, the server includes means for having a display device for inputting user information, means for transmitting the input user information and requests to a communication device, means for receiving responses from generative artificial intelligence, means for acquiring the user's current location information and optimizing response suggestions based on the location information, and means for displaying the generated suggestions in real time on a smartphone or smart glasses. This enables the user to fully enjoy the shopping experience in a physical store.

[1954] "User information" refers to basic data such as the user's age, gender, areas of interest, and current location.

[1955] A "display device" is an interface for users to input and confirm information, and includes devices such as smartphones and smart glasses.

[1956] A "communication device" is a device used to send user information and requests to a server.

[1957] "Generative artificial intelligence" refers to an artificial intelligence system that generates optimal suggestions based on user information and requests.

[1958] "Current location information" refers to the user's real-time geographical location data.

[1959] "Real-time" refers to a timeframe that responds immediately to the user's current actions and circumstances.

[1960] A "smartphone" is a type of mobile phone that is a device with multi-functional computer capabilities.

[1961] "Smart glasses" are wearable devices designed to enhance visual information.

[1962] The following describes specific embodiments for carrying out the present invention. The invention relates to a system for users to incorporate specific services into their lives and make the most of them. The specific configuration and operation will be described below.

[1963] System-wide configuration

[1964] The system primarily consists of the following elements: user terminals, servers, and generative artificial intelligence. These elements communicate with each other via the internet and cooperate to function as a whole system.

[1965] User terminal

[1966] A user terminal is equipped with a display device and provides an interface that allows the user to input information. Examples include smartphones and smart glasses.

[1967] 1. Enter basic information

[1968] The user enters basic information such as age, gender, and areas of interest on the display device of their user terminal.

[1969] 2. Input of request

[1970] Users utilize an interface to enter specific requests. For example, they might enter a request such as, "Please recommend some outdoor products."

[1971] 3. Sending data to the server

[1972] The user terminal transmits the entered user information and requests to the server via a communication device.

[1973] server

[1974] The server processes the data sent from the user terminal and generates appropriate requests for the generative artificial intelligence.

[1975] 4. Data processing and request generation

[1976] The server organizes the received user information and request details, and then sends the request to the generative artificial intelligence based on that information.

[1977] For example, generate a prompt such as, "Generate suggestions on what products a man in his 30s who enjoys outdoor activities should purchase at a physical store."

[1978] 5. Receiving a response from a generative artificial intelligence.

[1979] The server receives the response from the generative artificial intelligence and formats it into an appropriate format.

[1980] 6. Optimization and Display

[1981] The server obtains the user's current location information and optimizes response suggestions based on that location. The generated suggestions are then displayed in real time on the user's smartphone or smart glasses.

[1982] Generative artificial intelligence

[1983] Generative artificial intelligence receives requests from a server and generates specific lifestyle scenarios and product suggestions based on user information and requests.

[1984] Specific example

[1985] 1. Enter user information

[1986] Example: A user enters information such as "30s, male, enjoys outdoor activities" on their smartphone.

[1987] 2. Input of request

[1988] Example: A user voice-inputs, "Please recommend some outdoor products."

[1989] 3. Sending data to the server

[1990] The user terminal sends user information and requests to the server.

[1991] 4. Data processing and request generation

[1992] The server sends the prompt message, "A man in his 30s who enjoys the outdoors is seeking advice on recommended outdoor products available at physical stores," to the generative AI.

[1993] 5. Receiving a response from a generative artificial intelligence.

[1994] The generative artificial intelligence generates a response and returns it to the server. This response may include suggestions such as, "A new camping tent has recently arrived at a nearby sporting goods store."

[1995] 6. Optimization and Display

[1996] The server optimizes the suggested content based on the user's current location, sends it to the user's terminal, and displays it in real time.

[1997] In this way, the system incorporates user feedback and has the function of continuously and concretely suggesting how users can utilize the services provided in their actual daily lives.

[1998] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1999] Step 1:

[2000] The user enters basic information such as age, gender, and areas of interest using the display device on their terminal. For example, they might enter information such as "30s, male, enjoys outdoor activities." This input data is collected on the terminal.

[2001] Step 2:

[2002] The user terminal provides an interface for inputting specific requests from the user. Users input requests such as "Please recommend some outdoor products" via voice or text. This request data is also collected by the terminal.

[2003] Step 3:

[2004] The user terminal transmits the entered user information and requests to the server via a communication device. At this time, the input data and request data are combined into a single request packet and sent to the server.

[2005] Step 4:

[2006] The server processes the received user information and request details to generate an appropriate request for the generative artificial intelligence. For example, it might generate a prompt such as, "Generate suggestions on what products a man in his 30s who enjoys outdoor activities should purchase at a physical store." This prompt becomes the input data for the generative artificial intelligence.

[2007] Step 5:

[2008] Generative artificial intelligence receives prompt messages from a server and generates specific lifestyle scenarios and product suggestions based on user information and requests. For example, it might generate a suggestion such as, "A new camping tent has recently arrived at a nearby sports goods store." This suggestion is then sent back to the server.

[2009] Step 6:

[2010] The server receives a response from the generative artificial intelligence and obtains the user's current location information. Based on the obtained location information, it further optimizes the response. For example, it processes the data to provide information about the nearest store based on the user's location. This optimized suggestion is then sent to the user's terminal.

[2011] Step 7:

[2012] The user's device displays received suggestions in real time on their smartphone or smart glasses. This allows users to instantly see the best suggestions to enhance their in-store shopping experience.

[2013] Step 8:

[2014] Users provide feedback on the displayed suggestions. For example, they might enter feedback such as, "This information was helpful. I'd like to know more details." This user feedback is collected on the device.

[2015] Step 9:

[2016] The user terminal sends feedback from the user to the server. The feedback data arrives at the server.

[2017] Step 10:

[2018] Based on the feedback received, the server sends another request to the generative artificial intelligence. For example, it might generate a prompt such as, "A man in his 30s who enjoys the outdoors is looking for more detailed information on outdoor products," and send it to the generative AI.

[2019] Step 11:

[2020] The generative artificial intelligence generates a response again and sends it to the server. This process is repeated in accordance with user feedback, allowing the system to continuously provide the most relevant information to the user.

[2021] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[2022] As one embodiment of the present invention, a specific embodiment of a system that recognizes a user's emotions and makes more appropriate service suggestions based on those emotions is described below. This system provides the user with optimal information through the input of user information, suggestion generation by generative artificial intelligence, emotion recognition by an emotion engine, display of the suggested content, and processing of user feedback.

[2023] System-wide configuration

[2024] The system primarily consists of the following four parts: user terminal, server, generative artificial intelligence, and emotion engine. These elements communicate with each other via the internet and cooperate to enable the overall system to function.

[2025] User terminal

[2026] 1. Enter basic information

[2027] The device first displays a screen for user information input. The user enters basic information such as their age, gender, and areas of interest. This information is later processed by generative artificial intelligence and an emotion engine.

[2028] 2. Input of request

[2029] The terminal provides an interface for users to input specific requests (e.g., "Please tell me about situations in which PayPay can be used in daily life").

[2030] 3. Recognition of emotions

[2031] The device captures the user's facial expressions and voice as they enter their requests and sends them to the emotion engine.

[2032] 4. Sending data to the server

[2033] The device sends user information, requests, and sentiment analysis results from the sentiment engine to the server. The transmitted data includes the user's age, gender, areas of interest, request content, and sentiment analysis results.

[2034] server

[2035] 5. Data processing and request generation

[2036] The server receives data, including user information, request details, and sentiment analysis results, and then converts it into an appropriate request format for the generative artificial intelligence. For example, it might send a request to the generative AI such as, "Generate scenarios showing how a man in his 30s who enjoys the outdoors and has positive emotions can use PayPay in his daily life."

[2037] 6. Receiving a response from a generative artificial intelligence.

[2038] The server receives a response from the generative artificial intelligence. This response includes specific suggestions tailored to the user's life scenarios and is customized based on the sentiment analysis results.

[2039] 7. Formatting and sending the results

[2040] The server formats the response from the generative artificial intelligence into an appropriate format and sends it to the user terminal.

[2041] Generative artificial intelligence

[2042] 8. Proposal generation

[2043] Generative artificial intelligence receives requests from the server and generates specific life scenario suggestions based on user information, requests, and sentiment analysis results. These suggestions show how the user can specifically use the service and also take into account the user's emotional state.

[2044] Specific example

[2045] 1. Enter user information

[2046] Example: The user enters information on their device such as "30s, male, enjoys outdoor activities."

[2047] 2. Input of request

[2048] Example: The user enters a request such as, "Tell me 10 situations in everyday life where PayPay can be used."

[2049] 3. Recognition of emotions

[2050] The device captures the user's facial expressions with its camera, and an emotion engine analyzes them to determine if they are "positive."

[2051] 4. Sending data to the server

[2052] The device sends user information, requests, and sentiment analysis results to the server.

[2053] 5. Data processing and request generation

[2054] Based on the data the server receives, it sends a request to the generative artificial intelligence saying, "Please generate scenarios showing how a man in his 30s, who enjoys the outdoors and has positive emotions, can use PayPay in his daily life."

[2055] 6. Receiving a response from a generative artificial intelligence.

[2056] The generative artificial intelligence generates the following suggestions and returns them to the server:

[2057] 1. Purchase camping equipment

[2058] 2. Payment of admission fees for outdoor events

[2059] 3. Paying for meals at restaurants

[2060] 4. Paying for fuel at a gas station

[2061] 5. Payment of public transport fares

[2062] 6. Payment of sports facility usage fees

[2063] 7. Outdoor-related subscription services

[2064] 8. Payment of travel expenses at the travel agency

[2065] 9. Booking and payment for accommodation

[2066] 10. Booking and Payment for Activities

[2067] 7. Formatting and sending the results

[2068] The server formats the generated proposal into the appropriate data format and sends it to the user's terminal.

[2069] 8. Displaying results and receiving feedback

[2070] The device displays the results to the user, who then provides feedback on the suggestions. For example, the user might input, "Which store is best for purchasing camping equipment?"

[2071] 9. Processing and resubmitting feedback

[2072] The server receives user feedback and sends a request to the generative artificial intelligence. The generative AI generates specific suggestions again, which the server receives, formats appropriately, and sends back to the user's terminal.

[2073] Through these steps, the system takes user emotions into consideration and proposes services in a more appropriate way. This allows users to more concretely integrate the services offered into their own lives.

[2074] The following describes the processing flow.

[2075] Step 1:

[2076] The device displays a screen that prompts the user to enter basic information (age, gender, areas of interest, etc.).

[2077] Step 2:

[2078] The user enters information such as their age, gender, and areas of interest into the displayed input form.

[2079] Step 3:

[2080] The device sends basic information entered by the user to the server. This data includes the user's age, gender, and areas of interest.

[2081] Step 4:

[2082] The terminal displays an interface that prompts the user to input requests regarding specific service usage scenarios (e.g., "Please tell me about scenarios in which PayPay can be used in daily life").

[2083] Step 5:

[2084] The user enters their request into the terminal.

[2085] Step 6:

[2086] The terminal sends the user's request to the server. The transmitted data includes the details of the user's request.

[2087] Step 7:

[2088] The device captures the user's facial expressions and voice as they enter their requests and sends this information to the emotion engine.

[2089] Step 8:

[2090] The emotion engine analyzes captured facial and audio data to recognize the user's emotional state (e.g., positive, negative, neutral).

[2091] Step 9:

[2092] The emotion engine sends the analysis results to the server. The transmitted data includes the user's emotional state.

[2093] Step 10:

[2094] The server converts the received user information, request details, and sentiment analysis results into a request format appropriate for the generative artificial intelligence.

[2095] Step 11:

[2096] The server sends a request to the generative artificial intelligence. The request includes the user's age, gender, areas of interest, request content, and sentiment analysis results.

[2097] Step 12:

[2098] Generative artificial intelligence processes received requests and generates specific suggestions tailored to the user's life scenarios. These suggestions also reflect the user's emotional state.

[2099] Step 13:

[2100] The generative artificial intelligence sends the generated suggestions back to the server as a response.

[2101] Step 14:

[2102] The server receives the response from the generative artificial intelligence and formats it into an appropriate data format (e.g., JSON).

[2103] Step 15:

[2104] The server sends the formatted proposal to the terminal.

[2105] Step 16:

[2106] The device displays received suggestions to the user. These suggestions include specific examples of service usage tailored to the user's lifestyle.

[2107] Step 17:

[2108] Users provide feedback on the displayed suggestions (e.g., "Which store is best for purchasing camping equipment?").

[2109] Step 18:

[2110] The device sends user feedback to the server.

[2111] Step 19:

[2112] The server sends a new request to the generative artificial intelligence based on the user's feedback. This request includes additional information about the user.

[2113] Step 20:

[2114] The generative artificial intelligence receives a follow-up request and generates additional specific suggestions.

[2115] Step 21:

[2116] The generative artificial intelligence sends the generated additional suggestions back to the server as a response.

[2117] Step 22:

[2118] The server receives additional responses from the generative artificial intelligence, formats them into the appropriate data format, and sends them to the terminal.

[2119] Step 23:

[2120] The device displays the received additional suggestions to the user.

[2121] Through this series of steps, the system takes into account not only the user's requests but also their emotional state, enabling it to specifically suggest the optimal service usage scenario.

[2122] (Example 2)

[2123] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2124] Traditional systems often fail to consider user emotions when proposing services, resulting in the provision of information and service suggestions that are not optimal for user needs. This can lead to a diminished user experience and a decrease in the frequency of service usage. Furthermore, the lack of sufficient functionality to quickly revise suggestions based on feedback makes it difficult to promptly provide users with the specific information they require.

[2125] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting user information, means for transmitting the inputted user information and requests, means for capturing the facial expressions and voice of the user when they input a request and performing emotion analysis, means for analyzing the captured facial expressions and voice and transmitting the emotion analysis results, means for transmitting a request to a generative artificial intelligence based on the received user information, requests, and emotion analysis results, means for receiving a response from the generative artificial intelligence, means for formatting the received response into an appropriate data format and transmitting it to the user terminal for display, means for receiving feedback from the user, and means for transmitting a re-request to the generative artificial intelligence based on the feedback. This enables the realization of optimal service proposals that take into account the user's emotions and enables rapid re-proposals based on feedback.

[2126] "User information" refers to basic data such as the user's age, gender, and areas of interest.

[2127] A "request" refers to the content of a user's request to the system, such as their wishes, questions, or specific service usage scenarios.

[2128] A "terminal" is a device that functions as the user interface of a system, and is a device that performs tasks such as inputting user information, sending requests, and displaying results.

[2129] A "server" is a central processing unit that receives user information, requests, and sentiment analysis results, sends requests to generative artificial intelligence, and receives, processes, and transmits responses from generative artificial intelligence.

[2130] An "emotion engine" is software or a device that analyzes a user's emotional state by performing emotional analysis based on the user's facial expressions and voice data.

[2131] "Generative artificial intelligence" is an artificial intelligence system that generates optimal service suggestions based on received user information, requests, and sentiment analysis results.

[2132] "Feedback" refers to responses from users to proposed services or information, such as evaluations, additional requests, or questions.

[2133] "Formatting" is the process of converting responses and suggestions received from generative artificial intelligence into a format that is easy for the user to understand.

[2134] A "request" is an inquiry, such as instructions or questions, that a server sends to a generative artificial intelligence system based on user requests.

[2135] "Data format" refers to the format that represents the structure and layout of data, and specifically refers to formats such as HTML and JSON.

[2136] This invention is a system that analyzes a user's emotions and provides appropriate service suggestions based on the results. The system mainly consists of four elements: a user terminal, a server, a generative artificial intelligence system, and an emotion engine. These elements communicate with each other via the internet and operate in an integrated manner.

[2137] User terminal

[2138] The user terminal handles user information input, request input, emotion recognition, data transmission, and result display. The terminal performs the following specific actions:

[2139] 1. Enter user information

[2140] The device screen displays a screen where the user can enter basic information such as age, gender, and areas of interest.

[2141] Example: The user enters "30s, male, enjoys outdoor activities."

[2142] 2. Input of request

[2143] The user enters a specific request via their device. For example, they might type, "Please tell me how I can use PayPay in my daily life."

[2144] 3. Recognition of emotions

[2145] The device uses its camera and microphone to capture the user's facial expressions and voice, and sends them to the emotion engine. This emotion engine then uses specialized software to perform emotion analysis.

[2146] Specific example: The emotion engine analyzes the user's facial expression and determines it to be "positive."

[2147] 4. Sending data to the server

[2148] The terminal sends user information, requests, and sentiment analysis results to the server.

[2149] server

[2150] The server receives data sent from the user terminal, sends a request to the generative artificial intelligence, and receives a response. The specific operation of the server is as follows:

[2151] 5. Processing of received data

[2152] The server stores user information, request details, and sentiment analysis results in a database.

[2153] 6. Sending requests to generative artificial intelligence

[2154] The server generates and sends a prompt message to the generative artificial intelligence based on the received data.

[2155] Example of a prompt:

[2156] Please create scenarios showing how a man in his 30s, who enjoys the outdoors and has a positive outlook, can use PayPay in his daily life.

[2157] 7. Receiving a response from a generative artificial intelligence.

[2158] Generative artificial intelligence generates suggestions tailored to the user's daily life and sends them back to the server.

[2159] Specific example: The following suggestions may be generated.

[2160] 1. Purchase camping equipment

[2161] 2. Payment of admission fees for outdoor events

[2162] 3. Paying for meals at restaurants

[2163] 4. Paying for fuel at a gas station

[2164] 5. Payment of public transport fares

[2165] 6. Payment of sports facility usage fees

[2166] 7. Outdoor-related subscription services

[2167] 8. Payment of travel expenses at the travel agency

[2168] 9. Booking and payment for accommodation

[2169] 10. Booking and Payment for Activities

[2170] 8. Formatting and sending the results

[2171] The server formats the generated proposals into the appropriate data format and sends them to the user's terminal.

[2172] Feedback processing

[2173] Users can provide feedback on proposals and then make revised proposals based on that feedback. The feedback process is as follows:

[2174] 9. Receiving user feedback

[2175] Users provide feedback on the suggestions via their devices.

[2176] Specific example: A user provides feedback asking, "Which store is best for purchasing camping equipment?"

[2177] 10. Processing and resubmitting feedback

[2178] The server receives user feedback and, based on that, sends a new request to the generative artificial intelligence. This new request generates a new suggestion, which is then sent back to the user's terminal.

[2179] In this way, the present invention makes it possible to propose optimal services that take user emotions into consideration and improve the user experience.

[2180] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2181] Step 1:

[2182] Entering user information

[2183] The device displays a screen for the user to enter basic information such as age, gender, and areas of interest. The user then enters their information and presses the submit button.

[2184] Input: Information about the user's age, gender, and areas of interest.

[2185] Output: Input user information data.

[2186] Specific operation: The terminal converts user information data into JSON format and sends it to the server.

[2187] Step 2:

[2188] Request Input

[2189] The terminal provides an interface that allows the user to enter a specific request. The user enters the request and presses the submit button.

[2190] Input: The request entered by the user.

[2191] Output: The input request data.

[2192] Specific operation: The terminal converts the request data into JSON format and sends it to the server.

[2193] Step 3:

[2194] Recognition of emotions

[2195] The device uses its camera and microphone to capture the user's facial expressions and voice as they input requests. This data is then sent to an emotion engine for analysis.

[2196] Input: User's facial expression data and voice data.

[2197] Output: Emotion analysis results.

[2198] Specific operation: The device sends the captured data to the emotion engine in real time, and the emotion engine returns the emotion analysis result, such as "positive."

[2199] Step 4:

[2200] Sending data to the server

[2201] The device sends user information, requests, and sentiment analysis results to the server.

[2202] Inputs: User information data, request data, sentiment analysis results.

[2203] Output: Server received data (user information, requests, sentiment analysis results).

[2204] Specific operation: The terminal combines this data into one file and sends it to the server.

[2205] Step 5:

[2206] Processing of received data

[2207] The server stores the received user information, request details, and sentiment analysis results in its database. It then prepares to send the request to the generative artificial intelligence.

[2208] Input: Server received data (user information, requests, sentiment analysis results).

[2209] Output: Request data to be sent to the generative artificial intelligence.

[2210] Specific operation: The server analyzes and organizes the received data and generates a prompt message.

[2211] Step 6:

[2212] Sending a request to a generative artificial intelligence

[2213] The server sends a prompt message to the generative artificial intelligence.

[2214] Input: Request data (prompt text).

[2215] Output: Response from a generative artificial intelligence.

[2216] Specific operation: The server generates a prompt message like the following and sends it to the generative artificial intelligence:

[2217] Please create scenarios showing how a man in his 30s, who enjoys the outdoors and has a positive outlook, can use PayPay in his daily life.

[2218] Step 7:

[2219] Received a response from a generative artificial intelligence.

[2220] The server receives responses from the generative artificial intelligence. These responses include suggestions tailored to the user's daily life scenarios.

[2221] Input: Response data from a generative artificial intelligence.

[2222] Output: Proposed content.

[2223] Specific action: Save the received proposal content to the database.

[2224] Step 8:

[2225] Formatting and sending the results

[2226] The server formats the generated proposals into the appropriate data format and sends them to the user's terminal.

[2227] Input: Proposal content.

[2228] Output: Proposed data format for display.

[2229] Specific operation: Convert suggestions received from generative artificial intelligence (e.g., "Purchase camping equipment," "Pay admission fee for outdoor event," etc.) into HTML or JSON format and send them to the terminal.

[2230] Step 9:

[2231] Receiving user feedback

[2232] The device displays a form for the user to enter feedback on the proposal. The user enters their feedback and presses the submit button.

[2233] Input: User feedback.

[2234] Output: Feedback data.

[2235] Specific operation: The user enters "Which store is best for purchasing camping equipment?" and sends this to the server.

[2236] Step 10:

[2237] Processing and resubmitting feedback

[2238] Based on the feedback received from the user, the server sends a new request to the generative artificial intelligence.

[2239] Input: Feedback data.

[2240] Output: Response to the re-request.

[2241] Specific operation: The server uses feedback data to generate a new prompt and sends it to the generative artificial intelligence. Then, it receives a revised suggestion and sends it back to the user terminal.

[2242] In this way, the system can provide optimal service suggestions that take user emotions into consideration, thereby improving the user experience.

[2243] (Application Example 2)

[2244] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2245] Conventional real-time service suggestion systems lacked the ability to consider the user's emotional state. As a result, optimal suggestions were not made at the timing and with the nuances the user desired, leading to decreased service satisfaction. This invention aims to improve the user experience by recognizing the user's emotions and making suggestions based on those emotions.

[2246] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes a screen display means for inputting user information, a means for transmitting the input user information and requests to the server, a means for capturing the user's facial expressions and voice and transmitting them to the emotion engine, a means for transmitting the emotion analysis results analyzed by the emotion engine to the server, a means for sending a request to a generative artificial intelligence based on the received data, a means for receiving a response from the generative artificial intelligence, a means for transmitting the received response to the user terminal and displaying it, a means for receiving feedback from the user, and a means for sending a re-request to the generative artificial intelligence based on the feedback. This makes it possible to propose services optimized for the user's emotional state.

[2247] "User information input screen display means" refers to a device or program that provides an interface that allows a user to input their basic information (age, gender, areas of interest, etc.) and requests.

[2248] "Means for transmitting entered user information and requests to the server" refers to a device or program that transmits information and requests entered by a user to a server via the Internet.

[2249] "Means for capturing user facial expressions and voice and transmitting them to an emotion engine" refers to a device or program that acquires the user's face and voice using a camera and microphone and sends that data to an emotion analysis device.

[2250] "Means for sending emotion analysis results analyzed by the emotion engine to the server" refers to a device or program in which an emotion analysis device analyzes the user's emotional state and sends the analysis results to the server.

[2251] "Means for sending requests to generative artificial intelligence based on received data" refers to a device or program that sends appropriate requests to generative artificial intelligence based on user information, requests, and sentiment analysis results received by the server.

[2252] "Means for receiving responses from generative artificial intelligence" refers to a device or program that receives proposals and responses generated by generative artificial intelligence on a server.

[2253] "Means for sending and displaying received responses to a user terminal" refers to a device or program in which the server formats suggestions and responses obtained from a generative artificial intelligence into an appropriate format, and sends and displays them to the user's terminal.

[2254] "Means for receiving user feedback" refers to a device or program that collects opinions and requests from users regarding the proposed content.

[2255] "Means of sending a re-request to a generative artificial intelligence based on feedback" refers to a device or program that analyzes user feedback and sends a re-request to a generative artificial intelligence.

[2256] This invention is a system that recognizes user emotions and makes service suggestions that take them into consideration. The specific form of implementation is described below. The system uses a user terminal, a server, a generative artificial intelligence system, and an emotion engine.

[2257] User terminal

[2258] User terminals are devices such as smartphones and smart glasses. These terminals have the following main functions:

[2259] 1. Enter user information

[2260] The user terminal displays a screen for entering user information. Here, the user enters basic information such as their age, gender, and areas of interest, as well as specific requests.

[2261] 2. Emotion recognition

[2262] When a user enters information, the device's built-in camera and microphone are used to capture the user's facial expressions and voice. This data is then sent to the emotion engine for analysis.

[2263] server

[2264] The server performs the following roles:

[2265] 1. Data transmission

[2266] It receives user information, requests, and sentiment analysis results sent from the user's terminal, and converts them into a request format suitable for generative artificial intelligence.

[2267] 2. Requests to Generative Artificial Intelligence

[2268] A request is sent to a generative AI model (e.g., GPT-3) to generate specific suggestions.

[2269] 3. Receiving and formatting the response

[2270] The system formats the response received from the generative artificial intelligence into an appropriate data format and sends it to the user's terminal.

[2271] Generative artificial intelligence

[2272] Generative artificial intelligence receives requests from a server and generates optimal suggestions. This generation process takes into account the user's basic information, requests, and emotional state.

[2273] Emotional Engine

[2274] The emotion engine analyzes the user's facial expressions and voice to determine their emotional state. The results are then sent to the server.

[2275] Hardware and software used

[2276] User devices: smartphones, smart glasses

[2277] Server: Cloud server or on-premises server

[2278] Emotion Engine: Emotion Recognition API

[2279] Generative artificial intelligence: Generative AI models (e.g., GPT-3)

[2280] Specific example of processing

[2281] The following is an example of how to specifically suggest the most suitable product when a user is feeling relaxed.

[2282] 1. Enter user information:

[2283] Example: The user enters information such as "30s, female, enjoys outdoor activities" on their smartphone.

[2284] The user enters a request such as, "I want to relax, I want to find a book."

[2285] 2. Recognition of emotions:

[2286] The user's device captures the user's facial expressions, and the emotion engine analyzes that data to determine that the user is "relaxed."

[2287] 3. Example of a prompt:

[2288] "Generate product suggestions suitable for a female user in her 30s who enjoys the outdoors and wants to relax. The user's basic information is as follows: Age: 30s, Gender: Female, Interests: Outdoors, Emotion: Want to relax. Please generate five suggestions."

[2289] Sending this prompt to the generating AI model yields suggestions such as:

[2290] 1. Aromatherapy Candles

[2291] 2. Relaxation chair

[2292] 3. Relaxation music subscription

[2293] 4. Herb tea set

[2294] 5. Essential oils

[2295] This suggestion is displayed on the user's terminal, providing the user with optimal product knowledge. In this way, a system can be built that enables service suggestions that take into account the user's emotional state.

[2296] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2297] Step 1:

[2298] The user enters basic information and requests using a device. The user enters basic information such as age, gender, and areas of interest, as well as specific requests (e.g., "I want to relax, I want to find a book") through an input form displayed on the screen of a smartphone or smart glasses.

[2299] Input: User's basic information and requests

[2300] Output: The information entered by the user is temporarily stored on the device.

[2301] Step 2:

[2302] The device captures the user's facial expressions and voice. It uses the device's built-in camera and microphone to collect user facial expression and voice data.

[2303] Input: User's facial expressions and voice

[2304] Output: Captured facial expression data and audio data

[2305] Step 3:

[2306] The device sends the captured data to an emotion engine for analysis. The device then sends the data to emotion analysis software such as the Emotion Recognition API to obtain the emotion analysis results.

[2307] Input: Facial expression data and audio data

[2308] Output: Emotion analysis results (e.g., "Relaxed state")

[2309] Step 4:

[2310] The sentiment analysis results are sent from the terminal to the server. Along with the analysis results, the terminal also sends the user's basic information and requests to the server.

[2311] Input: Sentiment analysis results, basic information, request

[2312] Output: Data sent to the server

[2313] Step 5:

[2314] The server sends a request to the generative artificial intelligence based on the data it receives. The server analyzes the received data, converts it into a request format suitable for the generative AI model, and sends it. For example,

[2315] "Generate product suggestions suitable for a female user in her 30s who enjoys the outdoors and wants to relax. The user's basic information is as follows: Age: 30s, Gender: Female, Interests: Outdoors, Emotion: Want to relax. Please generate five suggestions."

[2316] Input: User information, requests, sentiment analysis results

[2317] Output: Request sent to the generative artificial intelligence

[2318] Step 6:

[2319] The server receives the response from the generative artificial intelligence. The server receives the suggestions sent by the generative AI model and performs analysis.

[2320] Input: Response from generative artificial intelligence

[2321] Output: Proposed ideas (e.g., aromatherapy candles, relaxation chairs, etc.)

[2322] Step 7:

[2323] The server sends the received response to the user terminal and displays it on the terminal. The server formats the proposed content into an appropriate format and sends it to the user terminal for display.

[2324] Input: Proposal Content

[2325] Output: Data sent to the user terminal

[2326] Step 8:

[2327] The device sends user feedback to the server. The user enters feedback on the displayed suggestions, and the device sends that information to the server.

[2328] Input: User feedback

[2329] Output: Feedback sent to the server

[2330] Step 9:

[2331] The server sends a new request to the generative artificial intelligence based on the feedback. The feedback is analyzed, and another request is sent to the generative AI model to generate new suggestions.

[2332] Input: Feedback Information

[2333] Output: Re-request sent to the generative AI

[2334] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2335] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2336] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[2337] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2338] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[2339] These emotions are distributed at the 3 o'clock position on the Emot...

Claims

1. A means of displaying a screen for entering user information, Means for sending the entered user information and requests to the server, A means for sending a request to a generative artificial intelligence based on user information and requests, A means of receiving a response from a generative artificial intelligence, A means for sending and displaying the received response on the user terminal, A means of receiving user feedback, A means of sending a new request to the generative artificial intelligence based on the feedback, A system that includes this.

2. The system according to claim 1, further comprising means for generating suggestions corresponding to the user's life scenes based on information received by a generative artificial intelligence.

3. The system according to claim 1, further comprising means for formatting the generated proposal into an appropriate data format and transmitting it to a user terminal.

4. The system according to claim 1, further comprising means for sending user feedback on the displayed suggestions back to the generative artificial intelligence and generating suggestions again.

Images