system

Abstract

Provide a system. [Solution] a speech recognition means for converting user input into text; a transmitting means for transmitting the text data to a server; A generation means for the server to generate a menu based on the user's input information and pre-registered information; A transmitting means for transmitting the generated menu information to a terminal; a speech synthesis means for converting the received menu information into speech; an output means for transmitting the synthesized voice menu information to a user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Struggling to decide what to make for dinner every day is a common problem for many modern households. Especially when deciding on a menu after visiting the supermarket, it's easy to get lost in the confusion of what to buy, often resulting in wasted time. While existing menu suggestion apps exist, there are no systems that allow users to consult with the system in real time while shopping, resulting in users often experiencing hassle and stress. The present invention aims to solve this problem by providing a system that allows users to consult with the system in real time while shopping. [Means for solving the problem]

[0005] The present invention is a system that includes a speech recognition unit that converts user-input speech into text, a transmission unit that transmits the text data to a server, a generation unit that generates a menu based on the user-input information and pre-registered information, a transmission unit that transmits the generated menu information to a terminal, a speech synthesis unit that converts the received menu information into speech, and an output unit that notifies the user of the synthesized speech-based menu information. The system also includes a process in which the user inputs a revision request by speech, converts it into text using the speech recognition unit, and transmits it again to the server, generating new menu information and notifying the user again. The system also includes a reference unit that stores pre-registered information such as the number of family members, likes and dislikes, allergies, and health status in a database and retrieves information from the storage unit when generating a menu. This allows users to efficiently decide on a menu in real time at the supermarket, making shopping easier and less stressful.

[0006] "Speech recognition means" refers to a device or software function that converts user-input speech into text data.

[0007] The "transmission means" refers to a communication device or software function for transmitting text data to a server.

[0008] "Generation means" refers to a device or AI model that the server uses to generate an appropriate menu based on the user's input information and pre-registered information.

[0009] A "speech synthesis means" is a device or software function for converting text data into speech.

[0010] "Output means" refers to a device such as an earphone or speaker that conveys the voice generated by voice synthesis to the user.

[0011] "Storage means" refers to a device or software function for storing information such as the number of family members, likes and dislikes, allergy information, and health status of the user in a database.

[0012] The "reference means" refers to a device or software function that references the user information stored in the storage means and uses it when generating a menu. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. This system is composed of multiple means such as voice recognition, data transmission, AI generation, voice synthesis, and output. Specific embodiments are described below.

[0035] User Registration

[0036] User: Downloads the smartphone app and creates an account. When creating an account, the user enters information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is used to suggest more appropriate meals for future user inquiries.

[0037] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[0038] Server: Stores the received information in a database. This stored information is referenced later when generating menus.

[0039] Voice input and request sending

[0040] User: Arrives at the supermarket and connects earphones to the device. Launches the smartphone app and selects voice input mode. Enters a question: "What should I have for dinner tonight?"

[0041] Terminal: Converts the user's voice into text (voice recognition), and sends the converted text data to the server.

[0042] Server: Refers to the text data and the user's pre-registered information to generate an appropriate menu. This is done using a generative AI model.

[0043] Suggestion generation and voice response

[0044] Server: The generated menu information is sent to the terminal in text format.

[0045] Terminal: Converts received text data into speech (speech synthesis), and transmits the synthesized menu information to the user through earphones.

[0046] Fix requests and iterations

[0047] User: If the user is not satisfied with the proposed menu, they can say something like, "Is there an easier dish?" This speech is converted into text again and sent to the server.

[0048] Server: Re-parses the new request, generates a new proposal and sends it to the device.

[0049] The device converts the new suggestion into speech and relays it to the user. This process can be repeated until the user is satisfied.

[0050] Specific examples

[0051] Example 1: A scene where the user is in a supermarket

[0052] User: "What should I have for dinner tonight?"

[0053] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[0054] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[0055] Device: Converts to speech and tells the user, "How about grilled fish? We'll suggest salmon fillets, vegetables, and ingredients for miso soup."

[0056] User: "That's great, but is there anything easier?"

[0057] Terminal: Sends a new request to the server.

[0058] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[0059] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[0060] This series of steps allows users to efficiently decide on the appropriate menu at the supermarket and go shopping smoothly.

[0061] The present invention significantly reduces the effort and stress of daily meal planning decisions and provides a more efficient shopping experience.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] User: Launches the smartphone app and opens the account registration screen. Enters information such as name, number of family members, likes and dislikes, allergies, and health conditions.

[0065] Step 2:

[0066] Device: The entered information is temporarily stored in local storage.

[0067] Step 3:

[0068] Terminal: To send locally stored information to the server, the data is encrypted via a communication protocol and the transmission process is performed.

[0069] Step 4:

[0070] Server: Stores the received user information in a database.

[0071] Step 5:

[0072] User: Arrives at the supermarket, plugs in earphones, restarts the smartphone app, and selects voice input mode.

[0073] Step 6:

[0074] User: "What should I have for dinner tonight?" is spoken through the earphones.

[0075] Step 7:

[0076] Device: Converts the user's speech into text (performs speech recognition processing).

[0077] Step 8:

[0078] Terminal: Sends the converted text data to the server.

[0079] Step 9:

[0080] Server: Analyzes the received text data and references the user's pre-registered information (number of family members, likes and dislikes, allergies, and health status).

[0081] Step 10:

[0082] Server: Uses a generative AI model to generate appropriate menus, taking into account user preferences and pre-registration information.

[0083] Step 11:

[0084] Server: The generated menu information is sent to the terminal in text format.

[0085] Step 12:

[0086] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[0087] Step 13:

[0088] Device: Menu information is synthesized and transmitted to the user through earphones.

[0089] Step 14:

[0090] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[0091] Step 15:

[0092] Device: Converts re-inputted speech back into text.

[0093] Step 16:

[0094] Terminal: Re-send the converted text data to the server.

[0095] Step 17:

[0096] Server: Re-parses the new request data and generates a new menu.

[0097] Step 18:

[0098] Server: Sends new menu information in text format to the terminal.

[0099] Step 19:

[0100] Terminal: Converts the received text data back into speech.

[0101] Step 20:

[0102] Device: New menu information is synthesized and communicated to the user through earphones.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Until now, it has been difficult for users to provide suggestions that take into account individual requests, allergy information, preferences, etc. when deciding on daily menus. Furthermore, there has been no support system that allows users to decide on appropriate menus in real time while shopping at a supermarket, etc. This has led to problems such as stressful daily menu decisions and inefficient, time-consuming shopping.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes a speech recognition means for converting speech input by the user into text, a transmission means for transmitting the text data to the server, a generation means by which the server generates a menu based on the information input by the user and pre-registered information, a transmission means for transmitting the generated menu information to the terminal, a speech synthesis means for converting the received menu information into speech, an output means for conveying the synthesized speech information to the user, an input means for the user to input registered information, and a storage means for saving the input information. This enables users to receive real-time menu suggestions that take into account their individual requests, allergies, likes and dislikes while shopping at the supermarket.

[0108] "Speech recognition means" refers to a technical means that has the function of converting the voice input by the user into text data.

[0109] The "transmission means" is a technical means having the function of sending the converted text data to the server and transmitting the generated menu information to the terminal.

[0110] "Generation means" refers to a technical means that allows the server to generate a menu based on user input information and pre-registered information.

[0111] "Speech synthesis means" refers to a technical means that has the function of converting received text data into speech and conveying it to the user.

[0112] "Output means" refers to a technical means that has the function of conveying voice-synthesized information to the user.

[0113] "Input means" refers to a technical means that allows a user to input information required when creating an account (such as the number of family members, likes and dislikes, allergies, and health status).

[0114] "Storage means" refers to the technical means for storing the input information and making it available for reference in subsequent processes.

[0115] A "reference means" is a technical means that has the function of retrieving stored information when generating a menu and using it in the generation process.

[0116] A "generative AI model" is an artificial intelligence model used to generate appropriate menu suggestions based on user input and pre-registered information.

[0117] A "prompt" is an instruction or question that is input to a generative AI model to obtain an appropriate output.

[0118] The present invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. This system is realized through multiple processes, including voice recognition, data transmission, AI generation, voice synthesis and output. Specific embodiments are described below.

[0119] User Registration

[0120] Users download the smartphone app and tap the "Create a new account" button to begin creating an account. They then enter detailed information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is entered into the device via an input device.

[0121] The terminal temporarily stores the input information in local storage, attaches a security token, and transmits it to the server. The server records the received user information in a database using a storage means, and transmits a response indicating that the information has been saved to the terminal.

[0122] Voice input and request sending

[0123] The user arrives at the supermarket and connects the earphones to the device. Then, they launch the smartphone app, select voice input mode, and insert the earphones. In this state, they input a question such as, "What should I have for dinner tonight?"

[0124] The terminal receives the user's voice input and converts the voice into text using a voice recognition means. The specific technology used in this process is a technology known as a general voice recognition API, for example, Google (registered trademark) Cloud's Speech-to-Text API. The converted text data, "What should we have for dinner tonight?", is sent to the server via a transmission means.

[0125] Suggestion generation and voice response

[0126] The server uses a reference means to obtain the received text data and the user's pre-registered information, and generates an appropriate menu using a generative AI model. This generation means uses, for example, the OpenAI (registered trademark) GPT series. The server generates a suggestion such as "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and transmits this suggestion to the terminal via a transmission means.

[0127] The device converts the received suggested text into speech using a speech synthesis method, for example, Google Cloud's Text-to-Speech API, and the synthesized speech is communicated to the user through earphones: "How about grilled fish? We suggest salmon fillet, vegetables, and ingredients for miso soup."

[0128] Fix requests and iterations

[0129] If the user is not satisfied with the proposed menu, they can say, "Is there an easier dish?" again. The device will convert the speech to text again and send a new request, "Is there an easier dish?" to the server.

[0130] The server analyzes the new request, generates a new suggestion, "How about a quick chicken stir fry and salad?", and sends it to the device. The device converts the new suggestion from text to speech and tells the user, "How about a quick chicken stir fry and salad?" This process can be repeated until the user is satisfied.

[0131] Specific examples

[0132] Example 1: A scene where the user is in a supermarket

[0133] The user speaks, "What should I have for dinner tonight?" The device converts the speech into text and sends "What should I have for dinner tonight?" to the server.

[0134] Based on the user information, the server generates a request saying, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device. The device converts the suggestion into voice and conveys it to the user. If the user says, "That sounds good, but is there an easier dish?", the device sends a new request to the server, and the server generates, "Well, how about a simple chicken saute and salad?", which the device converts into voice again and conveys to the user. This process is repeated to determine a menu that satisfies the user.

[0135] Prompt Sentence Examples

[0136] 1. User prompt: "What should I have for dinner tonight?"

[0137] 2. User prompt: "Give me some suggestions for easier meals."

[0138] As described above, this system provides support for users to efficiently decide on appropriate menus in supermarkets and to carry out shopping smoothly.

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] Step 1: User Enters Information

[0141] Users download the smartphone app and tap the "Create a new account" button. They then enter information such as the number of family members, likes and dislikes, allergy information, and health status. The smartphone's touchscreen is used as the input method. The input data includes the number of family members, likes and dislikes, allergy information, and health status, and this data is temporarily stored on the device.

[0142] Step 2: Save the data to the server

[0143] The entered user information is sent from the terminal to the server. As a transmission method, the terminal's network function is used to attach a security token and send the data to the server. The server stores the received data in a database using a storage method. This allows the user information to be used later when generating menus.

[0144] Step 3: Dictation in Supermarket

[0145] The user arrives at the supermarket, plugs earphones into the device, and launches the app. They select the voice input mode and say, "What should I have for dinner tonight?" This speech is converted into text by a speech recognition tool. The input is the user's voice, and the output is text data.

[0146] Step 4: Send the audio data to the server

[0147] The terminal transmits text data to the server. The data received by the server is the text "What should we have for dinner tonight?" converted by the speech recognition means. The network function is used as the transmission means.

[0148] Step 5: The server generates the menu

[0149] The server retrieves the received text data and the user's pre-registered information through a reference method, and generates an appropriate menu using a generative AI model. The input is the text data and user information, and the output is a suggestion such as, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup."

[0150] Step 6: Send the generated proposal to your device

[0151] The generated menu information is sent from the server to the terminal. The network function is used as the transmission means. The generated menu information is output in text format.

[0152] Step 7: Convert the server's proposal to speech

[0153] The device converts the received text data into speech using a speech synthesis means. Google Cloud's Text-to-Speech API is used as an example of the speech synthesis means. The input is text data and the output is speech data.

[0154] Step 8: Communicate the generated audio to the user

[0155] The synthesized menu information is transmitted to the user through earphones, and the output is audio information that the user hears.

[0156] Step 9: User makes correction request

[0157] If the user is dissatisfied with the proposed menu, they can say, "Is there an easier dish?" This speech is again converted into text by the speech recognition means. The input is the user's voice, and the output is text data.

[0158] Step 10: Send a new request to the server

[0159] The terminal sends new text data to the server. The network function is used as the transmission method. The input is the text data, and the output is the transmission to the server.

[0160] Step 11: The server generates the menu again

[0161] The server analyzes the new request and generates new suggestions again using the generative AI model. The input is the text data of the new request, and the output is the new suggestion result.

[0162] Step 12: Send the new proposal to your device and convert it

[0163] The server sends the new suggestion to the device, which then converts it into speech using a speech synthesiser. The output is the new speech suggestion. This process is repeated until the suggestion is communicated to the user.

[0164] (Application example 1)

[0165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0166] Currently, when shopping in a brick-and-mortar store, users spend a great deal of time and effort deciding on an appropriate meal plan, taking into account individual factors such as the number of people in their family, their likes and dislikes, allergies, and health conditions. This process is inefficient and can be stressful, especially for users who find meal planning difficult. Furthermore, current technology lacks a system that provides real-time meal suggestions optimized for each user's individual needs, and there is a need to solve this problem.

[0167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0168] In this invention, the server includes a speech recognition means for converting user input speech into text, a transmission means for transmitting the text data to the server, a generation means for generating a menu based on the user input information and pre-registered information, a transmission means for transmitting the generated menu information to the terminal, a speech synthesis means for converting the received menu information into speech, an output means for conveying the speech-synthesized menu information to the user, and a prompt generation means for generating prompt sentences for the generative AI model using user profile information. This allows users to receive real-time menu suggestions optimized for their individual needs in physical stores, improving shopping efficiency and reducing stress.

[0169] A "speech recognition means" is a device or technology that converts user-supplied speech into text.

[0170] "Transmission means" refers to a device or technology for transmitting text data to a server.

[0171] "Generation means" refers to a device or technology for generating a menu based on user input information and pre-registered information.

[0172] The "voice synthesis means" is a device or technology for converting received menu information into voice.

[0173] "Output means" refers to a device or technology for conveying the synthesized menu information to the user.

[0174] A "prompt generator" is a device or technology that uses user profile information to generate prompt sentences for a generative AI model.

[0175] "User profile information" refers to information such as the number of people in a user's family, their likes and dislikes, allergies, and health status.

[0176] The system of the present invention is designed to enable users to decide on a menu while interacting with AI in real time while shopping in a physical store. This system is composed of multiple means, such as voice recognition, data transmission, menu generation, voice synthesis and output, and is equipped with a means for generating prompts using user profile information. Specific embodiments are described below.

[0177] Components

[0178] User Registration

[0179] User: Downloads the smartphone app and creates an account. When creating an account, the user enters information such as the number of people in their family, their likes and dislikes, allergies, and health conditions. This information is used to suggest more appropriate meals for future user queries.

[0180] Device: The entered information is temporarily stored in local storage and sent to the server.

[0181] Server: The received information is stored in a database, which is then referenced when creating a menu.

[0182] Voice input and request sending

[0183] User: Arrives at a physical store and connects earphones to the device. Launches the smartphone app and selects voice input mode. Enters a question: "What should I have for dinner tonight?"

[0184] Terminal: Converts the user's voice into text (speech recognition), and sends the converted text data to the server.

[0185] Server: Refers to the text data and pre-registered user profile information and generates an appropriate menu using a generative AI model. At this time, a prompt generation means generates a prompt sentence based on the user profile information.

[0186] Suggestion generation and voice response

[0187] Server: Sends the generated menu information in text format to the terminal.

[0188] Device: Converts received text data into speech (speech synthesis), and transmits the synthesized menu information to the user through earphones.

[0189] Fix requests and iterations

[0190] User: If the user is not satisfied with the proposed menu, they can say something like, "Is there an easier dish?" This speech is converted to text again and sent to the server.

[0191] Server: Re-parses the new request, generates a new proposal and sends it to the device.

[0192] The device converts the new suggestion into speech and relays it to the user. This process can be repeated until the user is satisfied.

[0193] Specific examples

[0194] A scene where the user is in a physical store

[0195] User: "What should I have for dinner tonight?"

[0196] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[0197] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[0198] Device: Converts to speech and tells the user, "How about grilled fish? We'll suggest salmon fillets, vegetables, and ingredients for miso soup."

[0199] User: "That's great, but is there anything easier?"

[0200] Terminal: Sends a new request to the server.

[0201] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[0202] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[0203] Prompt Sentence Examples

[0204] An example of a prompt used for a generative AI model is as follows:

[0205] The user's family consists of four people and they prefer Japanese food. They have an allergy to nuts and are seeking a low-fat diet due to their health condition.

[0206] Question: "What should I have for dinner tonight?"

[0207] Example response: "How about grilled fish? Suggestions include salmon fillet, vegetables, and miso soup ingredients."

[0208] This prompt is used as an instruction to ask the generative AI model to suggest appropriate menu items based on the user's background information and current question.

[0209] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0210] Step 1:

[0211] A user downloads a smartphone app and creates an account. When creating an account, they enter information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is saved as user profile information. The input comes from the user, and the output is user profile information saved on the device.

[0212] Step 2:

[0213] The terminal temporarily stores the user profile information in local storage and then transmits it to the server. The input is the user profile information and the output is transmission to the server. The terminal processes the data and transmits it to the server according to the communication protocol.

[0214] Step 3:

[0215] The server stores the received user profile information in a database. This information is referenced later when generating menus. The input is the user profile information sent from the terminal, and the output is the information stored in the database.

[0216] Step 4:

[0217] The user arrives at the store and connects earphones to the device. They launch the smartphone app and select voice input mode. They then speak a question, such as "What should I have for dinner tonight?" The input is the user's voice, and the output is the start of voice recognition.

[0218] Step 5:

[0219] The device converts the user's voice into text. Using a speech recognition method (e.g., Google's speech recognition service), the voice data is converted into text data. The input is the user's voice, and the output is text data.

[0220] Step 6:

[0221] The terminal transmits the converted text data to the server. The transmission means is used to transmit the data securely to the server in accordance with the transmission protocol. The input is the converted text data, and the output is transmission to the server.

[0222] Step 7:

[0223] The server references the text data along with pre-registered user profile information, providing the basic data for generating an appropriate menu. The input is the text data and user profile information, and the output is analysis preparation based on the profile.

[0224] Step 8:

[0225] The server uses the prompt generation means to generate a prompt sentence based on the user profile information. The prompt sentence to be used for the generative AI model is generated. The input is the user profile information and the question text data, and the output is the generated prompt sentence.

[0226] Step 9:

[0227] The server uses a generative AI model to generate an appropriate menu based on the prompt. The generated menu information is sent from the server to the device in text format. The input is the prompt, and the output is the generated menu information.

[0228] Step 10:

[0229] The device converts the text data received into speech. It uses a speech synthesis tool (e.g., the pyttsx3 library) to convert the text data into speech, and transmits the generated speech to the user through headphones. The input is the text data received from the server, and the output is a synthesized speech log.

[0230] Step 11:

[0231] If the user is dissatisfied with the proposed menu, they can input a second voice request, such as "Is there an easier dish?" This voice request is converted back into text and sent to the server. The input is a voice request from the user, and the output is again text data.

[0232] Step 12:

[0233] The server re-analyzes the new request, generates new suggestions, and sends them to the terminal. This process is repeated until the user is satisfied. The input is the text data of the new request, and the output is the newly generated menu information.

[0234] This series of steps allows users to efficiently decide on the right menu in a physical store. Specifically, a system is provided in which voice input, text conversion, data transmission, prompt generation, AI menu generation, and voice output all work together seamlessly.

[0235] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0236] This invention is a system that allows users to consult with AI in real time while shopping at the supermarket to decide on a menu, and by combining it with an emotion engine, it is possible to realize menu suggestions that take the user's emotions into consideration. This system is composed of multiple means, such as voice recognition, data transmission, AI generation, voice synthesis, emotion recognition and output. Specific embodiments are described below.

[0237] User Registration

[0238] User: Downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, likes and dislikes, allergies, health conditions, etc. This information is used to suggest more appropriate meals for future user queries.

[0239] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[0240] Server: Stores the received user information in a database. This stored information is referenced later when generating menus.

[0241] Voice input and request sending

[0242] User: Arrives at the supermarket and connects earphones to the device. Launches the smartphone app again and selects voice input mode. Voice input of a question through the earphones: "What should I have for dinner tonight?"

[0243] Device: Converts the user's speech into text (performs speech recognition processing).

[0244] Terminal: Sends the converted text data to the server.

[0245] Server: Along with the text data, the server references the user's pre-registered information (number of family members, likes and dislikes, allergies, and health status).

[0246] Emotion Recognition and Suggestion Generation

[0247] Server: Analyzes emotional information from voice data using emotion recognition tools, detects the user's emotional state, and makes appropriate suggestions based on that.

[0248] Server: Using a generative AI model, it generates an appropriate menu, taking into account the user's input information, pre-registered information, and detected emotional state.

[0249] Server: The generated menu information is sent to the terminal in text format.

[0250] Suggestion generation and voice response

[0251] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[0252] Device: Menu information is synthesized and transmitted to the user through earphones.

[0253] Fix requests and iterations

[0254] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[0255] Device: Converts re-inputted speech back into text.

[0256] Terminal: Re-send the converted text data to the server.

[0257] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[0258] Server: Sends new menu information in text format to the terminal.

[0259] Terminal: Converts the received text data back into speech.

[0260] Device: New menu information is synthesized and communicated to the user through earphones.

[0261] Specific examples

[0262] Example 1: A scene where the user is in a supermarket

[0263] User: "What should I have for dinner tonight?"

[0264] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[0265] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[0266] Server: Analyzes the user's emotions from their voice and, if it determines that the user is a little tired, adjusts the suggestions to say, "How about some easy grilled fish today?"

[0267] Device: Converts into speech and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[0268] User: "That's great, but is there anything easier?"

[0269] Terminal: Sends a new request to the server.

[0270] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[0271] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[0272] This series of steps allows users to efficiently decide on the appropriate menu at the supermarket and make their shopping experience smoother. In addition, by combining it with emotion recognition, it becomes possible to make suggestions that take into account the user's mental state, providing even greater convenience.

[0273] The processing flow will be explained below.

[0274] Step 1:

[0275] User: Download the smartphone app and open the account registration screen. Enter your name, number of family members, likes and dislikes, allergies, health conditions, etc.

[0276] Step 2:

[0277] Device: The entered information is temporarily stored in local storage.

[0278] Step 3:

[0279] Terminal: To send locally stored information to the server, a communication protocol is used to securely encrypt the data and process the transmission.

[0280] Step 4:

[0281] Server: Stores the received user information in a database.

[0282] Step 5:

[0283] User: Arrives at the supermarket, plugs in earphones, restarts the smartphone app, and selects voice input mode.

[0284] Step 6:

[0285] User: "What should I have for dinner tonight?" is spoken through the earphones.

[0286] Step 7:

[0287] Device: Converts the user's speech into text (performs speech recognition processing).

[0288] Step 8:

[0289] Terminal: Sends the converted text data to the server.

[0290] Step 9:

[0291] Server: Analyzes the text data and references the user's pre-registered information (number of family members, likes and dislikes, allergies, health status).

[0292] Step 10:

[0293] Server: Analyzes the user's emotional information from the voice data using emotion recognition means.

[0294] Step 11:

[0295] Server: Uses a generative AI model to generate appropriate menus based on the user's input information, pre-registration information, and detected emotional state.

[0296] Step 12:

[0297] Server: The generated menu information is sent to the terminal in text format.

[0298] Step 13:

[0299] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[0300] Step 14:

[0301] Device: Menu information is synthesized and transmitted to the user through earphones.

[0302] Step 15:

[0303] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[0304] Step 16:

[0305] Device: Converts re-inputted speech back into text.

[0306] Step 17:

[0307] Terminal: Re-send the converted text data to the server.

[0308] Step 18:

[0309] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[0310] Step 19:

[0311] Server: Sends new menu information in text format to the terminal.

[0312] Step 20:

[0313] Terminal: Converts the received text data back into speech.

[0314] Step 21:

[0315] Device: New menu information is synthesized and communicated to the user through earphones.

[0316] Example 2

[0317] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0318] Conventional menu suggestion systems typically convert a user's voice input into text, then send the text data to a server to generate a menu. However, these systems do not take into account the user's emotional state or detailed personal information registered in advance, making it difficult to suggest an optimal menu that matches the user's situation and emotions. Furthermore, the process of modifying the menu in response to user requests tends to be cumbersome, resulting in a lack of convenience.

[0319] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0320] In this invention, the server includes a means for generating a menu based on user input information, pre-registered information, and emotional information, a means for converting the received menu information into audio, and an output means for conveying the information to the user. This allows for optimal menu suggestions that take into account the user's emotional state and pre-registered information. Furthermore, the system allows for quick and flexible menu revisions in response to user requests, greatly improving user convenience.

[0321] "Speech recognition means" is a technology that analyzes the voice input by the user and converts it into text data.

[0322] The "transmission means" is a technology for transmitting the text data converted by the voice recognition means to the server.

[0323] The "generation means" is a technology that generates an appropriate menu based on the received user input information, pre-registered information, and emotional information.

[0324] The "transmission means" (the means for transmitting the generated menu information) is a technique for transmitting the generated menu information to the terminal.

[0325] The "voice synthesis means" is a technology that converts received menu information into voice format.

[0326] The "output means" is a technology for transmitting the voice generated by the voice synthesis means to the user.

[0327] The "storage means" is a technology that stores pre-registered information such as the number of family members, likes and dislikes, allergy information, and health status in a database.

[0328] The "reference means" is a technology that obtains necessary information from the storage means when generating a menu and provides it to the generation means.

[0329] "Emotional information" is data that indicates the user's emotional state, analyzed based on the user's voice and other input data.

[0330] This invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. It also combines an emotion engine to realize menu suggestions that take the user's emotions into consideration. This system is composed of multiple means, including voice recognition, data transmission, AI generation, voice synthesis, emotion recognition and output. Specific embodiments are shown below.

[0331] Hardware and Software Configuration

[0332] Users access the system using devices such as smartphones and tablets, which have applications installed for voice recognition, text data transmission, and voice data output.

[0333] The device uses the Google Cloud Speech-to-Text API for speech recognition, parses the voice input, and converts it into text data, which is then converted into JSON format and sent to a server over the internet.

[0334] The server is hosted in a cloud environment and is responsible for performing multiple functions. The server runs a generative AI model (e.g., GPT-3®) to generate appropriate menus based on text data sent by users and pre-registered user information. It also uses an emotion engine (e.g., IBM Watson® Tone Analyzer) to analyze the user's emotional state.

[0335] The generated menu information is sent from the server to the device in JSON format. The device receives this JSON data and converts it into text format. Next, it uses the Google Cloud Text-to-Speech API for speech synthesis to convert the text data into audio data. The converted audio data is played back to the user through headphones.

[0336] Specific examples

[0337] Example 1: A scene where the user is in a supermarket

[0338] The user speaks into the smartphone's microphone, asking, "What should I have for dinner tonight?"

[0339] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[0340] After analyzing the user information, the server generates a message saying, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the terminal.

[0341] The server analyzes the user's emotions from their voice, and if it determines that the user is a little tired, it adjusts its suggestions, saying, "How about some easy grilled fish today?"

[0342] The device converts the information into voice and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[0343] The user makes another request, saying, "That's good, but is there anything a little easier?"

[0344] The terminal sends a new request to the server.

[0345] The server generates a new suggestion and sends it to the device: "How about a quick chicken stir fry and salad?"

[0346] The device converts the information into speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[0347] Prompt Sentence Examples

[0348] User information: "Name: Sato, Family size: 3, Likes and dislikes: Likes spicy food, Allergies: None, Health condition: Good, Emotions: A little tired"

[0349] User Question: "What should I have for dinner tonight?"

[0350] Suggested menu:

[0351] Step 1: "How about making a simple grilled fish dish for dinner tonight? We suggest a salmon fillet, vegetables, and miso soup."

[0352] Step 2: "If grilled fish seems too difficult, how about sautéed chicken and a salad?"

[0353] This allows users to efficiently decide on the appropriate menu at the supermarket and make shopping a smoother experience. In addition, by combining emotion recognition, suggestions can be made that take into account the user's mental state, providing even greater convenience.

[0354] The above is a specific embodiment for carrying out the present invention.

[0355] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0356] Step 1: User Registration

[0357] The user downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, likes and dislikes, allergies, health conditions, etc. This information is then entered into the app's input form. Once the information is complete, the user presses the "Register" button.

[0358] Input: User's personal information (name, number of family members, likes and dislikes, allergies, health status)

[0359] Output: User information data in JSON format

[0360] The device temporarily saves the input information in local storage and sends it to the server. Specifically, the input data is converted to JSON format and sent to the server via an HTTP POST request.

[0361] Input: JSON format user information data

[0362] Output: HTTP request sent to the server

[0363] The server saves the received user information in the database. The server parses the received JSON data and stores the necessary information in the appropriate table in the database.

[0364] Input: JSON data sent in the HTTP request to the server

[0365] Output: User information record in database

[0366] Step 2: Speak and send your request

[0367] The user arrives at the supermarket, connects the earphones to the device, restarts the smartphone app, selects voice input mode, and inputs a question through the earphones: "What should I have for dinner tonight?"

[0368] Input: User's voice data

[0369] Output: Audio data is input to the device

[0370] The device converts the user's voice into text using a speech recognition API (e.g., Google Cloud Speech-to-Text), sends the voice data to the API, and retrieves the converted text data.

[0371] Input: Audio data

[0372] Output: Text data converted from audio

[0373] The device sends the converted text data to the server, where it is converted to JSON format along with the user ID and sent via an HTTP POST request to the server.

[0374] Input: Text data, user ID

[0375] Output: HTTP request sent to the server

[0376] The server retrieves the user's pre-registered information (number of family members, likes and dislikes, allergies, health status) and emotional information from a database to generate a menu based on the text data and pre-registered information.

[0377] Input: Text data, user ID

[0378] Output: User details (number of family members, likes and dislikes, allergies, health status)

[0379] Step 3: Emotion recognition and suggestion generation

[0380] The server uses the acquired voice data to perform emotion recognition. It sends the voice data to an emotion recognition API (e.g., IBM Watson Tone Analyzer) and analyzes the user's emotional state.

[0381] Input: Audio data

[0382] Output: Emotional information (e.g., "tired")

[0383] The server uses a generative AI model (e.g., GPT-3) to generate an appropriate menu based on the user's input, pre-registered information, and the detected emotional state. The prompt text includes the user's name, number of family members, likes and dislikes, allergy information, health status, emotional state, and voice input text and is input into the model.

[0384] Input: User input information, pre-registration information, emotional information

[0385] Output: Generated menu information (text format)

[0386] The server converts the generated menu information into JSON format and sends it to the terminal as an HTTP response.

[0387] Input: Generated menu information

[0388] Output: HTTP response sent to the device

[0389] Step 4: Proposal generation and voice response

[0390] The device converts the received text data into speech using a speech synthesis API (e.g., Google Cloud Text-to-Speech). The device sends the text data to the API and receives the speech data.

[0391] Input: Received text data

[0392] Output: Synthesized voice data

[0393] The device transmits the menu information to the user through a headset, and by outputting the audio data to the headset, the user can receive suggestions by voice.

[0394] Input: Synthesized voice data

[0395] Output: Audio output to the user

[0396] Step 5: Request revisions and iterate

[0397] If the user is dissatisfied with the proposed menu, he or she can again input a voice command such as "Is there an easier dish?"

[0398] Input: User's voice data

[0399] Output: Audio data is input to the device

[0400] The device again uses the speech recognition API to convert the speech to text, sends the speech data to the API, and retrieves the converted text data.

[0401] Input: Audio data

[0402] Output: Text data converted from audio

[0403] The device sends the converted text data to the server again, converts the text data into JSON format, and sends it to the server via an HTTP POST request.

[0404] Input: Text data, user ID

[0405] Output: HTTP request sent to the server

[0406] The server reanalyzes the new request data and generates a new menu. The data is input into the generative AI model, which generates new suggestions. The server also reanalyzes changes in the user's emotions and optimizes the suggestions.

[0407] Input: New text data, pre-registration information, emotion information

[0408] Output: New menu information (text format)

[0409] The server converts the new menu information back into JSON format and sends it to the terminal as an HTTP response.

[0410] Input: New menu information

[0411] Output: HTTP response sent to the device

[0412] The device converts the received text data into speech using the speech synthesis API. The device sends the text data to the API and obtains the speech data.

[0413] Input: Received text data

[0414] Output: Synthesized voice data

[0415] The device then sends the new menu information via voice synthesis to the user through earphones, allowing the user to receive new suggestions by voice by outputting the audio data to the earphones.

[0416] Input: Synthesized voice data

[0417] Output: Audio output to the user

[0418] (Application example 2)

[0419] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0420] Existing menu suggestion systems can suggest general menus based on user input, but they are unable to consider the user's mental state or emotions. This makes it difficult to provide optimal menus when the user is tired or in a particular mood, potentially reducing user satisfaction. They also lack the flexibility to respond to re-suggestions. Therefore, there is a need for a system that can suggest menus that take the user's emotional state into account and improve the user's mental satisfaction.

[0421] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion analysis means that uses an emotion recognition means to analyze the user's emotional state and optimize menu suggestions based on that analysis, a voice synthesis means that audibly conveys the generated menu information to the user, and a re-suggestion means that generates new menu information in response to a user's request for revision and conveys it to the user again. This makes it possible to propose an optimal menu that takes the user's emotional state into consideration, thereby improving psychological satisfaction.

[0422] "Speech recognition means" refers to technology that converts the voice input by the user into text.

[0423] "Transmission means" refers to a technique for transmitting text data to a server.

[0424] The "generation means" is a technology that allows the server to generate a menu based on information entered by the user and information registered in advance.

[0425] The "voice synthesis means" is a technology for converting the generated menu information into voice.

[0426] "Output means" refers to technology that conveys menu information to users in the form of synthesized voice.

[0427] "Emotion recognition means" is a technology that analyzes the user's emotional state.

[0428] "Emotion analysis means" is a technology that uses emotion recognition means to analyze the user's emotional state and optimizes menu suggestions based on that.

[0429] "Storage means" refers to the technology that stores information such as the number of family members, likes and dislikes, allergy information, and health status in a database.

[0430] The "reference means" is a technique for obtaining information from the storage means when creating a menu.

[0431] The "means for generating a prompt sentence" is a technique for generating a prompt sentence based on the emotional state of the user.

[0432] This invention is a system that allows users to decide on a menu while shopping in a physical store, consulting with AI in real time. By incorporating emotion recognition, it is possible to make suggestions that take into account the user's mental state. This system is configured as follows.

[0433] User Registration

[0434] User: Downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, preferences, allergy information, health conditions, etc. This information is used to suggest more appropriate meals for future user queries.

[0435] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[0436] Server: Stores the received user information in a database. This stored information is referenced later when generating menus.

[0437] Voice input and request sending

[0438] User: Arrives at the store and connects the earphones to the device. Relaunches the smartphone app and selects voice input mode. Voice input of a question through the earphones: "What should I have for dinner tonight?"

[0439] On the device: Converts the user's speech into text. The "SpeechRecognition" library is used to implement the speech recognition method.

[0440] Terminal: Sends the converted text data to the server.

[0441] Emotion Recognition and Suggestion Generation

[0442] Server: Analyzes emotional information from voice data using emotion recognition. Detects the user's emotional state and makes appropriate suggestions based on that. A "generative AI model" is used to generate an appropriate menu, taking into account the user's input information, pre-registered information, and the detected emotional state.

[0443] Server: The generated menu information is sent to the terminal in text format.

[0444] Suggestion generation and voice response

[0445] Terminal: Converts received text data into speech. The "gTTS" library is used to perform text-to-speech synthesis processing.

[0446] Device: Menu information is synthesized and transmitted to the user through earphones.

[0447] Fix requests and iterations

[0448] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[0449] Terminal: Converts the re-inputted voice into text and sends it back to the server.

[0450] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[0451] Server: Sends new menu information to the terminal.

[0452] Terminal: Converts received text data into speech.

[0453] Device: New menu information is synthesized and communicated to the user through earphones.

[0454] Specific examples

[0455] The following is the sequence of events when a user is in a physical store:

[0456] User: "What should I have for dinner tonight?"

[0457] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[0458] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[0459] Server: Analyzes the user's emotions from their voice and, if it determines that the user is a little tired, adjusts the suggestions to say, "How about some easy grilled fish today?"

[0460] Device: Converts into speech and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[0461] User: "That's great, but is there anything easier?"

[0462] Terminal: Sends a new request to the server.

[0463] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[0464] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[0465] Prompt Sentence Examples

[0466] "What should I have for dinner tonight?"

[0467] By implementing this, users can efficiently decide on the appropriate menu in a physical store and shop smoothly. In addition, by combining it with emotion recognition, it becomes possible to make suggestions that take into account the user's mental state, providing even greater convenience.

[0468] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0469] Step 1:

[0470] A user downloads the smartphone app and opens the account registration screen. They enter information such as their name, number of family members, preferences, allergies, and health conditions into a form within the app. Based on this information, the device temporarily saves it in local storage and then sends it to the server. The server then saves the received user information in a database.

[0471] Step 2:

[0472] The user arrives at the store and connects the earphones to their smartphone. Next, they launch the smartphone app and select voice input mode. The user then speaks a question into the earphones, such as "What should I have for dinner tonight?" The voice data is then imported into the smartphone device.

[0473] Step 3:

[0474] The device uses the SpeechRecognition library to convert the captured voice data into text. Specifically, the voice data is input into an acoustic model, and patterns that match the phonemes are analyzed to generate text data. This converted text data is then sent to the server.

[0475] Step 4:

[0476] The server references the received text data and pre-registered user information (family size, likes and dislikes, allergies, and health status). It then uses emotion recognition to analyze the user's emotional state from the voice data. A generative AI model generates an optimal menu plan that takes into account the user's input information and emotional state.

[0477] Step 5:

[0478] The server then sends the suggestions obtained from the generative AI model in text format to the device, where the suggestions are adjusted, such as suggesting less strenuous meals if the emotional state is evaluated as "tired."

[0479] Step 6:

[0480] The device uses the gTTS library to convert the received text data into voice data. During this process, a synthetic speech engine analyzes the text data phonetically and generates a natural speaking voice. The generated voice data is transmitted to the user through earphones.

[0481] Step 7:

[0482] If the user is dissatisfied with the proposed menu, they can input a question by voice again, such as "Is there an easier dish?" The new voice data is then imported into the device.

[0483] Step 8:

[0484] The device then uses the SpeechRecognition library to convert the newly input voice data into text, which is then sent back to the server.

[0485] Step 9:

[0486] The server re-analyzes the new request data and generates new menu suggestions. Using the generative AI model and sentiment analysis, it re-measures the user's latest emotional state and optimizes the suggestions accordingly. The new suggestions are generated and sent to the device.

[0487] Step 10:

[0488] The device then uses the gTTS library again to convert the new text suggestions into audio, which is then transmitted to the user through the earphones.

[0489] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0490] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0491] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0492] [Second embodiment]

[0493] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0494] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0495] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0496] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0497] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0498] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0499] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0500] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0501] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0502] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0503] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0504] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0505] The present invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. This system is composed of multiple means such as voice recognition, data transmission, AI generation, voice synthesis, and output. Specific embodiments are described below.

[0506] User Registration

[0507] User: Downloads the smartphone app and creates an account. When creating an account, the user enters information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is used to suggest more appropriate meals for future user inquiries.

[0508] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[0509] Server: Stores the received information in a database. This stored information is referenced later when generating menus.

[0510] Voice input and request sending

[0511] User: Arrives at the supermarket and connects earphones to the device. Launches the smartphone app and selects voice input mode. Enters a question: "What should I have for dinner tonight?"

[0512] Terminal: Converts the user's voice into text (voice recognition), and sends the converted text data to the server.

[0513] Server: Refers to the text data and the user's pre-registered information to generate an appropriate menu. This is done using a generative AI model.

[0514] Suggestion generation and voice response

[0515] Server: The generated menu information is sent to the terminal in text format.

[0516] Terminal: Converts received text data into speech (speech synthesis), and transmits the synthesized menu information to the user through earphones.

[0517] Fix requests and iterations

[0518] User: If the user is not satisfied with the proposed menu, they can say something like, "Is there an easier dish?" This speech is converted into text again and sent to the server.

[0519] Server: Re-parses the new request, generates a new proposal and sends it to the device.

[0520] The device converts the new suggestion into speech and relays it to the user. This process can be repeated until the user is satisfied.

[0521] Specific examples

[0522] Example 1: A scene where the user is in a supermarket

[0523] User: "What should I have for dinner tonight?"

[0524] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[0525] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[0526] Device: Converts to speech and tells the user, "How about grilled fish? We'll suggest salmon fillets, vegetables, and ingredients for miso soup."

[0527] User: "That's great, but is there anything easier?"

[0528] Terminal: Sends a new request to the server.

[0529] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[0530] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[0531] This series of steps allows users to efficiently decide on the appropriate menu at the supermarket and go shopping smoothly.

[0532] The present invention significantly reduces the effort and stress of daily meal planning decisions and provides a more efficient shopping experience.

[0533] The processing flow will be explained below.

[0534] Step 1:

[0535] User: Launches the smartphone app and opens the account registration screen. Enters information such as name, number of family members, likes and dislikes, allergies, and health conditions.

[0536] Step 2:

[0537] Device: The entered information is temporarily stored in local storage.

[0538] Step 3:

[0539] Terminal: To send locally stored information to the server, the data is encrypted via a communication protocol and the transmission process is performed.

[0540] Step 4:

[0541] Server: Stores the received user information in a database.

[0542] Step 5:

[0543] User: Arrives at the supermarket, plugs in earphones, restarts the smartphone app, and selects voice input mode.

[0544] Step 6:

[0545] User: "What should I have for dinner tonight?" is spoken through the earphones.

[0546] Step 7:

[0547] Device: Converts the user's speech into text (performs speech recognition processing).

[0548] Step 8:

[0549] Terminal: Sends the converted text data to the server.

[0550] Step 9:

[0551] Server: Analyzes the received text data and references the user's pre-registered information (number of family members, likes and dislikes, allergies, and health status).

[0552] Step 10:

[0553] Server: Uses a generative AI model to generate appropriate menus, taking into account user preferences and pre-registration information.

[0554] Step 11:

[0555] Server: The generated menu information is sent to the terminal in text format.

[0556] Step 12:

[0557] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[0558] Step 13:

[0559] Device: Menu information is synthesized and transmitted to the user through earphones.

[0560] Step 14:

[0561] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[0562] Step 15:

[0563] Device: Converts re-inputted speech back into text.

[0564] Step 16:

[0565] Terminal: Re-send the converted text data to the server.

[0566] Step 17:

[0567] Server: Re-parses the new request data and generates a new menu.

[0568] Step 18:

[0569] Server: Sends new menu information in text format to the terminal.

[0570] Step 19:

[0571] Terminal: Converts the received text data back into speech.

[0572] Step 20:

[0573] Device: New menu information is synthesized and communicated to the user through earphones.

[0574] Example 1

[0575] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0576] Until now, it has been difficult for users to provide suggestions that take into account individual requests, allergy information, preferences, etc. when deciding on daily menus. Furthermore, there has been no support system that allows users to decide on appropriate menus in real time while shopping at a supermarket, etc. This has led to problems such as stressful daily menu decisions and inefficient, time-consuming shopping.

[0577] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0578] In this invention, the server includes a speech recognition means for converting speech input by the user into text, a transmission means for transmitting the text data to the server, a generation means by which the server generates a menu based on the information input by the user and pre-registered information, a transmission means for transmitting the generated menu information to the terminal, a speech synthesis means for converting the received menu information into speech, an output means for conveying the synthesized speech information to the user, an input means for the user to input registered information, and a storage means for saving the input information. This enables users to receive real-time menu suggestions that take into account their individual requests, allergies, likes and dislikes while shopping at the supermarket.

[0579] "Speech recognition means" refers to a technical means that has the function of converting the voice input by the user into text data.

[0580] The "transmission means" is a technical means having the function of sending the converted text data to the server and transmitting the generated menu information to the terminal.

[0581] "Generation means" refers to a technical means that allows the server to generate a menu based on user input information and pre-registered information.

[0582] "Speech synthesis means" refers to a technical means that has the function of converting received text data into speech and conveying it to the user.

[0583] "Output means" refers to a technical means that has the function of conveying voice-synthesized information to the user.

[0584] "Input means" refers to a technical means that allows a user to input information required when creating an account (such as the number of family members, likes and dislikes, allergies, and health status).

[0585] "Storage means" refers to the technical means for storing the input information and making it available for reference in subsequent processes.

[0586] A "reference means" is a technical means that has the function of retrieving stored information when generating a menu and using it in the generation process.

[0587] A "generative AI model" is an artificial intelligence model used to generate appropriate menu suggestions based on user input and pre-registered information.

[0588] A "prompt" is an instruction or question that is input to a generative AI model to obtain an appropriate output.

[0589] The present invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. This system is realized through multiple processes, including voice recognition, data transmission, AI generation, voice synthesis and output. Specific embodiments are described below.

[0590] User Registration

[0591] Users download the smartphone app and tap the "Create a new account" button to begin creating an account. They then enter detailed information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is entered into the device via an input device.

[0592] The terminal temporarily stores the input information in local storage, attaches a security token, and transmits it to the server. The server records the received user information in a database using a storage means, and transmits a response indicating that the information has been saved to the terminal.

[0593] Voice input and request sending

[0594] The user arrives at the supermarket and connects the earphones to the device. Then, they launch the smartphone app, select voice input mode, and insert the earphones. In this state, they input a question such as, "What should I have for dinner tonight?"

[0595] The device receives the user's voice input and converts the voice into text using a speech recognition means. The specific technology used in this process is known as a general speech recognition API, for example, Google Cloud's Speech-to-Text API. The converted text data, "What should we have for dinner tonight?", is sent to the server via a transmission means.

[0596] Suggestion generation and voice response

[0597] The server uses a reference means to obtain the received text data and the user's pre-registered information, and generates an appropriate menu using a generative AI model. This generation means uses, for example, OpenAI's GPT series. The server generates a suggestion such as "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and sends this suggestion to the terminal via a transmission means.

[0598] The device converts the received suggested text into speech using a speech synthesis method, for example, Google Cloud's Text-to-Speech API, and the synthesized speech is communicated to the user through earphones: "How about grilled fish? We suggest salmon fillet, vegetables, and ingredients for miso soup."

[0599] Fix requests and iterations

[0600] If the user is not satisfied with the proposed menu, they can say, "Is there an easier dish?" again. The device will convert the speech to text again and send a new request, "Is there an easier dish?" to the server.

[0601] The server analyzes the new request, generates a new suggestion, "How about a quick chicken stir fry and salad?", and sends it to the device. The device converts the new suggestion from text to speech and tells the user, "How about a quick chicken stir fry and salad?" This process can be repeated until the user is satisfied.

[0602] Specific examples

[0603] Example 1: A scene where the user is in a supermarket

[0604] The user speaks, "What should I have for dinner tonight?" The device converts the speech into text and sends "What should I have for dinner tonight?" to the server.

[0605] Based on the user information, the server generates a request saying, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device. The device converts the suggestion into voice and conveys it to the user. If the user says, "That sounds good, but is there an easier dish?", the device sends a new request to the server, and the server generates, "Well, how about a simple chicken saute and salad?", which the device converts into voice again and conveys to the user. This process is repeated to determine a menu that satisfies the user.

[0606] Prompt Sentence Examples

[0607] 1. User prompt: "What should I have for dinner tonight?"

[0608] 2. User prompt: "Give me some suggestions for easier meals."

[0609] As described above, this system provides support for users to efficiently decide on appropriate menus in supermarkets and to carry out shopping smoothly.

[0610] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0611] Step 1: User Enters Information

[0612] Users download the smartphone app and tap the "Create a new account" button. They then enter information such as the number of family members, likes and dislikes, allergy information, and health status. The smartphone's touchscreen is used as the input method. The input data includes the number of family members, likes and dislikes, allergy information, and health status, and this data is temporarily stored on the device.

[0613] Step 2: Save the data to the server

[0614] The entered user information is sent from the terminal to the server. As a transmission method, the terminal's network function is used to attach a security token and send the data to the server. The server stores the received data in a database using a storage method. This allows the user information to be used later when generating menus.

[0615] Step 3: Dictation in Supermarket

[0616] The user arrives at the supermarket, plugs earphones into the device, and launches the app. They select the voice input mode and say, "What should I have for dinner tonight?" This speech is converted into text by a speech recognition tool. The input is the user's voice, and the output is text data.

[0617] Step 4: Send the audio data to the server

[0618] The terminal transmits text data to the server. The data received by the server is the text "What should we have for dinner tonight?" converted by the speech recognition means. The network function is used as the transmission means.

[0619] Step 5: The server generates the menu

[0620] The server retrieves the received text data and the user's pre-registered information through a reference method, and generates an appropriate menu using a generative AI model. The input is the text data and user information, and the output is a suggestion such as, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup."

[0621] Step 6: Send the generated proposal to your device

[0622] The generated menu information is sent from the server to the terminal. The network function is used as the transmission means. The generated menu information is output in text format.

[0623] Step 7: Convert the server's proposal to speech

[0624] The device converts the received text data into speech using a speech synthesis means. Google Cloud's Text-to-Speech API is used as an example of the speech synthesis means. The input is text data and the output is speech data.

[0625] Step 8: Communicate the generated audio to the user

[0626] The synthesized menu information is transmitted to the user through earphones, and the output is audio information that the user hears.

[0627] Step 9: User makes correction request

[0628] If the user is dissatisfied with the proposed menu, they can say, "Is there an easier dish?" This speech is again converted into text by the speech recognition means. The input is the user's voice, and the output is text data.

[0629] Step 10: Send a new request to the server

[0630] The terminal sends new text data to the server. The network function is used as the transmission method. The input is the text data, and the output is the transmission to the server.

[0631] Step 11: The server generates the menu again

[0632] The server analyzes the new request and generates new suggestions again using the generative AI model. The input is the text data of the new request, and the output is the new suggestion result.

[0633] Step 12: Send the new proposal to your device and convert it

[0634] The server sends the new suggestion to the device, which then converts it into speech using a speech synthesiser. The output is the new speech suggestion. This process is repeated until the suggestion is communicated to the user.

[0635] (Application example 1)

[0636] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0637] Currently, when shopping in a brick-and-mortar store, users spend a great deal of time and effort deciding on an appropriate meal plan, taking into account individual factors such as the number of people in their family, their likes and dislikes, allergies, and health conditions. This process is inefficient and can be stressful, especially for users who find meal planning difficult. Furthermore, current technology lacks a system that provides real-time meal suggestions optimized for each user's individual needs, and there is a need to solve this problem.

[0638] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0639] In this invention, the server includes a speech recognition means for converting user input speech into text, a transmission means for transmitting the text data to the server, a generation means for generating a menu based on the user input information and pre-registered information, a transmission means for transmitting the generated menu information to the terminal, a speech synthesis means for converting the received menu information into speech, an output means for conveying the speech-synthesized menu information to the user, and a prompt generation means for generating prompt sentences for the generative AI model using user profile information. This allows users to receive real-time menu suggestions optimized for their individual needs in physical stores, improving shopping efficiency and reducing stress.

[0640] A "speech recognition means" is a device or technology that converts user-supplied speech into text.

[0641] "Transmission means" refers to a device or technology for transmitting text data to a server.

[0642] "Generation means" refers to a device or technology for generating a menu based on user input information and pre-registered information.

[0643] The "voice synthesis means" is a device or technology for converting received menu information into voice.

[0644] "Output means" refers to a device or technology for conveying the synthesized menu information to the user.

[0645] A "prompt generator" is a device or technology that uses user profile information to generate prompt sentences for a generative AI model.

[0646] "User profile information" refers to information such as the number of people in a user's family, their likes and dislikes, allergies, and health status.

[0647] The system of the present invention is designed to enable users to decide on a menu while interacting with AI in real time while shopping in a physical store. This system is composed of multiple means, such as voice recognition, data transmission, menu generation, voice synthesis and output, and is equipped with a means for generating prompts using user profile information. Specific embodiments are described below.

[0648] Components

[0649] User Registration

[0650] User: Downloads the smartphone app and creates an account. When creating an account, the user enters information such as the number of people in their family, their likes and dislikes, allergies, and health conditions. This information is used to suggest more appropriate meals for future user queries.

[0651] Device: The entered information is temporarily stored in local storage and sent to the server.

[0652] Server: The received information is stored in a database, which is then referenced when creating a menu.

[0653] Voice input and request sending

[0654] User: Arrives at a physical store and connects earphones to the device. Launches the smartphone app and selects voice input mode. Enters a question: "What should I have for dinner tonight?"

[0655] Terminal: Converts the user's voice into text (speech recognition), and sends the converted text data to the server.

[0656] Server: Refers to the text data and pre-registered user profile information and generates an appropriate menu using a generative AI model. At this time, a prompt generation means generates a prompt sentence based on the user profile information.

[0657] Suggestion generation and voice response

[0658] Server: Sends the generated menu information in text format to the terminal.

[0659] Device: Converts received text data into speech (speech synthesis), and transmits the synthesized menu information to the user through earphones.

[0660] Fix requests and iterations

[0661] User: If the user is not satisfied with the proposed menu, they can say something like, "Is there an easier dish?" This speech is converted to text again and sent to the server.

[0662] Server: Re-parses the new request, generates a new proposal and sends it to the device.

[0663] The device converts the new suggestion into speech and relays it to the user. This process can be repeated until the user is satisfied.

[0664] Specific examples

[0665] A scene where the user is in a physical store

[0666] User: "What should I have for dinner tonight?"

[0667] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[0668] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[0669] Device: Converts to speech and tells the user, "How about grilled fish? We'll suggest salmon fillets, vegetables, and ingredients for miso soup."

[0670] User: "That's great, but is there anything easier?"

[0671] Terminal: Sends a new request to the server.

[0672] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[0673] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[0674] Prompt Sentence Examples

[0675] An example of a prompt used for a generative AI model is as follows:

[0676] The user's family consists of four people and they prefer Japanese food. They have an allergy to nuts and are seeking a low-fat diet due to their health condition.

[0677] Question: "What should I have for dinner tonight?"

[0678] Example response: "How about grilled fish? Suggestions include salmon fillet, vegetables, and miso soup ingredients."

[0679] This prompt is used as an instruction to ask the generative AI model to suggest appropriate menu items based on the user's background information and current question.

[0680] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0681] Step 1:

[0682] A user downloads a smartphone app and creates an account. When creating an account, they enter information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is saved as user profile information. The input comes from the user, and the output is user profile information saved on the device.

[0683] Step 2:

[0684] The terminal temporarily stores the user profile information in local storage and then transmits it to the server. The input is the user profile information and the output is transmission to the server. The terminal processes the data and transmits it to the server according to the communication protocol.

[0685] Step 3:

[0686] The server stores the received user profile information in a database. This information is referenced later when generating menus. The input is the user profile information sent from the terminal, and the output is the information stored in the database.

[0687] Step 4:

[0688] The user arrives at the store and connects earphones to the device. They launch the smartphone app and select voice input mode. They then speak a question, such as "What should I have for dinner tonight?" The input is the user's voice, and the output is the start of voice recognition.

[0689] Step 5:

[0690] The device converts the user's voice into text. Using a speech recognition method (e.g., Google's speech recognition service), the voice data is converted into text data. The input is the user's voice, and the output is text data.

[0691] Step 6:

[0692] The terminal transmits the converted text data to the server. The transmission means is used to transmit the data securely to the server in accordance with the transmission protocol. The input is the converted text data, and the output is transmission to the server.

[0693] Step 7:

[0694] The server references the text data along with pre-registered user profile information, providing the basic data for generating an appropriate menu. The input is the text data and user profile information, and the output is analysis preparation based on the profile.

[0695] Step 8:

[0696] The server uses the prompt generation means to generate a prompt sentence based on the user profile information. The prompt sentence to be used for the generative AI model is generated. The input is the user profile information and the question text data, and the output is the generated prompt sentence.

[0697] Step 9:

[0698] The server uses a generative AI model to generate an appropriate menu based on the prompt. The generated menu information is sent from the server to the device in text format. The input is the prompt, and the output is the generated menu information.

[0699] Step 10:

[0700] The device converts the text data received into speech. It uses a speech synthesis tool (e.g., the pyttsx3 library) to convert the text data into speech, and transmits the generated speech to the user through headphones. The input is the text data received from the server, and the output is a synthesized speech log.

[0701] Step 11:

[0702] If the user is dissatisfied with the proposed menu, they can input a second voice request, such as "Is there an easier dish?" This voice request is converted back into text and sent to the server. The input is a voice request from the user, and the output is again text data.

[0703] Step 12:

[0704] The server re-analyzes the new request, generates new suggestions, and sends them to the terminal. This process is repeated until the user is satisfied. The input is the text data of the new request, and the output is the newly generated menu information.

[0705] This series of steps allows users to efficiently decide on the right menu in a physical store. Specifically, a system is provided in which voice input, text conversion, data transmission, prompt generation, AI menu generation, and voice output all work together seamlessly.

[0706] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0707] This invention is a system that allows users to consult with AI in real time while shopping at the supermarket to decide on a menu, and by combining it with an emotion engine, it is possible to realize menu suggestions that take the user's emotions into consideration. This system is composed of multiple means, such as voice recognition, data transmission, AI generation, voice synthesis, emotion recognition and output. Specific embodiments are described below.

[0708] User Registration

[0709] User: Downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, likes and dislikes, allergies, health conditions, etc. This information is used to suggest more appropriate meals for future user queries.

[0710] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[0711] Server: Stores the received user information in a database. This stored information is referenced later when generating menus.

[0712] Voice input and request sending

[0713] User: Arrives at the supermarket and connects earphones to the device. Launches the smartphone app again and selects voice input mode. Voice input of a question through the earphones: "What should I have for dinner tonight?"

[0714] Device: Converts the user's speech into text (performs speech recognition processing).

[0715] Terminal: Sends the converted text data to the server.

[0716] Server: Along with the text data, the server references the user's pre-registered information (number of family members, likes and dislikes, allergies, and health status).

[0717] Emotion Recognition and Suggestion Generation

[0718] Server: Analyzes emotional information from voice data using emotion recognition tools, detects the user's emotional state, and makes appropriate suggestions based on that.

[0719] Server: Using a generative AI model, it generates an appropriate menu, taking into account the user's input information, pre-registered information, and detected emotional state.

[0720] Server: The generated menu information is sent to the terminal in text format.

[0721] Suggestion generation and voice response

[0722] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[0723] Device: Menu information is synthesized and transmitted to the user through earphones.

[0724] Fix requests and iterations

[0725] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[0726] Device: Converts re-inputted speech back into text.

[0727] Terminal: Re-send the converted text data to the server.

[0728] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[0729] Server: Sends new menu information in text format to the terminal.

[0730] Terminal: Converts the received text data back into speech.

[0731] Device: New menu information is synthesized and communicated to the user through earphones.

[0732] Specific examples

[0733] Example 1: A scene where the user is in a supermarket

[0734] User: "What should I have for dinner tonight?"

[0735] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[0736] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[0737] Server: Analyzes the user's emotions from their voice and, if it determines that the user is a little tired, adjusts the suggestions to say, "How about some easy grilled fish today?"

[0738] Device: Converts into speech and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[0739] User: "That's great, but is there anything easier?"

[0740] Terminal: Sends a new request to the server.

[0741] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[0742] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[0743] This series of steps allows users to efficiently decide on the appropriate menu at the supermarket and make their shopping experience smoother. In addition, by combining it with emotion recognition, it becomes possible to make suggestions that take into account the user's mental state, providing even greater convenience.

[0744] The processing flow will be explained below.

[0745] Step 1:

[0746] User: Download the smartphone app and open the account registration screen. Enter your name, number of family members, likes and dislikes, allergies, health conditions, etc.

[0747] Step 2:

[0748] Device: The entered information is temporarily stored in local storage.

[0749] Step 3:

[0750] Terminal: To send locally stored information to the server, a communication protocol is used to securely encrypt the data and process the transmission.

[0751] Step 4:

[0752] Server: Stores the received user information in a database.

[0753] Step 5:

[0754] User: Arrives at the supermarket, plugs in earphones, restarts the smartphone app, and selects voice input mode.

[0755] Step 6:

[0756] User: "What should I have for dinner tonight?" is spoken through the earphones.

[0757] Step 7:

[0758] Device: Converts the user's speech into text (performs speech recognition processing).

[0759] Step 8:

[0760] Terminal: Sends the converted text data to the server.

[0761] Step 9:

[0762] Server: Analyzes the text data and references the user's pre-registered information (number of family members, likes and dislikes, allergies, health status).

[0763] Step 10:

[0764] Server: Analyzes the user's emotional information from the voice data using emotion recognition means.

[0765] Step 11:

[0766] Server: Uses a generative AI model to generate appropriate menus based on the user's input information, pre-registration information, and detected emotional state.

[0767] Step 12:

[0768] Server: The generated menu information is sent to the terminal in text format.

[0769] Step 13:

[0770] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[0771] Step 14:

[0772] Device: Menu information is synthesized and transmitted to the user through earphones.

[0773] Step 15:

[0774] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[0775] Step 16:

[0776] Device: Converts re-inputted speech back into text.

[0777] Step 17:

[0778] Terminal: Re-send the converted text data to the server.

[0779] Step 18:

[0780] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[0781] Step 19:

[0782] Server: Sends new menu information in text format to the terminal.

[0783] Step 20:

[0784] Terminal: Converts the received text data back into speech.

[0785] Step 21:

[0786] Device: New menu information is synthesized and communicated to the user through earphones.

[0787] Example 2

[0788] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0789] Conventional menu suggestion systems typically convert a user's voice input into text, then send the text data to a server to generate a menu. However, these systems do not take into account the user's emotional state or detailed personal information registered in advance, making it difficult to suggest an optimal menu that matches the user's situation and emotions. Furthermore, the process of modifying the menu in response to user requests tends to be cumbersome, resulting in a lack of convenience.

[0790] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0791] In this invention, the server includes a means for generating a menu based on user input information, pre-registered information, and emotional information, a means for converting the received menu information into audio, and an output means for conveying the information to the user. This allows for optimal menu suggestions that take into account the user's emotional state and pre-registered information. Furthermore, the system allows for quick and flexible menu revisions in response to user requests, greatly improving user convenience.

[0792] "Speech recognition means" is a technology that analyzes the voice input by the user and converts it into text data.

[0793] The "transmission means" is a technology for transmitting the text data converted by the voice recognition means to the server.

[0794] The "generation means" is a technology that generates an appropriate menu based on the received user input information, pre-registered information, and emotional information.

[0795] The "transmission means" (the means for transmitting the generated menu information) is a technique for transmitting the generated menu information to the terminal.

[0796] The "voice synthesis means" is a technology that converts received menu information into voice format.

[0797] The "output means" is a technology for transmitting the voice generated by the voice synthesis means to the user.

[0798] The "storage means" is a technology that stores pre-registered information such as the number of family members, likes and dislikes, allergy information, and health status in a database.

[0799] The "reference means" is a technology that obtains necessary information from the storage means when generating a menu and provides it to the generation means.

[0800] "Emotional information" is data that indicates the user's emotional state, analyzed based on the user's voice and other input data.

[0801] This invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. It also combines an emotion engine to realize menu suggestions that take the user's emotions into consideration. This system is composed of multiple means, including voice recognition, data transmission, AI generation, voice synthesis, emotion recognition and output. Specific embodiments are shown below.

[0802] Hardware and Software Configuration

[0803] Users access the system using devices such as smartphones and tablets, which have applications installed for voice recognition, text data transmission, and voice data output.

[0804] The device uses the Google Cloud Speech-to-Text API for speech recognition, parses the voice input, and converts it into text data, which is then converted into JSON format and sent to a server over the internet.

[0805] The server is hosted in a cloud environment and is responsible for performing multiple functions. The server runs a generative AI model (e.g., GPT-3) to generate appropriate menus based on text data sent by users and pre-registered user information. It also uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state.

[0806] The generated menu information is sent from the server to the device in JSON format. The device receives this JSON data and converts it into text format. Next, it uses the Google Cloud Text-to-Speech API for speech synthesis to convert the text data into audio data. The converted audio data is played back to the user through headphones.

[0807] Specific examples

[0808] Example 1: A scene where the user is in a supermarket

[0809] The user speaks into the smartphone's microphone, asking, "What should I have for dinner tonight?"

[0810] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[0811] After analyzing the user information, the server generates a message saying, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the terminal.

[0812] The server analyzes the user's emotions from their voice, and if it determines that the user is a little tired, it adjusts its suggestions, saying, "How about some easy grilled fish today?"

[0813] The device converts the information into voice and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[0814] The user makes another request, saying, "That's good, but is there anything a little easier?"

[0815] The terminal sends a new request to the server.

[0816] The server generates a new suggestion and sends it to the device: "How about a quick chicken stir fry and salad?"

[0817] The device converts the information into speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[0818] Prompt Sentence Examples

[0819] User information: "Name: Sato, Family size: 3, Likes and dislikes: Likes spicy food, Allergies: None, Health condition: Good, Emotions: A little tired"

[0820] User Question: "What should I have for dinner tonight?"

[0821] Suggested menu:

[0822] Step 1: "How about making a simple grilled fish dish for dinner tonight? We suggest a salmon fillet, vegetables, and miso soup."

[0823] Step 2: "If grilled fish seems too difficult, how about sautéed chicken and a salad?"

[0824] This allows users to efficiently decide on the appropriate menu at the supermarket and make shopping a smoother experience. In addition, by combining emotion recognition, suggestions can be made that take into account the user's mental state, providing even greater convenience.

[0825] The above is a specific embodiment for carrying out the present invention.

[0826] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0827] Step 1: User Registration

[0828] The user downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, likes and dislikes, allergies, health conditions, etc. This information is then entered into the app's input form. Once the information is complete, the user presses the "Register" button.

[0829] Input: User's personal information (name, number of family members, likes and dislikes, allergies, health status)

[0830] Output: User information data in JSON format

[0831] The device temporarily saves the input information in local storage and sends it to the server. Specifically, the input data is converted to JSON format and sent to the server via an HTTP POST request.

[0832] Input: JSON format user information data

[0833] Output: HTTP request sent to the server

[0834] The server saves the received user information in the database. The server parses the received JSON data and stores the necessary information in the appropriate table in the database.

[0835] Input: JSON data sent in the HTTP request to the server

[0836] Output: User information record in database

[0837] Step 2: Speak and send your request

[0838] The user arrives at the supermarket, connects the earphones to the device, restarts the smartphone app, selects voice input mode, and inputs a question through the earphones: "What should I have for dinner tonight?"

[0839] Input: User's voice data

[0840] Output: Audio data is input to the device

[0841] The device converts the user's voice into text using a speech recognition API (e.g., Google Cloud Speech-to-Text), sends the voice data to the API, and retrieves the converted text data.

[0842] Input: Audio data

[0843] Output: Text data converted from audio

[0844] The device sends the converted text data to the server, where it is converted to JSON format along with the user ID and sent via an HTTP POST request to the server.

[0845] Input: Text data, user ID

[0846] Output: HTTP request sent to the server

[0847] The server retrieves the user's pre-registered information (number of family members, likes and dislikes, allergies, health status) and emotional information from a database to generate a menu based on the text data and pre-registered information.

[0848] Input: Text data, user ID

[0849] Output: User details (number of family members, likes and dislikes, allergies, health status)

[0850] Step 3: Emotion recognition and suggestion generation

[0851] The server uses the acquired voice data to perform emotion recognition. It sends the voice data to an emotion recognition API (e.g., IBM Watson Tone Analyzer) and analyzes the user's emotional state.

[0852] Input: Audio data

[0853] Output: Emotional information (e.g., "tired")

[0854] The server uses a generative AI model (e.g., GPT-3) to generate an appropriate menu based on the user's input, pre-registered information, and the detected emotional state. The prompt text includes the user's name, number of family members, likes and dislikes, allergy information, health status, emotional state, and voice input text and is input into the model.

[0855] Input: User input information, pre-registration information, emotional information

[0856] Output: Generated menu information (text format)

[0857] The server converts the generated menu information into JSON format and sends it to the terminal as an HTTP response.

[0858] Input: Generated menu information

[0859] Output: HTTP response sent to the device

[0860] Step 4: Proposal generation and voice response

[0861] The device converts the received text data into speech using a speech synthesis API (e.g., Google Cloud Text-to-Speech). The device sends the text data to the API and receives the speech data.

[0862] Input: Received text data

[0863] Output: Synthesized voice data

[0864] The device transmits the menu information to the user through a headset, and by outputting the audio data to the headset, the user can receive suggestions by voice.

[0865] Input: Synthesized voice data

[0866] Output: Audio output to the user

[0867] Step 5: Request revisions and iterate

[0868] If the user is dissatisfied with the proposed menu, he or she can again input a voice command such as "Is there an easier dish?"

[0869] Input: User's voice data

[0870] Output: Audio data is input to the device

[0871] The device again uses the speech recognition API to convert the speech to text, sends the speech data to the API, and retrieves the converted text data.

[0872] Input: Audio data

[0873] Output: Text data converted from audio

[0874] The device sends the converted text data to the server again, converts the text data into JSON format, and sends it to the server via an HTTP POST request.

[0875] Input: Text data, user ID

[0876] Output: HTTP request sent to the server

[0877] The server reanalyzes the new request data and generates a new menu. The data is input into the generative AI model, which generates new suggestions. The server also reanalyzes changes in the user's emotions and optimizes the suggestions.

[0878] Input: New text data, pre-registration information, emotion information

[0879] Output: New menu information (text format)

[0880] The server converts the new menu information back into JSON format and sends it to the terminal as an HTTP response.

[0881] Input: New menu information

[0882] Output: HTTP response sent to the device

[0883] The device converts the received text data into speech using the speech synthesis API. The device sends the text data to the API and obtains the speech data.

[0884] Input: Received text data

[0885] Output: Synthesized voice data

[0886] The device then sends the new menu information via voice synthesis to the user through earphones, allowing the user to receive new suggestions by voice by outputting the audio data to the earphones.

[0887] Input: Synthesized voice data

[0888] Output: Audio output to the user

[0889] (Application example 2)

[0890] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0891] Existing menu suggestion systems can suggest general menus based on user input, but they are unable to consider the user's mental state or emotions. This makes it difficult to provide optimal menus when the user is tired or in a particular mood, potentially reducing user satisfaction. They also lack the flexibility to respond to re-suggestions. Therefore, there is a need for a system that can suggest menus that take the user's emotional state into account and improve the user's mental satisfaction.

[0892] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion analysis means that uses an emotion recognition means to analyze the user's emotional state and optimize menu suggestions based on that analysis, a voice synthesis means that audibly conveys the generated menu information to the user, and a re-suggestion means that generates new menu information in response to a user's request for revision and conveys it to the user again. This makes it possible to propose an optimal menu that takes the user's emotional state into consideration, thereby improving psychological satisfaction.

[0893] "Speech recognition means" refers to technology that converts the voice input by the user into text.

[0894] "Transmission means" refers to a technique for transmitting text data to a server.

[0895] The "generation means" is a technology that allows the server to generate a menu based on information entered by the user and information registered in advance.

[0896] The "voice synthesis means" is a technology for converting the generated menu information into voice.

[0897] "Output means" refers to technology that conveys menu information to users in the form of synthesized voice.

[0898] "Emotion recognition means" is a technology that analyzes the user's emotional state.

[0899] "Emotion analysis means" is a technology that uses emotion recognition means to analyze the user's emotional state and optimizes menu suggestions based on that.

[0900] "Storage means" refers to the technology that stores information such as the number of family members, likes and dislikes, allergy information, and health status in a database.

[0901] The "reference means" is a technique for obtaining information from the storage means when creating a menu.

[0902] The "means for generating a prompt sentence" is a technique for generating a prompt sentence based on the emotional state of the user.

[0903] This invention is a system that allows users to decide on a menu while shopping in a physical store, consulting with AI in real time. By incorporating emotion recognition, it is possible to make suggestions that take into account the user's mental state. This system is configured as follows.

[0904] User Registration

[0905] User: Downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, preferences, allergy information, health conditions, etc. This information is used to suggest more appropriate meals for future user queries.

[0906] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[0907] Server: Stores the received user information in a database. This stored information is referenced later when generating menus.

[0908] Voice input and request sending

[0909] User: Arrives at the store and connects the earphones to the device. Relaunches the smartphone app and selects voice input mode. Voice input of a question through the earphones: "What should I have for dinner tonight?"

[0910] On the device: Converts the user's speech into text. The "SpeechRecognition" library is used to implement the speech recognition method.

[0911] Terminal: Sends the converted text data to the server.

[0912] Emotion Recognition and Suggestion Generation

[0913] Server: Analyzes emotional information from voice data using emotion recognition. Detects the user's emotional state and makes appropriate suggestions based on that. A "generative AI model" is used to generate an appropriate menu, taking into account the user's input information, pre-registered information, and the detected emotional state.

[0914] Server: The generated menu information is sent to the terminal in text format.

[0915] Suggestion generation and voice response

[0916] Terminal: Converts received text data into speech. The "gTTS" library is used to perform text-to-speech synthesis processing.

[0917] Device: Menu information is synthesized and transmitted to the user through earphones.

[0918] Fix requests and iterations

[0919] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[0920] Terminal: Converts the re-inputted voice into text and sends it back to the server.

[0921] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[0922] Server: Sends new menu information to the terminal.

[0923] Terminal: Converts received text data into speech.

[0924] Device: New menu information is synthesized and communicated to the user through earphones.

[0925] Specific examples

[0926] The following is the sequence of events when a user is in a physical store:

[0927] User: "What should I have for dinner tonight?"

[0928] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[0929] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[0930] Server: Analyzes the user's emotions from their voice and, if it determines that the user is a little tired, adjusts the suggestions to say, "How about some easy grilled fish today?"

[0931] Device: Converts into speech and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[0932] User: "That's great, but is there anything easier?"

[0933] Terminal: Sends a new request to the server.

[0934] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[0935] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[0936] Prompt Sentence Examples

[0937] "What should I have for dinner tonight?"

[0938] By implementing this, users can efficiently decide on the appropriate menu in a physical store and shop smoothly. In addition, by combining it with emotion recognition, it becomes possible to make suggestions that take into account the user's mental state, providing even greater convenience.

[0939] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0940] Step 1:

[0941] A user downloads the smartphone app and opens the account registration screen. They enter information such as their name, number of family members, preferences, allergies, and health conditions into a form within the app. Based on this information, the device temporarily saves it in local storage and then sends it to the server. The server then saves the received user information in a database.

[0942] Step 2:

[0943] The user arrives at the store and connects the earphones to their smartphone. Next, they launch the smartphone app and select voice input mode. The user then speaks a question into the earphones, such as "What should I have for dinner tonight?" The voice data is then imported into the smartphone device.

[0944] Step 3:

[0945] The device uses the SpeechRecognition library to convert the captured voice data into text. Specifically, the voice data is input into an acoustic model, and patterns that match the phonemes are analyzed to generate text data. This converted text data is then sent to the server.

[0946] Step 4:

[0947] The server references the received text data and pre-registered user information (family size, likes and dislikes, allergies, and health status). It then uses emotion recognition to analyze the user's emotional state from the voice data. A generative AI model generates an optimal menu plan that takes into account the user's input information and emotional state.

[0948] Step 5:

[0949] The server then sends the suggestions obtained from the generative AI model in text format to the device, where the suggestions are adjusted, such as suggesting less strenuous meals if the emotional state is evaluated as "tired."

[0950] Step 6:

[0951] The device uses the gTTS library to convert the received text data into voice data. During this process, a synthetic speech engine analyzes the text data phonetically and generates a natural speaking voice. The generated voice data is transmitted to the user through earphones.

[0952] Step 7:

[0953] If the user is dissatisfied with the proposed menu, they can input a question by voice again, such as "Is there an easier dish?" The new voice data is then imported into the device.

[0954] Step 8:

[0955] The device then uses the SpeechRecognition library to convert the newly input voice data into text, which is then sent back to the server.

[0956] Step 9:

[0957] The server re-analyzes the new request data and generates new menu suggestions. Using the generative AI model and sentiment analysis, it re-measures the user's latest emotional state and optimizes the suggestions accordingly. The new suggestions are generated and sent to the device.

[0958] Step 10:

[0959] The device then uses the gTTS library again to convert the new text suggestions into audio, which is then transmitted to the user through the earphones.

[0960] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0961] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0962] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0963] [Third embodiment]

[0964] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0965] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0966] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0967] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0968] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0969] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0970] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0971] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0972] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0973] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0974] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0975] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0976] The present invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. This system is composed of multiple means such as voice recognition, data transmission, AI generation, voice synthesis, and output. Specific embodiments are described below.

[0977] User Registration

[0978] User: Downloads the smartphone app and creates an account. When creating an account, the user enters information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is used to suggest more appropriate meals for future user inquiries.

[0979] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[0980] Server: Stores the received information in a database. This stored information is referenced later when generating menus.

[0981] Voice input and request sending

[0982] User: Arrives at the supermarket and connects earphones to the device. Launches the smartphone app and selects voice input mode. Enters a question: "What should I have for dinner tonight?"

[0983] Terminal: Converts the user's voice into text (voice recognition), and sends the converted text data to the server.

[0984] Server: Refers to the text data and the user's pre-registered information to generate an appropriate menu. This is done using a generative AI model.

[0985] Suggestion generation and voice response

[0986] Server: The generated menu information is sent to the terminal in text format.

[0987] Terminal: Converts received text data into speech (speech synthesis), and transmits the synthesized menu information to the user through earphones.

[0988] Fix requests and iterations

[0989] User: If the user is not satisfied with the proposed menu, they can say something like, "Is there an easier dish?" This speech is converted into text again and sent to the server.

[0990] Server: Re-parses the new request, generates a new proposal and sends it to the device.

[0991] The device converts the new suggestion into speech and relays it to the user. This process can be repeated until the user is satisfied.

[0992] Specific examples

[0993] Example 1: A scene where the user is in a supermarket

[0994] User: "What should I have for dinner tonight?"

[0995] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[0996] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[0997] Device: Converts to speech and tells the user, "How about grilled fish? We'll suggest salmon fillets, vegetables, and ingredients for miso soup."

[0998] User: "That's great, but is there anything easier?"

[0999] Terminal: Sends a new request to the server.

[1000] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[1001] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[1002] This series of steps allows users to efficiently decide on the appropriate menu at the supermarket and go shopping smoothly.

[1003] The present invention significantly reduces the effort and stress of daily meal planning decisions and provides a more efficient shopping experience.

[1004] The processing flow will be explained below.

[1005] Step 1:

[1006] User: Launches the smartphone app and opens the account registration screen. Enters information such as name, number of family members, likes and dislikes, allergies, and health conditions.

[1007] Step 2:

[1008] Device: The entered information is temporarily stored in local storage.

[1009] Step 3:

[1010] Terminal: To send locally stored information to the server, the data is encrypted via a communication protocol and the transmission process is performed.

[1011] Step 4:

[1012] Server: Stores the received user information in a database.

[1013] Step 5:

[1014] User: Arrives at the supermarket, plugs in earphones, restarts the smartphone app, and selects voice input mode.

[1015] Step 6:

[1016] User: "What should I have for dinner tonight?" is spoken through the earphones.

[1017] Step 7:

[1018] Device: Converts the user's speech into text (performs speech recognition processing).

[1019] Step 8:

[1020] Terminal: Sends the converted text data to the server.

[1021] Step 9:

[1022] Server: Analyzes the received text data and references the user's pre-registered information (number of family members, likes and dislikes, allergies, and health status).

[1023] Step 10:

[1024] Server: Uses a generative AI model to generate appropriate menus, taking into account user preferences and pre-registration information.

[1025] Step 11:

[1026] Server: The generated menu information is sent to the terminal in text format.

[1027] Step 12:

[1028] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[1029] Step 13:

[1030] Device: Menu information is synthesized and transmitted to the user through earphones.

[1031] Step 14:

[1032] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[1033] Step 15:

[1034] Device: Converts re-inputted speech back into text.

[1035] Step 16:

[1036] Terminal: Re-send the converted text data to the server.

[1037] Step 17:

[1038] Server: Re-parses the new request data and generates a new menu.

[1039] Step 18:

[1040] Server: Sends new menu information in text format to the terminal.

[1041] Step 19:

[1042] Terminal: Converts the received text data back into speech.

[1043] Step 20:

[1044] Device: New menu information is synthesized and communicated to the user through earphones.

[1045] Example 1

[1046] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1047] Until now, it has been difficult for users to provide suggestions that take into account individual requests, allergy information, preferences, etc. when deciding on daily menus. Furthermore, there has been no support system that allows users to decide on appropriate menus in real time while shopping at a supermarket, etc. This has led to problems such as stressful daily menu decisions and inefficient, time-consuming shopping.

[1048] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1049] In this invention, the server includes a speech recognition means for converting speech input by the user into text, a transmission means for transmitting the text data to the server, a generation means by which the server generates a menu based on the information input by the user and pre-registered information, a transmission means for transmitting the generated menu information to the terminal, a speech synthesis means for converting the received menu information into speech, an output means for conveying the synthesized speech information to the user, an input means for the user to input registered information, and a storage means for saving the input information. This enables users to receive real-time menu suggestions that take into account their individual requests, allergies, likes and dislikes while shopping at the supermarket.

[1050] "Speech recognition means" refers to a technical means that has the function of converting the voice input by the user into text data.

[1051] The "transmission means" is a technical means having the function of sending the converted text data to the server and transmitting the generated menu information to the terminal.

[1052] "Generation means" refers to a technical means that allows the server to generate a menu based on user input information and pre-registered information.

[1053] "Speech synthesis means" refers to a technical means that has the function of converting received text data into speech and conveying it to the user.

[1054] "Output means" refers to a technical means that has the function of conveying voice-synthesized information to the user.

[1055] "Input means" refers to a technical means that allows a user to input information required when creating an account (such as the number of family members, likes and dislikes, allergies, and health status).

[1056] "Storage means" refers to the technical means for storing the input information and making it available for reference in subsequent processes.

[1057] A "reference means" is a technical means that has the function of retrieving stored information when generating a menu and using it in the generation process.

[1058] A "generative AI model" is an artificial intelligence model used to generate appropriate menu suggestions based on user input and pre-registered information.

[1059] A "prompt" is an instruction or question that is input to a generative AI model to obtain an appropriate output.

[1060] The present invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. This system is realized through multiple processes, including voice recognition, data transmission, AI generation, voice synthesis and output. Specific embodiments are described below.

[1061] User Registration

[1062] Users download the smartphone app and tap the "Create a new account" button to begin creating an account. They then enter detailed information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is entered into the device via an input device.

[1063] The terminal temporarily stores the input information in local storage, attaches a security token, and transmits it to the server. The server records the received user information in a database using a storage means, and transmits a response indicating that the information has been saved to the terminal.

[1064] Voice input and request sending

[1065] The user arrives at the supermarket and connects the earphones to the device. Then, they launch the smartphone app, select voice input mode, and insert the earphones. In this state, they input a question such as, "What should I have for dinner tonight?"

[1066] The device receives the user's voice input and converts the voice into text using a speech recognition means. The specific technology used in this process is known as a general speech recognition API, for example, Google Cloud's Speech-to-Text API. The converted text data, "What should we have for dinner tonight?", is sent to the server via a transmission means.

[1067] Suggestion generation and voice response

[1068] The server uses a reference means to obtain the received text data and the user's pre-registered information, and generates an appropriate menu using a generative AI model. This generation means uses, for example, OpenAI's GPT series. The server generates a suggestion such as "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and sends this suggestion to the terminal via a transmission means.

[1069] The device converts the received suggested text into speech using a speech synthesis method, for example, Google Cloud's Text-to-Speech API, and the synthesized speech is communicated to the user through earphones: "How about grilled fish? We suggest salmon fillet, vegetables, and ingredients for miso soup."

[1070] Fix requests and iterations

[1071] If the user is not satisfied with the proposed menu, they can say, "Is there an easier dish?" again. The device will convert the speech to text again and send a new request, "Is there an easier dish?" to the server.

[1072] The server analyzes the new request, generates a new suggestion, "How about a quick chicken stir fry and salad?", and sends it to the device. The device converts the new suggestion from text to speech and tells the user, "How about a quick chicken stir fry and salad?" This process can be repeated until the user is satisfied.

[1073] Specific examples

[1074] Example 1: A scene where the user is in a supermarket

[1075] The user speaks, "What should I have for dinner tonight?" The device converts the speech into text and sends "What should I have for dinner tonight?" to the server.

[1076] Based on the user information, the server generates a request saying, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device. The device converts the suggestion into voice and conveys it to the user. If the user says, "That sounds good, but is there an easier dish?", the device sends a new request to the server, and the server generates, "Well, how about a simple chicken saute and salad?", which the device converts into voice again and conveys to the user. This process is repeated to determine a menu that satisfies the user.

[1077] Prompt Sentence Examples

[1078] 1. User prompt: "What should I have for dinner tonight?"

[1079] 2. User prompt: "Give me some suggestions for easier meals."

[1080] As described above, this system provides support for users to efficiently decide on appropriate menus in supermarkets and to carry out shopping smoothly.

[1081] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1082] Step 1: User Enters Information

[1083] Users download the smartphone app and tap the "Create a new account" button. They then enter information such as the number of family members, likes and dislikes, allergy information, and health status. The smartphone's touchscreen is used as the input method. The input data includes the number of family members, likes and dislikes, allergy information, and health status, and this data is temporarily stored on the device.

[1084] Step 2: Save the data to the server

[1085] The entered user information is sent from the terminal to the server. As a transmission method, the terminal's network function is used to attach a security token and send the data to the server. The server stores the received data in a database using a storage method. This allows the user information to be used later when generating menus.

[1086] Step 3: Dictation in Supermarket

[1087] The user arrives at the supermarket, plugs earphones into the device, and launches the app. They select the voice input mode and say, "What should I have for dinner tonight?" This speech is converted into text by a speech recognition tool. The input is the user's voice, and the output is text data.

[1088] Step 4: Send the audio data to the server

[1089] The terminal transmits text data to the server. The data received by the server is the text "What should we have for dinner tonight?" converted by the speech recognition means. The network function is used as the transmission means.

[1090] Step 5: The server generates the menu

[1091] The server retrieves the received text data and the user's pre-registered information through a reference method, and generates an appropriate menu using a generative AI model. The input is the text data and user information, and the output is a suggestion such as, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup."

[1092] Step 6: Send the generated proposal to your device

[1093] The generated menu information is sent from the server to the terminal. The network function is used as the transmission means. The generated menu information is output in text format.

[1094] Step 7: Convert the server's proposal to speech

[1095] The device converts the received text data into speech using a speech synthesis means. Google Cloud's Text-to-Speech API is used as an example of the speech synthesis means. The input is text data and the output is speech data.

[1096] Step 8: Communicate the generated audio to the user

[1097] The synthesized menu information is transmitted to the user through earphones, and the output is audio information that the user hears.

[1098] Step 9: User makes correction request

[1099] If the user is dissatisfied with the proposed menu, they can say, "Is there an easier dish?" This speech is again converted into text by the speech recognition means. The input is the user's voice, and the output is text data.

[1100] Step 10: Send a new request to the server

[1101] The terminal sends new text data to the server. The network function is used as the transmission method. The input is the text data, and the output is the transmission to the server.

[1102] Step 11: The server generates the menu again

[1103] The server analyzes the new request and generates new suggestions again using the generative AI model. The input is the text data of the new request, and the output is the new suggestion result.

[1104] Step 12: Send the new proposal to your device and convert it

[1105] The server sends the new suggestion to the device, which then converts it into speech using a speech synthesiser. The output is the new speech suggestion. This process is repeated until the suggestion is communicated to the user.

[1106] (Application example 1)

[1107] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1108] Currently, when shopping in a brick-and-mortar store, users spend a great deal of time and effort deciding on an appropriate meal plan, taking into account individual factors such as the number of people in their family, their likes and dislikes, allergies, and health conditions. This process is inefficient and can be stressful, especially for users who find meal planning difficult. Furthermore, current technology lacks a system that provides real-time meal suggestions optimized for each user's individual needs, and there is a need to solve this problem.

[1109] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1110] In this invention, the server includes a speech recognition means for converting user input speech into text, a transmission means for transmitting the text data to the server, a generation means for generating a menu based on the user input information and pre-registered information, a transmission means for transmitting the generated menu information to the terminal, a speech synthesis means for converting the received menu information into speech, an output means for conveying the speech-synthesized menu information to the user, and a prompt generation means for generating prompt sentences for the generative AI model using user profile information. This allows users to receive real-time menu suggestions optimized for their individual needs in physical stores, improving shopping efficiency and reducing stress.

[1111] A "speech recognition means" is a device or technology that converts user-supplied speech into text.

[1112] "Transmission means" refers to a device or technology for transmitting text data to a server.

[1113] "Generation means" refers to a device or technology for generating a menu based on user input information and pre-registered information.

[1114] The "voice synthesis means" is a device or technology for converting received menu information into voice.

[1115] "Output means" refers to a device or technology for conveying the synthesized menu information to the user.

[1116] A "prompt generator" is a device or technology that uses user profile information to generate prompt sentences for a generative AI model.

[1117] "User profile information" refers to information such as the number of people in a user's family, their likes and dislikes, allergies, and health status.

[1118] The system of the present invention is designed to enable users to decide on a menu while interacting with AI in real time while shopping in a physical store. This system is composed of multiple means, such as voice recognition, data transmission, menu generation, voice synthesis and output, and is equipped with a means for generating prompts using user profile information. Specific embodiments are described below.

[1119] Components

[1120] User Registration

[1121] User: Downloads the smartphone app and creates an account. When creating an account, the user enters information such as the number of people in their family, their likes and dislikes, allergies, and health conditions. This information is used to suggest more appropriate meals for future user queries.

[1122] Device: The entered information is temporarily stored in local storage and sent to the server.

[1123] Server: The received information is stored in a database, which is then referenced when creating a menu.

[1124] Voice input and request sending

[1125] User: Arrives at a physical store and connects earphones to the device. Launches the smartphone app and selects voice input mode. Enters a question: "What should I have for dinner tonight?"

[1126] Terminal: Converts the user's voice into text (speech recognition), and sends the converted text data to the server.

[1127] Server: Refers to the text data and pre-registered user profile information and generates an appropriate menu using a generative AI model. At this time, a prompt generation means generates a prompt sentence based on the user profile information.

[1128] Suggestion generation and voice response

[1129] Server: Sends the generated menu information in text format to the terminal.

[1130] Device: Converts received text data into speech (speech synthesis), and transmits the synthesized menu information to the user through earphones.

[1131] Fix requests and iterations

[1132] User: If the user is not satisfied with the proposed menu, they can say something like, "Is there an easier dish?" This speech is converted to text again and sent to the server.

[1133] Server: Re-parses the new request, generates a new proposal and sends it to the device.

[1134] The device converts the new suggestion into speech and relays it to the user. This process can be repeated until the user is satisfied.

[1135] Specific examples

[1136] A scene where the user is in a physical store

[1137] User: "What should I have for dinner tonight?"

[1138] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[1139] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[1140] Device: Converts to speech and tells the user, "How about grilled fish? We'll suggest salmon fillets, vegetables, and ingredients for miso soup."

[1141] User: "That's great, but is there anything easier?"

[1142] Terminal: Sends a new request to the server.

[1143] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[1144] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[1145] Prompt Sentence Examples

[1146] An example of a prompt used for a generative AI model is as follows:

[1147] The user's family consists of four people and they prefer Japanese food. They have an allergy to nuts and are seeking a low-fat diet due to their health condition.

[1148] Question: "What should I have for dinner tonight?"

[1149] Example response: "How about grilled fish? Suggestions include salmon fillet, vegetables, and miso soup ingredients."

[1150] This prompt is used as an instruction to ask the generative AI model to suggest appropriate menu items based on the user's background information and current question.

[1151] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1152] Step 1:

[1153] A user downloads a smartphone app and creates an account. When creating an account, they enter information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is saved as user profile information. The input comes from the user, and the output is user profile information saved on the device.

[1154] Step 2:

[1155] The terminal temporarily stores the user profile information in local storage and then transmits it to the server. The input is the user profile information and the output is transmission to the server. The terminal processes the data and transmits it to the server according to the communication protocol.

[1156] Step 3:

[1157] The server stores the received user profile information in a database. This information is referenced later when generating menus. The input is the user profile information sent from the terminal, and the output is the information stored in the database.

[1158] Step 4:

[1159] The user arrives at the store and connects earphones to the device. They launch the smartphone app and select voice input mode. They then speak a question, such as "What should I have for dinner tonight?" The input is the user's voice, and the output is the start of voice recognition.

[1160] Step 5:

[1161] The device converts the user's voice into text. Using a speech recognition method (e.g., Google's speech recognition service), the voice data is converted into text data. The input is the user's voice, and the output is text data.

[1162] Step 6:

[1163] The terminal transmits the converted text data to the server. The transmission means is used to transmit the data securely to the server in accordance with the transmission protocol. The input is the converted text data, and the output is transmission to the server.

[1164] Step 7:

[1165] The server references the text data along with pre-registered user profile information, providing the basic data for generating an appropriate menu. The input is the text data and user profile information, and the output is analysis preparation based on the profile.

[1166] Step 8:

[1167] The server uses the prompt generation means to generate a prompt sentence based on the user profile information. The prompt sentence to be used for the generative AI model is generated. The input is the user profile information and the question text data, and the output is the generated prompt sentence.

[1168] Step 9:

[1169] The server uses a generative AI model to generate an appropriate menu based on the prompt. The generated menu information is sent from the server to the device in text format. The input is the prompt, and the output is the generated menu information.

[1170] Step 10:

[1171] The device converts the text data received into speech. It uses a speech synthesis tool (e.g., the pyttsx3 library) to convert the text data into speech, and transmits the generated speech to the user through headphones. The input is the text data received from the server, and the output is a synthesized speech log.

[1172] Step 11:

[1173] If the user is dissatisfied with the proposed menu, they can input a second voice request, such as "Is there an easier dish?" This voice request is converted back into text and sent to the server. The input is a voice request from the user, and the output is again text data.

[1174] Step 12:

[1175] The server re-analyzes the new request, generates new suggestions, and sends them to the terminal. This process is repeated until the user is satisfied. The input is the text data of the new request, and the output is the newly generated menu information.

[1176] This series of steps allows users to efficiently decide on the right menu in a physical store. Specifically, a system is provided in which voice input, text conversion, data transmission, prompt generation, AI menu generation, and voice output all work together seamlessly.

[1177] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1178] This invention is a system that allows users to consult with AI in real time while shopping at the supermarket to decide on a menu, and by combining it with an emotion engine, it is possible to realize menu suggestions that take the user's emotions into consideration. This system is composed of multiple means, such as voice recognition, data transmission, AI generation, voice synthesis, emotion recognition and output. Specific embodiments are described below.

[1179] User Registration

[1180] User: Downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, likes and dislikes, allergies, health conditions, etc. This information is used to suggest more appropriate meals for future user queries.

[1181] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[1182] Server: Stores the received user information in a database. This stored information is referenced later when generating menus.

[1183] Voice input and request sending

[1184] User: Arrives at the supermarket and connects earphones to the device. Launches the smartphone app again and selects voice input mode. Voice input of a question through the earphones: "What should I have for dinner tonight?"

[1185] Device: Converts the user's speech into text (performs speech recognition processing).

[1186] Terminal: Sends the converted text data to the server.

[1187] Server: Along with the text data, the server references the user's pre-registered information (number of family members, likes and dislikes, allergies, and health status).

[1188] Emotion Recognition and Suggestion Generation

[1189] Server: Analyzes emotional information from voice data using emotion recognition tools, detects the user's emotional state, and makes appropriate suggestions based on that.

[1190] Server: Using a generative AI model, it generates an appropriate menu, taking into account the user's input information, pre-registered information, and detected emotional state.

[1191] Server: The generated menu information is sent to the terminal in text format.

[1192] Suggestion generation and voice response

[1193] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[1194] Device: Menu information is synthesized and transmitted to the user through earphones.

[1195] Fix requests and iterations

[1196] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[1197] Device: Converts re-inputted speech back into text.

[1198] Terminal: Re-send the converted text data to the server.

[1199] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[1200] Server: Sends new menu information in text format to the terminal.

[1201] Terminal: Converts the received text data back into speech.

[1202] Device: New menu information is synthesized and communicated to the user through earphones.

[1203] Specific examples

[1204] Example 1: A scene where the user is in a supermarket

[1205] User: "What should I have for dinner tonight?"

[1206] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[1207] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[1208] Server: Analyzes the user's emotions from their voice and, if it determines that the user is a little tired, adjusts the suggestions to say, "How about some easy grilled fish today?"

[1209] Device: Converts into speech and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[1210] User: "That's great, but is there anything easier?"

[1211] Terminal: Sends a new request to the server.

[1212] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[1213] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[1214] This series of steps allows users to efficiently decide on the appropriate menu at the supermarket and make their shopping experience smoother. In addition, by combining it with emotion recognition, it becomes possible to make suggestions that take into account the user's mental state, providing even greater convenience.

[1215] The processing flow will be explained below.

[1216] Step 1:

[1217] User: Download the smartphone app and open the account registration screen. Enter your name, number of family members, likes and dislikes, allergies, health conditions, etc.

[1218] Step 2:

[1219] Device: The entered information is temporarily stored in local storage.

[1220] Step 3:

[1221] Terminal: To send locally stored information to the server, a communication protocol is used to securely encrypt the data and process the transmission.

[1222] Step 4:

[1223] Server: Stores the received user information in a database.

[1224] Step 5:

[1225] User: Arrives at the supermarket, plugs in earphones, restarts the smartphone app, and selects voice input mode.

[1226] Step 6:

[1227] User: "What should I have for dinner tonight?" is spoken through the earphones.

[1228] Step 7:

[1229] Device: Converts the user's speech into text (performs speech recognition processing).

[1230] Step 8:

[1231] Terminal: Sends the converted text data to the server.

[1232] Step 9:

[1233] Server: Analyzes the text data and references the user's pre-registered information (number of family members, likes and dislikes, allergies, health status).

[1234] Step 10:

[1235] Server: Analyzes the user's emotional information from the voice data using emotion recognition means.

[1236] Step 11:

[1237] Server: Uses a generative AI model to generate appropriate menus based on the user's input information, pre-registration information, and detected emotional state.

[1238] Step 12:

[1239] Server: The generated menu information is sent to the terminal in text format.

[1240] Step 13:

[1241] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[1242] Step 14:

[1243] Device: Menu information is synthesized and transmitted to the user through earphones.

[1244] Step 15:

[1245] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[1246] Step 16:

[1247] Device: Converts re-inputted speech back into text.

[1248] Step 17:

[1249] Terminal: Re-send the converted text data to the server.

[1250] Step 18:

[1251] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[1252] Step 19:

[1253] Server: Sends new menu information in text format to the terminal.

[1254] Step 20:

[1255] Terminal: Converts the received text data back into speech.

[1256] Step 21:

[1257] Device: New menu information is synthesized and communicated to the user through earphones.

[1258] Example 2

[1259] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1260] Conventional menu suggestion systems typically convert a user's voice input into text, then send the text data to a server to generate a menu. However, these systems do not take into account the user's emotional state or detailed personal information registered in advance, making it difficult to suggest an optimal menu that matches the user's situation and emotions. Furthermore, the process of modifying the menu in response to user requests tends to be cumbersome, resulting in a lack of convenience.

[1261] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1262] In this invention, the server includes a means for generating a menu based on user input information, pre-registered information, and emotional information, a means for converting the received menu information into audio, and an output means for conveying the information to the user. This allows for optimal menu suggestions that take into account the user's emotional state and pre-registered information. Furthermore, the system allows for quick and flexible menu revisions in response to user requests, greatly improving user convenience.

[1263] "Speech recognition means" is a technology that analyzes the voice input by the user and converts it into text data.

[1264] The "transmission means" is a technology for transmitting the text data converted by the voice recognition means to the server.

[1265] The "generation means" is a technology that generates an appropriate menu based on the received user input information, pre-registered information, and emotional information.

[1266] The "transmission means" (the means for transmitting the generated menu information) is a technique for transmitting the generated menu information to the terminal.

[1267] The "voice synthesis means" is a technology that converts received menu information into voice format.

[1268] The "output means" is a technology for transmitting the voice generated by the voice synthesis means to the user.

[1269] The "storage means" is a technology that stores pre-registered information such as the number of family members, likes and dislikes, allergy information, and health status in a database.

[1270] The "reference means" is a technology that obtains necessary information from the storage means when generating a menu and provides it to the generation means.

[1271] "Emotional information" is data that indicates the user's emotional state, analyzed based on the user's voice and other input data.

[1272] This invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. It also combines an emotion engine to realize menu suggestions that take the user's emotions into consideration. This system is composed of multiple means, including voice recognition, data transmission, AI generation, voice synthesis, emotion recognition and output. Specific embodiments are shown below.

[1273] Hardware and Software Configuration

[1274] Users access the system using devices such as smartphones and tablets, which have applications installed for voice recognition, text data transmission, and voice data output.

[1275] The device uses the Google Cloud Speech-to-Text API for speech recognition, parses the voice input, and converts it into text data, which is then converted into JSON format and sent to a server over the internet.

[1276] The server is hosted in a cloud environment and is responsible for performing multiple functions. The server runs a generative AI model (e.g., GPT-3) to generate appropriate menus based on text data sent by users and pre-registered user information. It also uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state.

[1277] The generated menu information is sent from the server to the device in JSON format. The device receives this JSON data and converts it into text format. Next, it uses the Google Cloud Text-to-Speech API for speech synthesis to convert the text data into audio data. The converted audio data is played back to the user through headphones.

[1278] Specific examples

[1279] Example 1: A scene where the user is in a supermarket

[1280] The user speaks into the smartphone's microphone, asking, "What should I have for dinner tonight?"

[1281] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[1282] After analyzing the user information, the server generates a message saying, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the terminal.

[1283] The server analyzes the user's emotions from their voice, and if it determines that the user is a little tired, it adjusts its suggestions, saying, "How about some easy grilled fish today?"

[1284] The device converts the information into voice and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[1285] The user makes another request, saying, "That's good, but is there anything a little easier?"

[1286] The terminal sends a new request to the server.

[1287] The server generates a new suggestion and sends it to the device: "How about a quick chicken stir fry and salad?"

[1288] The device converts the information into speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[1289] Prompt Sentence Examples

[1290] User information: "Name: Sato, Family size: 3, Likes and dislikes: Likes spicy food, Allergies: None, Health condition: Good, Emotions: A little tired"

[1291] User Question: "What should I have for dinner tonight?"

[1292] Suggested menu:

[1293] Step 1: "How about making a simple grilled fish dish for dinner tonight? We suggest a salmon fillet, vegetables, and miso soup."

[1294] Step 2: "If grilled fish seems too difficult, how about sautéed chicken and a salad?"

[1295] This allows users to efficiently decide on the appropriate menu at the supermarket and make shopping a smoother experience. In addition, by combining emotion recognition, suggestions can be made that take into account the user's mental state, providing even greater convenience.

[1296] The above is a specific embodiment for carrying out the present invention.

[1297] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1298] Step 1: User Registration

[1299] The user downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, likes and dislikes, allergies, health conditions, etc. This information is then entered into the app's input form. Once the information is complete, the user presses the "Register" button.

[1300] Input: User's personal information (name, number of family members, likes and dislikes, allergies, health status)

[1301] Output: User information data in JSON format

[1302] The device temporarily saves the input information in local storage and sends it to the server. Specifically, the input data is converted to JSON format and sent to the server via an HTTP POST request.

[1303] Input: JSON format user information data

[1304] Output: HTTP request sent to the server

[1305] The server saves the received user information in the database. The server parses the received JSON data and stores the necessary information in the appropriate table in the database.

[1306] Input: JSON data sent in the HTTP request to the server

[1307] Output: User information record in database

[1308] Step 2: Speak and send your request

[1309] The user arrives at the supermarket, connects the earphones to the device, restarts the smartphone app, selects voice input mode, and inputs a question through the earphones: "What should I have for dinner tonight?"

[1310] Input: User's voice data

[1311] Output: Audio data is input to the device

[1312] The device converts the user's voice into text using a speech recognition API (e.g., Google Cloud Speech-to-Text), sends the voice data to the API, and retrieves the converted text data.

[1313] Input: Audio data

[1314] Output: Text data converted from audio

[1315] The device sends the converted text data to the server, where it is converted to JSON format along with the user ID and sent via an HTTP POST request to the server.

[1316] Input: Text data, user ID

[1317] Output: HTTP request sent to the server

[1318] The server retrieves the user's pre-registered information (number of family members, likes and dislikes, allergies, health status) and emotional information from a database to generate a menu based on the text data and pre-registered information.

[1319] Input: Text data, user ID

[1320] Output: User details (number of family members, likes and dislikes, allergies, health status)

[1321] Step 3: Emotion recognition and suggestion generation

[1322] The server uses the acquired voice data to perform emotion recognition. It sends the voice data to an emotion recognition API (e.g., IBM Watson Tone Analyzer) and analyzes the user's emotional state.

[1323] Input: Audio data

[1324] Output: Emotional information (e.g., "tired")

[1325] The server uses a generative AI model (e.g., GPT-3) to generate an appropriate menu based on the user's input, pre-registered information, and the detected emotional state. The prompt text includes the user's name, number of family members, likes and dislikes, allergy information, health status, emotional state, and voice input text and is input into the model.

[1326] Input: User input information, pre-registration information, emotional information

[1327] Output: Generated menu information (text format)

[1328] The server converts the generated menu information into JSON format and sends it to the terminal as an HTTP response.

[1329] Input: Generated menu information

[1330] Output: HTTP response sent to the device

[1331] Step 4: Proposal generation and voice response

[1332] The device converts the received text data into speech using a speech synthesis API (e.g., Google Cloud Text-to-Speech). The device sends the text data to the API and receives the speech data.

[1333] Input: Received text data

[1334] Output: Synthesized voice data

[1335] The device transmits the menu information to the user through a headset, and by outputting the audio data to the headset, the user can receive suggestions by voice.

[1336] Input: Synthesized voice data

[1337] Output: Audio output to the user

[1338] Step 5: Request revisions and iterate

[1339] If the user is dissatisfied with the proposed menu, he or she can again input a voice command such as "Is there an easier dish?"

[1340] Input: User's voice data

[1341] Output: Audio data is input to the device

[1342] The device again uses the speech recognition API to convert the speech to text, sends the speech data to the API, and retrieves the converted text data.

[1343] Input: Audio data

[1344] Output: Text data converted from audio

[1345] The device sends the converted text data to the server again, converts the text data into JSON format, and sends it to the server via an HTTP POST request.

[1346] Input: Text data, user ID

[1347] Output: HTTP request sent to the server

[1348] The server reanalyzes the new request data and generates a new menu. The data is input into the generative AI model, which generates new suggestions. The server also reanalyzes changes in the user's emotions and optimizes the suggestions.

[1349] Input: New text data, pre-registration information, emotion information

[1350] Output: New menu information (text format)

[1351] The server converts the new menu information back into JSON format and sends it to the terminal as an HTTP response.

[1352] Input: New menu information

[1353] Output: HTTP response sent to the device

[1354] The device converts the received text data into speech using the speech synthesis API. The device sends the text data to the API and obtains the speech data.

[1355] Input: Received text data

[1356] Output: Synthesized voice data

[1357] The device then sends the new menu information via voice synthesis to the user through earphones, allowing the user to receive new suggestions by voice by outputting the audio data to the earphones.

[1358] Input: Synthesized voice data

[1359] Output: Audio output to the user

[1360] (Application example 2)

[1361] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1362] Existing menu suggestion systems can suggest general menus based on user input, but they are unable to consider the user's mental state or emotions. This makes it difficult to provide optimal menus when the user is tired or in a particular mood, potentially reducing user satisfaction. They also lack the flexibility to respond to re-suggestions. Therefore, there is a need for a system that can suggest menus that take the user's emotional state into account and improve the user's mental satisfaction.

[1363] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion analysis means that uses an emotion recognition means to analyze the user's emotional state and optimize menu suggestions based on that analysis, a voice synthesis means that audibly conveys the generated menu information to the user, and a re-suggestion means that generates new menu information in response to a user's request for revision and conveys it to the user again. This makes it possible to propose an optimal menu that takes the user's emotional state into consideration, thereby improving psychological satisfaction.

[1364] "Speech recognition means" refers to technology that converts the voice input by the user into text.

[1365] "Transmission means" refers to a technique for transmitting text data to a server.

[1366] The "generation means" is a technology that allows the server to generate a menu based on information entered by the user and information registered in advance.

[1367] The "voice synthesis means" is a technology for converting the generated menu information into voice.

[1368] "Output means" refers to technology that conveys menu information to users in the form of synthesized voice.

[1369] "Emotion recognition means" is a technology that analyzes the user's emotional state.

[1370] "Emotion analysis means" is a technology that uses emotion recognition means to analyze the user's emotional state and optimizes menu suggestions based on that.

[1371] "Storage means" refers to the technology that stores information such as the number of family members, likes and dislikes, allergy information, and health status in a database.

[1372] The "reference means" is a technique for obtaining information from the storage means when creating a menu.

[1373] The "means for generating a prompt sentence" is a technique for generating a prompt sentence based on the emotional state of the user.

[1374] This invention is a system that allows users to decide on a menu while shopping in a physical store, consulting with AI in real time. By incorporating emotion recognition, it is possible to make suggestions that take into account the user's mental state. This system is configured as follows.

[1375] User Registration

[1376] User: Downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, preferences, allergy information, health conditions, etc. This information is used to suggest more appropriate meals for future user queries.

[1377] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[1378] Server: Stores the received user information in a database. This stored information is referenced later when generating menus.

[1379] Voice input and request sending

[1380] User: Arrives at the store and connects the earphones to the device. Relaunches the smartphone app and selects voice input mode. Voice input of a question through the earphones: "What should I have for dinner tonight?"

[1381] On the device: Converts the user's speech into text. The "SpeechRecognition" library is used to implement the speech recognition method.

[1382] Terminal: Sends the converted text data to the server.

[1383] Emotion Recognition and Suggestion Generation

[1384] Server: Analyzes emotional information from voice data using emotion recognition. Detects the user's emotional state and makes appropriate suggestions based on that. A "generative AI model" is used to generate an appropriate menu, taking into account the user's input information, pre-registered information, and the detected emotional state.

[1385] Server: The generated menu information is sent to the terminal in text format.

[1386] Suggestion generation and voice response

[1387] Terminal: Converts received text data into speech. The "gTTS" library is used to perform text-to-speech synthesis processing.

[1388] Device: Menu information is synthesized and transmitted to the user through earphones.

[1389] Fix requests and iterations

[1390] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[1391] Terminal: Converts the re-inputted voice into text and sends it back to the server.

[1392] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[1393] Server: Sends new menu information to the terminal.

[1394] Terminal: Converts received text data into speech.

[1395] Device: New menu information is synthesized and communicated to the user through earphones.

[1396] Specific examples

[1397] The following is the sequence of events when a user is in a physical store:

[1398] User: "What should I have for dinner tonight?"

[1399] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[1400] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[1401] Server: Analyzes the user's emotions from their voice and, if it determines that the user is a little tired, adjusts the suggestions to say, "How about some easy grilled fish today?"

[1402] Device: Converts into speech and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[1403] User: "That's great, but is there anything easier?"

[1404] Terminal: Sends a new request to the server.

[1405] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[1406] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[1407] Prompt Sentence Examples

[1408] "What should I have for dinner tonight?"

[1409] By implementing this, users can efficiently decide on the appropriate menu in a physical store and shop smoothly. In addition, by combining it with emotion recognition, it becomes possible to make suggestions that take into account the user's mental state, providing even greater convenience.

[1410] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1411] Step 1:

[1412] A user downloads the smartphone app and opens the account registration screen. They enter information such as their name, number of family members, preferences, allergies, and health conditions into a form within the app. Based on this information, the device temporarily saves it in local storage and then sends it to the server. The server then saves the received user information in a database.

[1413] Step 2:

[1414] The user arrives at the store and connects the earphones to their smartphone. Next, they launch the smartphone app and select voice input mode. The user then speaks a question into the earphones, such as "What should I have for dinner tonight?" The voice data is then imported into the smartphone device.

[1415] Step 3:

[1416] The device uses the SpeechRecognition library to convert the captured voice data into text. Specifically, the voice data is input into an acoustic model, and patterns that match the phonemes are analyzed to generate text data. This converted text data is then sent to the server.

[1417] Step 4:

[1418] The server references the received text data and pre-registered user information (family size, likes and dislikes, allergies, and health status). It then uses emotion recognition to analyze the user's emotional state from the voice data. A generative AI model generates an optimal menu plan that takes into account the user's input information and emotional state.

[1419] Step 5:

[1420] The server then sends the suggestions obtained from the generative AI model in text format to the device, where the suggestions are adjusted, such as suggesting less strenuous meals if the emotional state is evaluated as "tired."

[1421] Step 6:

[1422] The device uses the gTTS library to convert the received text data into voice data. During this process, a synthetic speech engine analyzes the text data phonetically and generates a natural speaking voice. The generated voice data is transmitted to the user through earphones.

[1423] Step 7:

[1424] If the user is dissatisfied with the proposed menu, they can input a question by voice again, such as "Is there an easier dish?" The new voice data is then imported into the device.

[1425] Step 8:

[1426] The device then uses the SpeechRecognition library to convert the newly input voice data into text, which is then sent back to the server.

[1427] Step 9:

[1428] The server re-analyzes the new request data and generates new menu suggestions. Using the generative AI model and sentiment analysis, it re-measures the user's latest emotional state and optimizes the suggestions accordingly. The new suggestions are generated and sent to the device.

[1429] Step 10:

[1430] The device then uses the gTTS library again to convert the new text suggestions into audio, which is then transmitted to the user through the earphones.

[1431] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1432] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1433] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1434] [Fourth embodiment]

[1435] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1436] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1437] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1438] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1439] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1440] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1441] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1442] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1443] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1444] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1445] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1446] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1447] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1448] The present invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. This system is composed of multiple means such as voice recognition, data transmission, AI generation, voice synthesis, and output. Specific embodiments are described below.

[1449] User Registration

[1450] User: Downloads the smartphone app and creates an account. When creating an account, the user enters information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is used to suggest more appropriate meals for future user inquiries.

[1451] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[1452] Server: Stores the received information in a database. This stored information is referenced later when generating menus.

[1453] Voice input and request sending

[1454] User: Arrives at the supermarket and connects earphones to the device. Launches the smartphone app and selects voice input mode. Enters a question: "What should I have for dinner tonight?"

[1455] Terminal: Converts the user's voice into text (voice recognition), and sends the converted text data to the server.

[1456] Server: Refers to the text data and the user's pre-registered information to generate an appropriate menu. This is done using a generative AI model.

[1457] Suggestion generation and voice response

[1458] Server: The generated menu information is sent to the terminal in text format.

[1459] Terminal: Converts received text data into speech (speech synthesis), and transmits the synthesized menu information to the user through earphones.

[1460] Fix requests and iterations

[1461] User: If the user is not satisfied with the proposed menu, they can say something like, "Is there an easier dish?" This speech is converted into text again and sent to the server.

[1462] Server: Re-parses the new request, generates a new proposal and sends it to the device.

[1463] The device converts the new suggestion into speech and relays it to the user. This process can be repeated until the user is satisfied.

[1464] Specific examples

[1465] Example 1: A scene where the user is in a supermarket

[1466] User: "What should I have for dinner tonight?"

[1467] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[1468] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[1469] Device: Converts to speech and tells the user, "How about grilled fish? We'll suggest salmon fillets, vegetables, and ingredients for miso soup."

[1470] User: "That's great, but is there anything easier?"

[1471] Terminal: Sends a new request to the server.

[1472] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[1473] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[1474] This series of steps allows users to efficiently decide on the appropriate menu at the supermarket and go shopping smoothly.

[1475] The present invention significantly reduces the effort and stress of daily meal planning decisions and provides a more efficient shopping experience.

[1476] The processing flow will be explained below.

[1477] Step 1:

[1478] User: Launches the smartphone app and opens the account registration screen. Enters information such as name, number of family members, likes and dislikes, allergies, and health conditions.

[1479] Step 2:

[1480] Device: The entered information is temporarily stored in local storage.

[1481] Step 3:

[1482] Terminal: To send locally stored information to the server, the data is encrypted via a communication protocol and the transmission process is performed.

[1483] Step 4:

[1484] Server: Stores the received user information in a database.

[1485] Step 5:

[1486] User: Arrives at the supermarket, plugs in earphones, restarts the smartphone app, and selects voice input mode.

[1487] Step 6:

[1488] User: "What should I have for dinner tonight?" is spoken through the earphones.

[1489] Step 7:

[1490] Device: Converts the user's speech into text (performs speech recognition processing).

[1491] Step 8:

[1492] Terminal: Sends the converted text data to the server.

[1493] Step 9:

[1494] Server: Analyzes the received text data and references the user's pre-registered information (number of family members, likes and dislikes, allergies, and health status).

[1495] Step 10:

[1496] Server: Uses a generative AI model to generate appropriate menus, taking into account user preferences and pre-registration information.

[1497] Step 11:

[1498] Server: The generated menu information is sent to the terminal in text format.

[1499] Step 12:

[1500] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[1501] Step 13:

[1502] Device: Menu information is synthesized and transmitted to the user through earphones.

[1503] Step 14:

[1504] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[1505] Step 15:

[1506] Device: Converts re-inputted speech back into text.

[1507] Step 16:

[1508] Terminal: Re-send the converted text data to the server.

[1509] Step 17:

[1510] Server: Re-parses the new request data and generates a new menu.

[1511] Step 18:

[1512] Server: Sends new menu information in text format to the terminal.

[1513] Step 19:

[1514] Terminal: Converts the received text data back into speech.

[1515] Step 20:

[1516] Device: New menu information is synthesized and communicated to the user through earphones.

[1517] Example 1

[1518] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1519] Until now, it has been difficult for users to provide suggestions that take into account individual requests, allergy information, preferences, etc. when deciding on daily menus. Furthermore, there has been no support system that allows users to decide on appropriate menus in real time while shopping at a supermarket, etc. This has led to problems such as stressful daily menu decisions and inefficient, time-consuming shopping.

[1520] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1521] In this invention, the server includes a speech recognition means for converting speech input by the user into text, a transmission means for transmitting the text data to the server, a generation means by which the server generates a menu based on the information input by the user and pre-registered information, a transmission means for transmitting the generated menu information to the terminal, a speech synthesis means for converting the received menu information into speech, an output means for conveying the synthesized speech information to the user, an input means for the user to input registered information, and a storage means for saving the input information. This enables users to receive real-time menu suggestions that take into account their individual requests, allergies, likes and dislikes while shopping at the supermarket.

[1522] "Speech recognition means" refers to a technical means that has the function of converting the voice input by the user into text data.

[1523] The "transmission means" is a technical means having the function of sending the converted text data to the server and transmitting the generated menu information to the terminal.

[1524] "Generation means" refers to a technical means that allows the server to generate a menu based on user input information and pre-registered information.

[1525] "Speech synthesis means" refers to a technical means that has the function of converting received text data into speech and conveying it to the user.

[1526] "Output means" refers to a technical means that has the function of conveying voice-synthesized information to the user.

[1527] "Input means" refers to a technical means that allows a user to input information required when creating an account (such as the number of family members, likes and dislikes, allergies, and health status).

[1528] "Storage means" refers to the technical means for storing the input information and making it available for reference in subsequent processes.

[1529] A "reference means" is a technical means that has the function of retrieving stored information when generating a menu and using it in the generation process.

[1530] A "generative AI model" is an artificial intelligence model used to generate appropriate menu suggestions based on user input and pre-registered information.

[1531] A "prompt" is an instruction or question that is input to a generative AI model to obtain an appropriate output.

[1532] The present invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. This system is realized through multiple processes, including voice recognition, data transmission, AI generation, voice synthesis and output. Specific embodiments are described below.

[1533] User Registration

[1534] Users download the smartphone app and tap the "Create a new account" button to begin creating an account. They then enter detailed information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is entered into the device via an input device.

[1535] The terminal temporarily stores the input information in local storage, attaches a security token, and transmits it to the server. The server records the received user information in a database using a storage means, and transmits a response indicating that the information has been saved to the terminal.

[1536] Voice input and request sending

[1537] The user arrives at the supermarket and connects the earphones to the device. Then, they launch the smartphone app, select voice input mode, and insert the earphones. In this state, they input a question such as, "What should I have for dinner tonight?"

[1538] The device receives the user's voice input and converts the voice into text using a speech recognition means. The specific technology used in this process is known as a general speech recognition API, for example, Google Cloud's Speech-to-Text API. The converted text data, "What should we have for dinner tonight?", is sent to the server via a transmission means.

[1539] Suggestion generation and voice response

[1540] The server uses a reference means to obtain the received text data and the user's pre-registered information, and generates an appropriate menu using a generative AI model. This generation means uses, for example, OpenAI's GPT series. The server generates a suggestion such as "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and sends this suggestion to the terminal via a transmission means.

[1541] The device converts the received suggested text into speech using a speech synthesis method, for example, Google Cloud's Text-to-Speech API, and the synthesized speech is communicated to the user through earphones: "How about grilled fish? We suggest salmon fillet, vegetables, and ingredients for miso soup."

[1542] Fix requests and iterations

[1543] If the user is not satisfied with the proposed menu, they can say, "Is there an easier dish?" again. The device will convert the speech to text again and send a new request, "Is there an easier dish?" to the server.

[1544] The server analyzes the new request, generates a new suggestion, "How about a quick chicken stir fry and salad?", and sends it to the device. The device converts the new suggestion from text to speech and tells the user, "How about a quick chicken stir fry and salad?" This process can be repeated until the user is satisfied.

[1545] Specific examples

[1546] Example 1: A scene where the user is in a supermarket

[1547] The user speaks, "What should I have for dinner tonight?" The device converts the speech into text and sends "What should I have for dinner tonight?" to the server.

[1548] Based on the user information, the server generates a request saying, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device. The device converts the suggestion into voice and conveys it to the user. If the user says, "That sounds good, but is there an easier dish?", the device sends a new request to the server, and the server generates, "Well, how about a simple chicken saute and salad?", which the device converts into voice again and conveys to the user. This process is repeated to determine a menu that satisfies the user.

[1549] Prompt Sentence Examples

[1550] 1. User prompt: "What should I have for dinner tonight?"

[1551] 2. User prompt: "Give me some suggestions for easier meals."

[1552] As described above, this system provides support for users to efficiently decide on appropriate menus in supermarkets and to carry out shopping smoothly.

[1553] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1554] Step 1: User Enters Information

[1555] Users download the smartphone app and tap the "Create a new account" button. They then enter information such as the number of family members, likes and dislikes, allergy information, and health status. The smartphone's touchscreen is used as the input method. The input data includes the number of family members, likes and dislikes, allergy information, and health status, and this data is temporarily stored on the device.

[1556] Step 2: Save the data to the server

[1557] The entered user information is sent from the terminal to the server. As a transmission method, the terminal's network function is used to attach a security token and send the data to the server. The server stores the received data in a database using a storage method. This allows the user information to be used later when generating menus.

[1558] Step 3: Dictation in Supermarket

[1559] The user arrives at the supermarket, plugs earphones into the device, and launches the app. They select the voice input mode and say, "What should I have for dinner tonight?" This speech is converted into text by a speech recognition tool. The input is the user's voice, and the output is text data.

[1560] Step 4: Send the audio data to the server

[1561] The terminal transmits text data to the server. The data received by the server is the text "What should we have for dinner tonight?" converted by the speech recognition means. The network function is used as the transmission means.

[1562] Step 5: The server generates the menu

[1563] The server retrieves the received text data and the user's pre-registered information through a reference method, and generates an appropriate menu using a generative AI model. The input is the text data and user information, and the output is a suggestion such as, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup."

[1564] Step 6: Send the generated proposal to your device

[1565] The generated menu information is sent from the server to the terminal. The network function is used as the transmission means. The generated menu information is output in text format.

[1566] Step 7: Convert the server's proposal to speech

[1567] The device converts the received text data into speech using a speech synthesis means. Google Cloud's Text-to-Speech API is used as an example of the speech synthesis means. The input is text data and the output is speech data.

[1568] Step 8: Communicate the generated audio to the user

[1569] The synthesized menu information is transmitted to the user through earphones, and the output is audio information that the user hears.

[1570] Step 9: User makes correction request

[1571] If the user is dissatisfied with the proposed menu, they can say, "Is there an easier dish?" This speech is again converted into text by the speech recognition means. The input is the user's voice, and the output is text data.

[1572] Step 10: Send a new request to the server

[1573] The terminal sends new text data to the server. The network function is used as the transmission method. The input is the text data, and the output is the transmission to the server.

[1574] Step 11: The server generates the menu again

[1575] The server analyzes the new request and generates new suggestions again using the generative AI model. The input is the text data of the new request, and the output is the new suggestion result.

[1576] Step 12: Send the new proposal to your device and convert it

[1577] The server sends the new suggestion to the device, which then converts it into speech using a speech synthesiser. The output is the new speech suggestion. This process is repeated until the suggestion is communicated to the user.

[1578] (Application example 1)

[1579] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1580] Currently, when shopping in a brick-and-mortar store, users spend a great deal of time and effort deciding on an appropriate meal plan, taking into account individual factors such as the number of people in their family, their likes and dislikes, allergies, and health conditions. This process is inefficient and can be stressful, especially for users who find meal planning difficult. Furthermore, current technology lacks a system that provides real-time meal suggestions optimized for each user's individual needs, and there is a need to solve this problem.

[1581] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1582] In this invention, the server includes a speech recognition means for converting user input speech into text, a transmission means for transmitting the text data to the server, a generation means for generating a menu based on the user input information and pre-registered information, a transmission means for transmitting the generated menu information to the terminal, a speech synthesis means for converting the received menu information into speech, an output means for conveying the speech-synthesized menu information to the user, and a prompt generation means for generating prompt sentences for the generative AI model using user profile information. This allows users to receive real-time menu suggestions optimized for their individual needs in physical stores, improving shopping efficiency and reducing stress.

[1583] A "speech recognition means" is a device or technology that converts user-supplied speech into text.

[1584] "Transmission means" refers to a device or technology for transmitting text data to a server.

[1585] "Generation means" refers to a device or technology for generating a menu based on user input information and pre-registered information.

[1586] The "voice synthesis means" is a device or technology for converting received menu information into voice.

[1587] "Output means" refers to a device or technology for conveying the synthesized menu information to the user.

[1588] A "prompt generator" is a device or technology that uses user profile information to generate prompt sentences for a generative AI model.

[1589] "User profile information" refers to information such as the number of people in a user's family, their likes and dislikes, allergies, and health status.

[1590] The system of the present invention is designed to enable users to decide on a menu while interacting with AI in real time while shopping in a physical store. This system is composed of multiple means, such as voice recognition, data transmission, menu generation, voice synthesis and output, and is equipped with a means for generating prompts using user profile information. Specific embodiments are described below.

[1591] Components

[1592] User Registration

[1593] User: Downloads the smartphone app and creates an account. When creating an account, the user enters information such as the number of people in their family, their likes and dislikes, allergies, and health conditions. This information is used to suggest more appropriate meals for future user queries.

[1594] Device: The entered information is temporarily stored in local storage and sent to the server.

[1595] Server: The received information is stored in a database, which is then referenced when creating a menu.

[1596] Voice input and request sending

[1597] User: Arrives at a physical store and connects earphones to the device. Launches the smartphone app and selects voice input mode. Enters a question: "What should I have for dinner tonight?"

[1598] Terminal: Converts the user's voice into text (speech recognition), and sends the converted text data to the server.

[1599] Server: Refers to the text data and pre-registered user profile information and generates an appropriate menu using a generative AI model. At this time, a prompt generation means generates a prompt sentence based on the user profile information.

[1600] Suggestion generation and voice response

[1601] Server: Sends the generated menu information in text format to the terminal.

[1602] Device: Converts received text data into speech (speech synthesis), and transmits the synthesized menu information to the user through earphones.

[1603] Fix requests and iterations

[1604] User: If the user is not satisfied with the proposed menu, they can say something like, "Is there an easier dish?" This speech is converted to text again and sent to the server.

[1605] Server: Re-parses the new request, generates a new proposal and sends it to the device.

[1606] The device converts the new suggestion into speech and relays it to the user. This process can be repeated until the user is satisfied.

[1607] Specific examples

[1608] A scene where the user is in a physical store

[1609] User: "What should I have for dinner tonight?"

[1610] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[1611] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[1612] Device: Converts to speech and tells the user, "How about grilled fish? We'll suggest salmon fillets, vegetables, and ingredients for miso soup."

[1613] User: "That's great, but is there anything easier?"

[1614] Terminal: Sends a new request to the server.

[1615] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[1616] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[1617] Prompt Sentence Examples

[1618] An example of a prompt used for a generative AI model is as follows:

[1619] The user's family consists of four people and they prefer Japanese food. They have an allergy to nuts and are seeking a low-fat diet due to their health condition.

[1620] Question: "What should I have for dinner tonight?"

[1621] Example response: "How about grilled fish? Suggestions include salmon fillet, vegetables, and miso soup ingredients."

[1622] This prompt is used as an instruction to ask the generative AI model to suggest appropriate menu items based on the user's background information and current question.

[1623] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1624] Step 1:

[1625] A user downloads a smartphone app and creates an account. When creating an account, they enter information such as the number of family members, likes and dislikes, allergies, and health conditions. This information is saved as user profile information. The input comes from the user, and the output is user profile information saved on the device.

[1626] Step 2:

[1627] The terminal temporarily stores the user profile information in local storage and then transmits it to the server. The input is the user profile information and the output is transmission to the server. The terminal processes the data and transmits it to the server according to the communication protocol.

[1628] Step 3:

[1629] The server stores the received user profile information in a database. This information is referenced later when generating menus. The input is the user profile information sent from the terminal, and the output is the information stored in the database.

[1630] Step 4:

[1631] The user arrives at the store and connects earphones to the device. They launch the smartphone app and select voice input mode. They then speak a question, such as "What should I have for dinner tonight?" The input is the user's voice, and the output is the start of voice recognition.

[1632] Step 5:

[1633] The device converts the user's voice into text. Using a speech recognition method (e.g., Google's speech recognition service), the voice data is converted into text data. The input is the user's voice, and the output is text data.

[1634] Step 6:

[1635] The terminal transmits the converted text data to the server. The transmission means is used to transmit the data securely to the server in accordance with the transmission protocol. The input is the converted text data, and the output is transmission to the server.

[1636] Step 7:

[1637] The server references the text data along with pre-registered user profile information, providing the basic data for generating an appropriate menu. The input is the text data and user profile information, and the output is analysis preparation based on the profile.

[1638] Step 8:

[1639] The server uses the prompt generation means to generate a prompt sentence based on the user profile information. The prompt sentence to be used for the generative AI model is generated. The input is the user profile information and the question text data, and the output is the generated prompt sentence.

[1640] Step 9:

[1641] The server uses a generative AI model to generate an appropriate menu based on the prompt. The generated menu information is sent from the server to the device in text format. The input is the prompt, and the output is the generated menu information.

[1642] Step 10:

[1643] The device converts the text data received into speech. It uses a speech synthesis tool (e.g., the pyttsx3 library) to convert the text data into speech, and transmits the generated speech to the user through headphones. The input is the text data received from the server, and the output is a synthesized speech log.

[1644] Step 11:

[1645] If the user is dissatisfied with the proposed menu, they can input a second voice request, such as "Is there an easier dish?" This voice request is converted back into text and sent to the server. The input is a voice request from the user, and the output is again text data.

[1646] Step 12:

[1647] The server re-analyzes the new request, generates new suggestions, and sends them to the terminal. This process is repeated until the user is satisfied. The input is the text data of the new request, and the output is the newly generated menu information.

[1648] This series of steps allows users to efficiently decide on the right menu in a physical store. Specifically, a system is provided in which voice input, text conversion, data transmission, prompt generation, AI menu generation, and voice output all work together seamlessly.

[1649] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1650] This invention is a system that allows users to consult with AI in real time while shopping at the supermarket to decide on a menu, and by combining it with an emotion engine, it is possible to realize menu suggestions that take the user's emotions into consideration. This system is composed of multiple means, such as voice recognition, data transmission, AI generation, voice synthesis, emotion recognition and output. Specific embodiments are described below.

[1651] User Registration

[1652] User: Downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, likes and dislikes, allergies, health conditions, etc. This information is used to suggest more appropriate meals for future user queries.

[1653] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[1654] Server: Stores the received user information in a database. This stored information is referenced later when generating menus.

[1655] Voice input and request sending

[1656] User: Arrives at the supermarket and connects earphones to the device. Launches the smartphone app again and selects voice input mode. Voice input of a question through the earphones: "What should I have for dinner tonight?"

[1657] Device: Converts the user's speech into text (performs speech recognition processing).

[1658] Terminal: Sends the converted text data to the server.

[1659] Server: Along with the text data, the server references the user's pre-registered information (number of family members, likes and dislikes, allergies, and health status).

[1660] Emotion Recognition and Suggestion Generation

[1661] Server: Analyzes emotional information from voice data using emotion recognition tools, detects the user's emotional state, and makes appropriate suggestions based on that.

[1662] Server: Using a generative AI model, it generates an appropriate menu, taking into account the user's input information, pre-registered information, and detected emotional state.

[1663] Server: The generated menu information is sent to the terminal in text format.

[1664] Suggestion generation and voice response

[1665] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[1666] Device: Menu information is synthesized and transmitted to the user through earphones.

[1667] Fix requests and iterations

[1668] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[1669] Device: Converts re-inputted speech back into text.

[1670] Terminal: Re-send the converted text data to the server.

[1671] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[1672] Server: Sends new menu information in text format to the terminal.

[1673] Terminal: Converts the received text data back into speech.

[1674] Device: New menu information is synthesized and communicated to the user through earphones.

[1675] Specific examples

[1676] Example 1: A scene where the user is in a supermarket

[1677] User: "What should I have for dinner tonight?"

[1678] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[1679] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[1680] Server: Analyzes the user's emotions from their voice and, if it determines that the user is a little tired, adjusts the suggestions to say, "How about some easy grilled fish today?"

[1681] Device: Converts into speech and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[1682] User: "That's great, but is there anything easier?"

[1683] Terminal: Sends a new request to the server.

[1684] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[1685] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[1686] This series of steps allows users to efficiently decide on the appropriate menu at the supermarket and make their shopping experience smoother. In addition, by combining it with emotion recognition, it becomes possible to make suggestions that take into account the user's mental state, providing even greater convenience.

[1687] The processing flow will be explained below.

[1688] Step 1:

[1689] User: Download the smartphone app and open the account registration screen. Enter your name, number of family members, likes and dislikes, allergies, health conditions, etc.

[1690] Step 2:

[1691] Device: The entered information is temporarily stored in local storage.

[1692] Step 3:

[1693] Terminal: To send locally stored information to the server, a communication protocol is used to securely encrypt the data and process the transmission.

[1694] Step 4:

[1695] Server: Stores the received user information in a database.

[1696] Step 5:

[1697] User: Arrives at the supermarket, plugs in earphones, restarts the smartphone app, and selects voice input mode.

[1698] Step 6:

[1699] User: "What should I have for dinner tonight?" is spoken through the earphones.

[1700] Step 7:

[1701] Device: Converts the user's speech into text (performs speech recognition processing).

[1702] Step 8:

[1703] Terminal: Sends the converted text data to the server.

[1704] Step 9:

[1705] Server: Analyzes the text data and references the user's pre-registered information (number of family members, likes and dislikes, allergies, health status).

[1706] Step 10:

[1707] Server: Analyzes the user's emotional information from the voice data using emotion recognition means.

[1708] Step 11:

[1709] Server: Uses a generative AI model to generate appropriate menus based on the user's input information, pre-registration information, and detected emotional state.

[1710] Step 12:

[1711] Server: The generated menu information is sent to the terminal in text format.

[1712] Step 13:

[1713] Terminal: Converts received text data into speech (performs text-to-speech synthesis processing).

[1714] Step 14:

[1715] Device: Menu information is synthesized and transmitted to the user through earphones.

[1716] Step 15:

[1717] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[1718] Step 16:

[1719] Device: Converts re-inputted speech back into text.

[1720] Step 17:

[1721] Terminal: Re-send the converted text data to the server.

[1722] Step 18:

[1723] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[1724] Step 19:

[1725] Server: Sends new menu information in text format to the terminal.

[1726] Step 20:

[1727] Terminal: Converts the received text data back into speech.

[1728] Step 21:

[1729] Device: New menu information is synthesized and communicated to the user through earphones.

[1730] Example 2

[1731] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1732] Conventional menu suggestion systems typically convert a user's voice input into text, then send the text data to a server to generate a menu. However, these systems do not take into account the user's emotional state or detailed personal information registered in advance, making it difficult to suggest an optimal menu that matches the user's situation and emotions. Furthermore, the process of modifying the menu in response to user requests tends to be cumbersome, resulting in a lack of convenience.

[1733] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1734] In this invention, the server includes a means for generating a menu based on user input information, pre-registered information, and emotional information, a means for converting the received menu information into audio, and an output means for conveying the information to the user. This allows for optimal menu suggestions that take into account the user's emotional state and pre-registered information. Furthermore, the system allows for quick and flexible menu revisions in response to user requests, greatly improving user convenience.

[1735] "Speech recognition means" is a technology that analyzes the voice input by the user and converts it into text data.

[1736] The "transmission means" is a technology for transmitting the text data converted by the voice recognition means to the server.

[1737] The "generation means" is a technology that generates an appropriate menu based on the received user input information, pre-registered information, and emotional information.

[1738] The "transmission means" (the means for transmitting the generated menu information) is a technique for transmitting the generated menu information to the terminal.

[1739] The "voice synthesis means" is a technology that converts received menu information into voice format.

[1740] The "output means" is a technology for transmitting the voice generated by the voice synthesis means to the user.

[1741] The "storage means" is a technology that stores pre-registered information such as the number of family members, likes and dislikes, allergy information, and health status in a database.

[1742] The "reference means" is a technology that obtains necessary information from the storage means when generating a menu and provides it to the generation means.

[1743] "Emotional information" is data that indicates the user's emotional state, analyzed based on the user's voice and other input data.

[1744] This invention is a system that allows users to consult with AI in real time while shopping at a supermarket to decide on a menu. It also combines an emotion engine to realize menu suggestions that take the user's emotions into consideration. This system is composed of multiple means, including voice recognition, data transmission, AI generation, voice synthesis, emotion recognition and output. Specific embodiments are shown below.

[1745] Hardware and Software Configuration

[1746] Users access the system using devices such as smartphones and tablets, which have applications installed for voice recognition, text data transmission, and voice data output.

[1747] The device uses the Google Cloud Speech-to-Text API for speech recognition, parses the voice input, and converts it into text data, which is then converted into JSON format and sent to a server over the internet.

[1748] The server is hosted in a cloud environment and is responsible for performing multiple functions. The server runs a generative AI model (e.g., GPT-3) to generate appropriate menus based on text data sent by users and pre-registered user information. It also uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state.

[1749] The generated menu information is sent from the server to the device in JSON format. The device receives this JSON data and converts it into text format. Next, it uses the Google Cloud Text-to-Speech API for speech synthesis to convert the text data into audio data. The converted audio data is played back to the user through headphones.

[1750] Specific examples

[1751] Example 1: A scene where the user is in a supermarket

[1752] The user speaks into the smartphone's microphone, asking, "What should I have for dinner tonight?"

[1753] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[1754] After analyzing the user information, the server generates a message saying, "How about grilled fish? We suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the terminal.

[1755] The server analyzes the user's emotions from their voice, and if it determines that the user is a little tired, it adjusts its suggestions, saying, "How about some easy grilled fish today?"

[1756] The device converts the information into voice and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[1757] The user makes another request, saying, "That's good, but is there anything a little easier?"

[1758] The terminal sends a new request to the server.

[1759] The server generates a new suggestion and sends it to the device: "How about a quick chicken stir fry and salad?"

[1760] The device converts the information into speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[1761] Prompt Sentence Examples

[1762] User information: "Name: Sato, Family size: 3, Likes and dislikes: Likes spicy food, Allergies: None, Health condition: Good, Emotions: A little tired"

[1763] User Question: "What should I have for dinner tonight?"

[1764] Suggested menu:

[1765] Step 1: "How about making a simple grilled fish dish for dinner tonight? We suggest a salmon fillet, vegetables, and miso soup."

[1766] Step 2: "If grilled fish seems too difficult, how about sautéed chicken and a salad?"

[1767] This allows users to efficiently decide on the appropriate menu at the supermarket and make shopping a smoother experience. In addition, by combining emotion recognition, suggestions can be made that take into account the user's mental state, providing even greater convenience.

[1768] The above is a specific embodiment for carrying out the present invention.

[1769] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1770] Step 1: User Registration

[1771] The user downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, likes and dislikes, allergies, health conditions, etc. This information is then entered into the app's input form. Once the information is complete, the user presses the "Register" button.

[1772] Input: User's personal information (name, number of family members, likes and dislikes, allergies, health status)

[1773] Output: User information data in JSON format

[1774] The device temporarily saves the input information in local storage and sends it to the server. Specifically, the input data is converted to JSON format and sent to the server via an HTTP POST request.

[1775] Input: JSON format user information data

[1776] Output: HTTP request sent to the server

[1777] The server saves the received user information in the database. The server parses the received JSON data and stores the necessary information in the appropriate table in the database.

[1778] Input: JSON data sent in the HTTP request to the server

[1779] Output: User information record in database

[1780] Step 2: Speak and send your request

[1781] The user arrives at the supermarket, connects the earphones to the device, restarts the smartphone app, selects voice input mode, and inputs a question through the earphones: "What should I have for dinner tonight?"

[1782] Input: User's voice data

[1783] Output: Audio data is input to the device

[1784] The device converts the user's voice into text using a speech recognition API (e.g., Google Cloud Speech-to-Text), sends the voice data to the API, and retrieves the converted text data.

[1785] Input: Audio data

[1786] Output: Text data converted from audio

[1787] The device sends the converted text data to the server, where it is converted to JSON format along with the user ID and sent via an HTTP POST request to the server.

[1788] Input: Text data, user ID

[1789] Output: HTTP request sent to the server

[1790] The server retrieves the user's pre-registered information (number of family members, likes and dislikes, allergies, health status) and emotional information from a database to generate a menu based on the text data and pre-registered information.

[1791] Input: Text data, user ID

[1792] Output: User details (number of family members, likes and dislikes, allergies, health status)

[1793] Step 3: Emotion recognition and suggestion generation

[1794] The server uses the acquired voice data to perform emotion recognition. It sends the voice data to an emotion recognition API (e.g., IBM Watson Tone Analyzer) and analyzes the user's emotional state.

[1795] Input: Audio data

[1796] Output: Emotional information (e.g., "tired")

[1797] The server uses a generative AI model (e.g., GPT-3) to generate an appropriate menu based on the user's input, pre-registered information, and the detected emotional state. The prompt text includes the user's name, number of family members, likes and dislikes, allergy information, health status, emotional state, and voice input text and is input into the model.

[1798] Input: User input information, pre-registration information, emotional information

[1799] Output: Generated menu information (text format)

[1800] The server converts the generated menu information into JSON format and sends it to the terminal as an HTTP response.

[1801] Input: Generated menu information

[1802] Output: HTTP response sent to the device

[1803] Step 4: Proposal generation and voice response

[1804] The device converts the received text data into speech using a speech synthesis API (e.g., Google Cloud Text-to-Speech). The device sends the text data to the API and receives the speech data.

[1805] Input: Received text data

[1806] Output: Synthesized voice data

[1807] The device transmits the menu information to the user through a headset, and by outputting the audio data to the headset, the user can receive suggestions by voice.

[1808] Input: Synthesized voice data

[1809] Output: Audio output to the user

[1810] Step 5: Request revisions and iterate

[1811] If the user is dissatisfied with the proposed menu, he or she can again input a voice command such as "Is there an easier dish?"

[1812] Input: User's voice data

[1813] Output: Audio data is input to the device

[1814] The device again uses the speech recognition API to convert the speech to text, sends the speech data to the API, and retrieves the converted text data.

[1815] Input: Audio data

[1816] Output: Text data converted from audio

[1817] The device sends the converted text data to the server again, converts the text data into JSON format, and sends it to the server via an HTTP POST request.

[1818] Input: Text data, user ID

[1819] Output: HTTP request sent to the server

[1820] The server reanalyzes the new request data and generates a new menu. The data is input into the generative AI model, which generates new suggestions. The server also reanalyzes changes in the user's emotions and optimizes the suggestions.

[1821] Input: New text data, pre-registration information, emotion information

[1822] Output: New menu information (text format)

[1823] The server converts the new menu information back into JSON format and sends it to the terminal as an HTTP response.

[1824] Input: New menu information

[1825] Output: HTTP response sent to the device

[1826] The device converts the received text data into speech using the speech synthesis API. The device sends the text data to the API and obtains the speech data.

[1827] Input: Received text data

[1828] Output: Synthesized voice data

[1829] The device then sends the new menu information via voice synthesis to the user through earphones, allowing the user to receive new suggestions by voice by outputting the audio data to the earphones.

[1830] Input: Synthesized voice data

[1831] Output: Audio output to the user

[1832] (Application example 2)

[1833] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1834] Existing menu suggestion systems can suggest general menus based on user input, but they are unable to consider the user's mental state or emotions. This makes it difficult to provide optimal menus when the user is tired or in a particular mood, potentially reducing user satisfaction. They also lack the flexibility to respond to re-suggestions. Therefore, there is a need for a system that can suggest menus that take the user's emotional state into account and improve the user's mental satisfaction.

[1835] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion analysis means that uses an emotion recognition means to analyze the user's emotional state and optimize menu suggestions based on that analysis, a voice synthesis means that audibly conveys the generated menu information to the user, and a re-suggestion means that generates new menu information in response to a user's request for revision and conveys it to the user again. This makes it possible to propose an optimal menu that takes the user's emotional state into consideration, thereby improving psychological satisfaction.

[1836] "Speech recognition means" refers to technology that converts the voice input by the user into text.

[1837] "Transmission means" refers to a technique for transmitting text data to a server.

[1838] The "generation means" is a technology that allows the server to generate a menu based on information entered by the user and information registered in advance.

[1839] The "voice synthesis means" is a technology for converting the generated menu information into voice.

[1840] "Output means" refers to technology that conveys menu information to users in the form of synthesized voice.

[1841] "Emotion recognition means" is a technology that analyzes the user's emotional state.

[1842] "Emotion analysis means" is a technology that uses emotion recognition means to analyze the user's emotional state and optimizes menu suggestions based on that.

[1843] "Storage means" refers to the technology that stores information such as the number of family members, likes and dislikes, allergy information, and health status in a database.

[1844] The "reference means" is a technique for obtaining information from the storage means when creating a menu.

[1845] The "means for generating a prompt sentence" is a technique for generating a prompt sentence based on the emotional state of the user.

[1846] This invention is a system that allows users to decide on a menu while shopping in a physical store, consulting with AI in real time. By incorporating emotion recognition, it is possible to make suggestions that take into account the user's mental state. This system is configured as follows.

[1847] User Registration

[1848] User: Downloads the smartphone app and opens the account registration screen. They enter their name, number of family members, preferences, allergy information, health conditions, etc. This information is used to suggest more appropriate meals for future user queries.

[1849] Terminal: The entered information is temporarily stored in local storage and sent to the server.

[1850] Server: Stores the received user information in a database. This stored information is referenced later when generating menus.

[1851] Voice input and request sending

[1852] User: Arrives at the store and connects the earphones to the device. Relaunches the smartphone app and selects voice input mode. Voice input of a question through the earphones: "What should I have for dinner tonight?"

[1853] On the device: Converts the user's speech into text. The "SpeechRecognition" library is used to implement the speech recognition method.

[1854] Terminal: Sends the converted text data to the server.

[1855] Emotion Recognition and Suggestion Generation

[1856] Server: Analyzes emotional information from voice data using emotion recognition. Detects the user's emotional state and makes appropriate suggestions based on that. A "generative AI model" is used to generate an appropriate menu, taking into account the user's input information, pre-registered information, and the detected emotional state.

[1857] Server: The generated menu information is sent to the terminal in text format.

[1858] Suggestion generation and voice response

[1859] Terminal: Converts received text data into speech. The "gTTS" library is used to perform text-to-speech synthesis processing.

[1860] Device: Menu information is synthesized and transmitted to the user through earphones.

[1861] Fix requests and iterations

[1862] User: If you are not satisfied with the suggested menu, you can say something like, "Is there an easier dish?"

[1863] Terminal: Converts the re-inputted voice into text and sends it back to the server.

[1864] Server: Re-analyzes the new request data and generates a new menu. It also measures changes in the user's emotions and optimizes the suggestions.

[1865] Server: Sends new menu information to the terminal.

[1866] Terminal: Converts received text data into speech.

[1867] Device: New menu information is synthesized and communicated to the user through earphones.

[1868] Specific examples

[1869] The following is the sequence of events when a user is in a physical store:

[1870] User: "What should I have for dinner tonight?"

[1871] Device: After converting the speech to text, send "What should I have for dinner tonight?" to the server.

[1872] Server: After analyzing the user information, it generates a message saying, "How about grilling some? We'll suggest salmon fillets, vegetables, and ingredients for miso soup," and sends it to the device.

[1873] Server: Analyzes the user's emotions from their voice and, if it determines that the user is a little tired, adjusts the suggestions to say, "How about some easy grilled fish today?"

[1874] Device: Converts into speech and tells the user, "How about grilling some fish? We'll suggest some salmon fillets, vegetables, and ingredients for a simple miso soup."

[1875] User: "That's great, but is there anything easier?"

[1876] Terminal: Sends a new request to the server.

[1877] Server: Generate a new suggestion and send it to the device: "How about a quick chicken stir fry and salad?"

[1878] Device: Converts to speech and tells the user, "So, how about a quick chicken stir fry and salad?"

[1879] Prompt Sentence Examples

[1880] "What should I have for dinner tonight?"

[1881] By implementing this, users can efficiently decide on the appropriate menu in a physical store and shop smoothly. In addition, by combining it with emotion recognition, it becomes possible to make suggestions that take into account the user's mental state, providing even greater convenience.

[1882] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1883] Step 1:

[1884] A user downloads the smartphone app and opens the account registration screen. They enter information such as their name, number of family members, preferences, allergies, and health conditions into a form within the app. Based on this information, the device temporarily saves it in local storage and then sends it to the server. The server then saves the received user information in a database.

[1885] Step 2:

[1886] The user arrives at the store and connects the earphones to their smartphone. Next, they launch the smartphone app and select voice input mode. The user then speaks a question into the earphones, such as "What should I have for dinner tonight?" The voice data is then imported into the smartphone device.

[1887] Step 3:

[1888] The device uses the SpeechRecognition library to convert the captured voice data into text. Specifically, the voice data is input into an acoustic model, and patterns that match the phonemes are analyzed to generate text data. This converted text data is then sent to the server.

[1889] Step 4:

[1890] The server references the received text data and pre-registered user information (family size, likes and dislikes, allergies, and health status). It then uses emotion recognition to analyze the user's emotional state from the voice data. A generative AI model generates an optimal menu plan that takes into account the user's input information and emotional state.

[1891] Step 5:

[1892] The server then sends the suggestions obtained from the generative AI model in text format to the device, where the suggestions are adjusted, such as suggesting less strenuous meals if the emotional state is evaluated as "tired."

[1893] Step 6:

[1894] The device uses the gTTS library to convert the received text data into voice data. During this process, a synthetic speech engine analyzes the text data phonetically and generates a natural speaking voice. The generated voice data is transmitted to the user through earphones.

[1895] Step 7:

[1896] If the user is dissatisfied with the proposed menu, they can input a question by voice again, such as "Is there an easier dish?" The new voice data is then imported into the device.

[1897] Step 8:

[1898] The device then uses the SpeechRecognition library to convert the newly input voice data into text, which is then sent back to the server.

[1899] Step 9:

[1900] The server re-analyzes the new request data and generates new menu suggestions. Using the generative AI model and sentiment analysis, it re-measures the user's latest emotional state and optimizes the suggestions accordingly. The new suggestions are generated and sent to the device.

[1901] Step 10:

[1902] The device then uses the gTTS library again to convert the new text suggestions into audio, which is then transmitted to the user through the earphones.

[1903] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1904] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1905] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1906] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1907] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1908] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1909] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1910] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1911] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1912] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1913] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1914] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1915] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1916] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1917] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1918] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1919] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1920] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1921] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1922] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1923] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1924] The following is further disclosed regarding the above embodiment.

[1925] (Claim 1)

[1926] a speech recognition means for converting user input into text;

[1927] a transmitting means for transmitting the text data to a server;

[1928] A generation means for the server to generate a menu based on the user's input information and pre-registered information;

[1929] A transmitting means for transmitting the generated menu information to a terminal;

[1930] a speech synthesis means for converting the received menu information into speech;

[1931] an output means for transmitting the synthesized voice menu information to a user;

[1932] A system including:

[1933] (Claim 2)

[1934] The system according to claim 1, further comprising a process in which the user inputs a request for correction by voice, converts the request into text by a voice recognition means, transmits the text again to the server, generates new menu information, and notifies the user again.

[1935] (Claim 3)

[1936] The system of claim 1 includes a storage means for storing pre-registered information such as the number of family members, likes and dislikes, allergy information, and health condition of the user in a database, and a reference means for retrieving information from the storage means when generating a menu.

[1937] "Example 1"

[1938] (Claim 1)

[1939] a speech recognition means for converting user input into text;

[1940] a transmitting means for transmitting the text data to a server;

[1941] A generation means for the server to generate a menu based on the user's input information and pre-registered information;

[1942] A transmitting means for transmitting the generated menu information to a terminal;

[1943] a speech synthesis means for converting the received menu information into speech;

[1944] an output means for transmitting the synthesized voice menu information to a user;

[1945] an input means for a user to input registration information;

[1946] a storage means for storing the input information;

[1947] A system including:

[1948] (Claim 2)

[1949] The system according to claim 1, further comprising a process in which the user inputs a request for correction by voice, converts the request into text by a voice recognition means, transmits the text again to the server, generates new menu information, and notifies the user again.

[1950] (Claim 3)

[1951] The system of claim 1 includes a storage means for storing pre-registered information such as the number of family members, likes and dislikes, allergy information, and health condition of the user in a database, and a reference means for retrieving information from the storage means when generating a menu.

[1952] "Application Example 1"

[1953] (Claim 1)

[1954] a speech recognition means for converting user input into text;

[1955] a transmitting means for transmitting the text data to a server;

[1956] A generation means for the server to generate a menu based on the user's input information and pre-registered information;

[1957] A transmitting means for transmitting the generated menu information to a terminal;

[1958] a speech synthesis means for converting the received menu information into speech;

[1959] an output means for transmitting the synthesized voice menu information to a user;

[1960] a prompt generation means for generating a prompt sentence for the generative AI model using the user profile information;

[1961] A system including:

[1962] (Claim 2)

[1963] The system according to claim 1, further comprising a process in which the user inputs a request for correction by voice, converts the request into text by a voice recognition means, transmits the text again to the server, generates new menu information, and notifies the user again.

[1964] (Claim 3)

[1965] The system of claim 1 includes a storage means for storing pre-registered information such as the number of family members, likes and dislikes, allergy information, and health condition of the user in a database, a reference means for retrieving information from the storage means when generating a menu, and a prompt generation means for dynamically generating prompt sentences for the generation AI model.

[1966] "Example 2: Combining Emotion Engines"

[1967] (Claim 1)

[1968] a speech recognition means for converting user input into text;

[1969] a transmitting means for transmitting the text data to a server;

[1970] A generation means for generating a menu based on the user's input information, pre-registered information, and emotion information;

[1971] A transmitting means for transmitting the generated menu information to a terminal;

[1972] a speech synthesis means for converting the received menu information into speech;

[1973] and an output means for transmitting the synthesized menu information to the user.

[1974] (Claim 2)

[1975] The system according to claim 1, further comprising a process in which the user inputs a request for correction by voice, converts the request into text by a voice recognition means, transmits the text again to the server, generates new menu information, and notifies the user again.

[1976] (Claim 3)

[1977] The system of claim 1 includes a storage means for storing pre-registered information such as the number of family members, likes and dislikes, allergy information, and health condition of the user in a database, and a reference means for acquiring information from the storage means when generating a menu and passing it to the generation means.

[1978] "Application example 2 when combining emotion engines"

[1979] (Claim 1)

[1980] a speech recognition means for converting user input into text;

[1981] a transmitting means for transmitting the text data to a server;

[1982] A generation means for the server to generate a menu based on the user's input information and pre-registered information;

[1983] A transmitting means for transmitting the generated menu information to a terminal;

[1984] a speech synthesis means for converting the received menu information into speech;

[1985] an output means for transmitting the synthesized voice menu information to a user;

[1986] an emotion analysis means for analyzing the user's emotional state using an emotion recognition means and optimizing menu suggestions based on the analysis;

[1987] A system including:

[1988] (Claim 2)

[1989] The system according to claim 1, further comprising a process in which the user inputs a request for correction by voice, converts the request into text by a voice recognition means, transmits the text again to the server, generates new menu information, and notifies the user again.

[1990] (Claim 3)

[1991] The system of claim 1 includes a storage means for storing pre-registered information such as the number of family members, likes and dislikes, allergy information, and health condition of the user in a database, a reference means for acquiring information from the storage means when generating a menu, and a means for generating a prompt sentence based on the user's emotional state. [Explanation of symbols]

[1992] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a speech recognition means for converting user input into text; a transmitting means for transmitting the text data to a server; A generation means for the server to generate a menu based on the user's input information and pre-registered information; A transmitting means for transmitting the generated menu information to a terminal; a speech synthesis means for converting the received menu information into speech; an output means for transmitting the synthesized voice menu information to a user; A system including:

2. 2. The system according to claim 1, further comprising a process in which a user inputs a request for correction by voice, converts the request into text by a voice recognition means, transmits the text again to the server, generates new menu information, and notifies the user again.

3. The system of claim 1 includes a storage means for storing pre-registered information such as the number of family members, likes and dislikes, allergy information, and health condition of the user in a database, and a reference means for acquiring information from the storage means when generating a menu.

Images