System

Abstract

To provide a system.SOLUTION: A system includes: first means for receiving sound input indicating a request from a user; second means for converting voice data of sound input indicating the request from the user into first text data; third means for generating a first prompt statement for outputting a response to the request from the user on the basis of the first text data; fourth means for inputting the generated first prompt statement to the generative AI model and generating the response; and fifth means for reproducing the generated response from a smart speaker and suggesting it to the user. The first means receives sound input indicating the answer of the user to the response reproduced by the smart speaker. The second means includes sixth means for converting voice data of sound input indicating the user's answer into second text data and setting smart consumer electronics shown in the response on the basis of an analysis result of the second text data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When introducing smart home appliances, it is difficult to understand which combination is optimal and what operations are possible, and the settings are also complicated. [Means for solving the problem]

[0005] Using a smart speaker linked to generative AI, the system will suggest, purchase, and assist with the setup of smart home appliances through a conversational format with the user. Furthermore, the smart speaker will automatically connect to other smart home appliances, simplifying user operation. [Brief explanation of the drawings]

[0006] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One embodiment of the present invention provides a smart speaker that works in conjunction with generative AI. This smart speaker suggests, purchases, and assists with the setup of smart home appliances through a conversation with the user. Specifically, when the user says, "I want to heat my room," the smart speaker works in conjunction with the generative AI to suggest, "A smart heater would be suitable. Would you like help setting it up?" When the user requests assistance, the smart speaker automatically connects to the smart heater and sets it up.

[0029] "Example 2"

[0030] Furthermore, generative AI can suggest optimal combinations of smart home appliances based on user requests. For example, if a user says, "I want to make coffee automatically in the morning," the smart speaker will suggest, "By combining a smart coffee maker with a smart plug, you can make coffee automatically at the specified time."

[0031] "Example 3"

[0032] Generative AI can also learn about a user's lifestyle and preferences and optimize the operation of smart home appliances accordingly. For example, if a user learns that they go to bed at 10 p.m. every night, the smart speaker will automatically turn off the smart lights at 10 p.m.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The user says to the smart speaker, "I want to heat up my room."

[0036] Step 2: The smart speaker works with the generative AI to determine the best smart appliance for the user's needs. In this case, it determines that a smart heater is suitable.

[0037] Step 3: The smart speaker will suggest to the user, "A smart heater is suitable. Would you like help setting it up?"

[0038] Step 4: When the user requests assistance, the smart speaker will automatically connect to the smart heater and set it up.

[0039] "Example 2"

[0040] Step 1: The user says to the smart speaker, "I want to automatically brew coffee in the morning."

[0041] Step 2: The smart speaker works with the generative AI to determine the optimal combination of smart appliances for the user's needs. In this case, it determines that a smart coffee maker and a smart plug are the best combination.

[0042] Step 3: The smart speaker suggests to the user, "By combining your smart coffee maker with the smart plug, you can automatically brew coffee at the time you specify."

[0043] "Example 3"

[0044] Step 1: The generative AI learns the user's habits and preferences. In this case, it learns that the user goes to bed at 10pm every night.

[0045] Step 2: The smart speaker will use the learned information to optimize the operation of smart home appliances, in this case automatically turning off the smart lights at 10 o'clock.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] Conventional smart speakers only respond to users' voice inputs and lack the functionality to suggest appropriate smart home appliances based on the user's specific needs and assist with their setup. They also lack the functionality to optimize the operation of smart home appliances based on the user's lifestyle and preferences. This requires users to manually configure the appliances, reducing convenience.

[0049] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0050] In this invention, the server includes means for receiving a user's voice input, means for converting the voice into text, means for transmitting the text data to the server, means for the server to generate a response using a generative AI model, means for returning the response to the terminal, means for the terminal to convert the response into voice and suggest it to the user, means for receiving the user's response, and means for connecting to and configuring smart home appliances. This makes it possible to suggest appropriate smart home appliances based on the user's specific needs and automatically configure them. It also makes it possible to optimize the operation of smart home appliances based on the user's lifestyle and preferences.

[0051] "Means for receiving user voice input" refers to devices or software that capture voices uttered by the user and input them into the system.

[0052] "Speech-to-text means" refers to speech recognition technology for analyzing and converting captured speech data into corresponding text data.

[0053] The "means for transmitting text data to a server" refers to a communication protocol or interface for transmitting the converted text data to a server via a network.

[0054] "Means by which the server generates a response using a generative AI model" refers to the process by which the server generates an appropriate response using a generative AI model based on the text data received by the server.

[0055] The "means for returning a response to the terminal" refers to a communication protocol or interface for returning the generated response to the terminal via the network.

[0056] The "means for the terminal to convert the response into voice and make a suggestion to the user" refers to a device or software that converts the text response received from the server into voice using speech synthesis technology and makes a suggestion to the user.

[0057] The "means for receiving user responses" refers to a device or software for capturing responses made by users to suggestions and inputting them into the system.

[0058] "Means for connecting to and configuring smart home appliances" refers to devices or software that connect to the proposed smart home appliances via a communication protocol and automatically perform the necessary settings.

[0059] This invention is a system that uses a smart speaker linked to a generative AI model to suggest, purchase, and assist with the setup of smart home appliances based on user voice input. Specific embodiments of this system are described below.

[0060] Hardware and software used

[0061] Hardware

[0062] Smart speaker: A device that accepts a user's voice input and plays audio.

[0063] Server: A computer system for hosting generative AI models and performing data processing.

[0064] Smart appliances: Home appliances that can be controlled over a network, such as a smart heater.

[0065] software

[0066] Speech Recognition Software: Technology for converting speech to text, such as the Google® Cloud Speech-to-Text API.

[0067] Generative AI model: An artificial intelligence model for generating appropriate responses based on user requests, such as OpenAI's GPT-3.

[0068] Text-to-Speech Software: Technology for converting text to speech, such as Amazon Polly®.

[0069] Communication Protocol: The protocol used to send and receive data, such as HTTPS.

[0070] System Operation

[0071] User voice input

[0072] The user says to the smart speaker, "I want to heat up the room," and the smart speaker captures the voice through its built-in microphone.

[0073] Speech-to-text

[0074] The smart speaker uses voice recognition software to convert the captured voice data into text data, which includes the phrase "I want to heat my room."

[0075] Sending text data

[0076] The smart speaker then sends the converted text data to the server using the HTTPS protocol to ensure data security.

[0077] Generating a response

[0078] The server inputs the received text data into the generative AI model, which generates an appropriate response based on the user's request. In this case, it generates the response, "A smart heater is suitable. Would you like help setting it up?"

[0079] Returning a response

[0080] The server then sends the generated response back to the smart speaker, again using the HTTPS protocol.

[0081] Playing audio

[0082] The smart speaker converts the text response received from the server into speech using speech synthesis software, and plays the converted speech from the speaker, suggesting to the user, "A smart heater is suitable. Would you like help setting it up?"

[0083] User Response

[0084] If the user responds "yes," the smart speaker again captures the audio and converts it to text using speech recognition software. This text data is then analyzed to confirm that the user is requesting assistance.

[0085] Smart appliance settings

[0086] The smart speaker automatically connects to the smart heater using communication protocols such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). Once the connection is established, the smart speaker configures the smart heater, such as setting the temperature and timer.

[0087] Examples and prompts

[0088] Specific examples

[0089] When a user says, "I want to warm up the room," the smart speaker will suggest, "A smart heater would be suitable. Would you like help setting it up?" If the user answers "yes," the smart speaker will automatically connect to the smart heater and complete the setup. For example, you can set the heater temperature to 22 degrees and set a timer to automatically turn it off after one hour.

[0090] Prompt Sentence Examples

[0091] "When a user says they want to heat their room, explain the system that suggests appropriate smart appliances and assists them in setting them up."

[0092] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0093] Step 1:

[0094] Accepting user voice input

[0095] A user says to a smart speaker, "I want to heat the room." The device (smart speaker) captures the voice through its built-in microphone. The input is the user's voice data, and the output is the captured voice data.

[0096] Step 2:

[0097] Convert speech to text

[0098] The device converts the captured voice data into text data using speech recognition software (e.g., Google Cloud Speech-to-Text API). The input is the captured voice data, and the output is the text data "I want to heat the room."

[0099] Step 3:

[0100] Send text data to the server

[0101] The terminal sends the converted text data to the server. The HTTPS protocol is used for transmission to ensure data security. The input is text data, and the output is the text data sent to the server.

[0102] Step 4:

[0103] The server generates a response using the generative AI model

[0104] The server inputs the received text data into a generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates an appropriate response based on the user's request. The input is text data, and the output is the response, "A smart heater is suitable. Would you like help setting it up?"

[0105] Step 5:

[0106] Sends the response back to the terminal

[0107] The server generates a response and sends it back to the device, again using the HTTPS protocol. The input is the generated response, and the output is the response sent back to the device.

[0108] Step 6:

[0109] The device converts the response into speech and suggests it to the user.

[0110] The device converts the text response received from the server into speech using speech synthesis software (e.g., Amazon Polly). The converted speech is played back through the speaker, suggesting to the user, "This smart heater is suitable. Would you like help setting it up?" The input is the text response, and the output is the speech response.

[0111] Step 7:

[0112] Receive the user's response

[0113] If the user responds "yes," the device again captures the voice and converts it to text using speech recognition software. This text data is then parsed to determine that the user is requesting assistance. The input is the user's voice response, and the output is text data.

[0114] Step 8:

[0115] Connect to smart home appliances and set them up

[0116] The terminal automatically connects to the smart heater using communication protocols such as Wi-Fi or Bluetooth. Once the connection is established, the terminal configures the smart heater, such as setting the temperature and timer. The input is the user's configuration request, and the output is the smart heater after the configuration is complete.

[0117] (Application example 1)

[0118] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0119] While existing smart speakers have the ability to suggest smart home appliances and assist with setup based on user needs, they lack functionality to improve the shopping experience in physical stores. As a result, users have difficulty finding the appropriate product section when searching for products in physical stores, and there is a lack of in-store navigation.

[0120] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0121] In this invention, the server includes a smart speaker linked with a generative AI, a means for the smart speaker to suggest, purchase, and set up products that can be linked with the user through a conversation, a means for the smart speaker to automatically connect with other smart home appliances, and a means for the smart speaker to suggest an appropriate product section based on the user's request in a physical store and provide in-store navigation, thereby improving the user's shopping experience in the physical store and enabling them to quickly find the appropriate product section.

[0122] "Generative AI" is an artificial intelligence technology that generates responses in natural language based on user input.

[0123] A "smart speaker" is a voice assistant device that has voice recognition capabilities and can perform various operations through conversations with the user.

[0124] "Conversational" refers to a form of interaction in which the user asks questions or gives instructions by voice, and the device responds by voice.

[0125] "Product suggestion" is the act of selecting appropriate products based on the user's requests and introducing them to the user.

[0126] "Purchase" refers to the user's action of taking steps to acquire the suggested product.

[0127] "Setup assistance" refers to the act of assisting users with initial setup and operation so that they can properly use the product they have purchased.

[0128] "Automatically connect to other smart home appliances" is a function that allows the smart speaker to communicate with and operate other smart home appliances via wireless communication based on user instructions.

[0129] "Suggesting the appropriate product section in a physical store based on the user's request" refers to the act of guiding a user to the section where a specific product is located when the user is searching for that product in a physical store.

[0130] "Providing in-store navigation" refers to a function that provides route guidance for users to reach the desired product section within a physical store.

[0131] As an embodiment of this invention, we will explain a system that uses a smart speaker linked to generative AI. This system provides product suggestions, purchasing assistance, and setup assistance through a conversation with the user, and also provides navigation within a physical store.

[0132] First, the smart speaker has a voice recognition function and converts the user's speech into text data. This text data is sent to a generative AI, which generates an appropriate response based on the user's request. The generative AI is implemented using, for example, OpenAI's API.

[0133] Generative AI suggests appropriate products based on user input. For example, if a user says, "I want warm clothes," the generative AI responds, "We have warm clothes in this section. Would you like me to show you?" This response is transmitted to the user through a smart speaker.

[0134] Next, if the user requests directions, the smart speaker will activate the in-store navigation function. This navigation function is linked to a database containing information about the sections in the store and will guide the user to the desired section. Navigation information is provided, for example, using Wi-Fi or Bluetooth beacons in the store.

[0135] As a concrete example, by inputting the following prompt sentence into a generative AI, a response that meets the user's request can be generated.

[0136] Example prompt sentence:

[0137] "If a user says, 'I want warm clothes,' generate an Assistant response that suggests appropriate products and provides navigation within the store."

[0138] The system will enhance the user shopping experience by integrating smart speakers, generative AI, and navigation functions. Users can quickly find the product section they are looking for in a physical store, and can also receive suggestions and setup assistance for smart home appliances, greatly improving user convenience.

[0139] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0140] Step 1:

[0141] A user says to a smart speaker, "I want warm clothes." The smart speaker uses its voice recognition function to convert this statement into text data. The input is the user's voice data, and the output is text data.

[0142] Step 2:

[0143] The smart speaker sends the converted text data to the generative AI, which takes this text data as input and creates a prompt to generate an appropriate response. The input is text data, and the output is a prompt.

[0144] Step 3:

[0145] The generative AI generates a response based on the prompt text according to the user's request. Specifically, the generative AI generates the response, "There are warm clothes in this section. Can you show me?" The input is the prompt text, and the output is the generated response text.

[0146] Step 4:

[0147] The smart speaker converts the generated response text into speech and transmits it to the user. The user listens to the smart speaker's response and decides whether to ask for guidance. The input is the generated response text, and the output is speech data.

[0148] Step 5:

[0149] When a user requests guidance, the smart speaker activates the navigation function, which connects to a database containing information about sections in the store and generates route information to guide the user to the desired section. The input is the user's instructions, and the output is route information.

[0150] Step 6:

[0151] The smart speaker then communicates the generated route information to the user by voice and guides the user to the desired section. The input is the route information and the output is voice data.

[0152] Step 7:

[0153] The user follows the guidance of the smart speaker to move around the store and reach the desired section. The input is voice data, and the output is the user's actions.

[0154] Example 2

[0155] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0156] Conventional smart home appliance systems required users to individually configure and operate each appliance, resulting in low convenience. Furthermore, they lacked the functionality to suggest optimal combinations of appliances based on the user's needs, requiring the user to spend time and effort finding the optimal combination. Furthermore, they lacked the functionality to optimize the operation of appliances based on the user's lifestyle and preferences. To address these issues, a system is needed that can suggest optimal combinations of appliances based on the user's needs and further optimize their operation.

[0157] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0158] In this invention, the server includes a voice recognition device linked to a generative artificial intelligence, means for the voice recognition device to suggest, purchase, and assist with setting up products that can be linked in a conversational format with the user, means for the voice recognition device to automatically connect to other electronic devices, means for converting the user's voice into text data, means for generating prompt sentences for inputting the text data to the generative artificial intelligence, means for the generative artificial intelligence to suggest an optimal combination of electronic devices, and means for communicating the suggestion to the user by voice. This makes it possible to automatically suggest an optimal combination of home appliances based on the user's requests and further optimize their operation.

[0159] "Generative AI" is an AI system that generates appropriate responses and suggestions based on user input.

[0160] A "voice recognition device" is a device that converts a user's voice into text data.

[0161] "User" refers to a person who uses this system.

[0162] "Conversational" is a format in which the user and the system interact, with the system responding to what the user says.

[0163] "Product suggestion" is the act of recommending appropriate products based on the user's requests.

[0164] "Purchase" is the act of a user acquiring a suggested product.

[0165] "Setting assistance" refers to the act of providing assistance to a user in properly setting up a proposed product.

[0166] "Electronic devices" refers to digital devices such as smart home appliances.

[0167] "Text data" is character information converted by a voice recognition device.

[0168] A "prompt sentence" is an instruction sentence to be input into generative artificial intelligence.

[0169] A "suggestion" is the optimal solution provided by generative artificial intelligence based on the user's request.

[0170] The "audio means" is a method for transmitting the generated suggestions to the user as audio.

[0171] This invention is a system that uses a voice recognition device linked to generative artificial intelligence to propose the optimal combination of electronic devices based on the user's request and optimize their operation. This system is implemented using the following hardware and software.

[0172] Hardware and software used

[0173] Hardware:

[0174] 1. Voice recognition devices (e.g. smart speakers)

[0175] 2. Server (a server for hosting generative AI)

[0176] software:

[0177] 1. Speech recognition software (e.g., software that converts speech into text data)

[0178] 2. Generative AI models (e.g., OpenAI's GPT-4 (registered trademark))

[0179] Data processing and calculation

[0180] Voice recognition device:

[0181] When a user speaks a request to the voice recognition device, the voice recognition device uses built-in voice recognition software to convert the speech into text data. For example, if the user says, "I want to make coffee automatically in the morning," this speech is converted into text data that says, "I want to make coffee automatically in the morning."

[0182] server:

[0183] The server receives the text data sent from the speech recognition device. The server then generates a prompt to be input to the generative AI model. For example, this prompt might be in the form of, "The user wants to automatically make coffee in the morning. Which electronic device should I pair it with?"

[0184] The generative AI model analyzes this prompt and suggests optimal combinations of electronic devices, such as "Pairing a smart coffee maker with a smart plug will allow you to automatically brew coffee at a specified time."

[0185] The server sends the generated suggestions to a speech recognizer, which converts the suggestions into speech using speech synthesis software. Ultimately, the speech recognizer tells the user, "By combining your smart coffee maker with your smart plug, you can automatically brew coffee at the time you specify."

[0186] Specific examples

[0187] While getting ready in the morning, a user might say to the smart speaker, "I want to have coffee brewed automatically this morning." This triggers the system. The smart speaker uses voice recognition software to convert this speech into text data and send it to the server.

[0188] The server generates a prompt based on the received text data to be input to the generative AI model. An example of a prompt is as follows:

[0189] A user says he wants to automatically brew coffee in the morning. What electronic device would be a good combination?

[0190] A generative AI model analyzes this prompt and generates the following suggestions:

[0191] By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time.

[0192] The server sends the suggestion to the smart speaker, which uses speech synthesis software to convert the suggestion into speech and convey it to the user.

[0193] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0194] Step 1:

[0195] The user speaks their request to the smart speaker.

[0196] Specific behavior:

[0197] The user speaks to the smart speaker, "I want to have coffee brewed automatically in the morning."

[0198] Input: User's voice

[0199] Output: Audio data

[0200] Step 2:

[0201] The device (smart speaker) converts the voice into text data.

[0202] Specific behavior:

[0203] The device uses built-in voice recognition software to convert the user's speech into text data.

[0204] Input: Audio data

[0205] Output: Text data "I want to make coffee automatically in the morning."

[0206] Step 3:

[0207] The server receives the text data and generates a prompt sentence for the generative AI model.

[0208] Specific behavior:

[0209] The server receives the text data sent from the terminal and generates a prompt sentence to be input to the generative artificial intelligence model.

[0210] Input: Text data "I want to make coffee automatically in the morning"

[0211] Output: Prompt: "The user wants to automatically brew coffee in the morning. Which electronic device should they pair with?"

[0212] Step 4:

[0213] The server inputs prompts into the generative AI model, which then suggests the optimal combination of electronic devices.

[0214] Specific behavior:

[0215] The server inputs the generated prompt sentence into a generative artificial intelligence model, which then suggests the optimal combination of electronic devices.

[0216] Input: Prompt: "The user wants to automatically brew coffee in the morning. Which electronic device would be best to pair it with?"

[0217] Output: Suggestion: "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time."

[0218] Step 5:

[0219] The server receives the generated proposal and sends it to the terminal.

[0220] Specific behavior:

[0221] The server transmits the proposals received from the generative artificial intelligence model to the terminal.

[0222] Input: "Pair your smart coffee maker with a smart plug to automatically brew coffee at a specified time."

[0223] Output: Text data of the proposal

[0224] Step 6:

[0225] The device will communicate the suggestions to the user by voice.

[0226] Specific behavior:

[0227] The device converts the received suggestions into speech using speech synthesis software and conveys them to the user.

[0228] Input: Proposal text data

[0229] Output: "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time."

[0230] (Application example 2)

[0231] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0232] Conventional systems using voice recognition devices and generative AI have limited functionality for suggesting optimal product and electronic device combinations based on user requests, and lack specific suggestions for improving the shopping experience, particularly in brick-and-mortar stores. Furthermore, they lack the functionality to optimize electronic device operation based on the user's lifestyle and preferences. This makes it difficult for users to shop efficiently and satisfactorily in brick-and-mortar stores.

[0233] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition device linked with a generative AI, means for the voice recognition device to suggest, purchase, and assist with setting up products that can be linked with the user in a conversational format, means for the voice recognition device to automatically connect with other electronic devices, and means for the voice recognition device to suggest optimal products and combinations based on the user's requests in order to improve the shopping experience in physical stores. This enables users to shop efficiently and satisfactorily in physical stores.

[0234] "Generative AI" is a type of artificial intelligence that generates optimal suggestions and answers based on user requests and input.

[0235] A "voice recognition device" is a device that recognizes a user's voice and processes instructions or information based on that voice.

[0236] "Conversational" means that the user and the system exchange information through natural dialogue.

[0237] "Product suggestion" is the act of recommending the most suitable product based on the user's requests and needs.

[0238] "Purchase" refers to the user's action of taking steps to acquire the suggested product.

[0239] "Setup assistance" refers to the act of assisting users in setting up purchased products or electronic devices.

[0240] "Electronic devices" is a general term for machines and devices that operate using electricity.

[0241] A "brick and mortar store" is a store that exists in a physical location and where customers can visit in person to purchase products.

[0242] "Shopping experience" refers to the overall experience and feelings a user has while selecting and purchasing a product.

[0243] "Lifestyle habits" refers to the user's behavioral patterns and habits in their daily lives.

[0244] "Preferences" refer to the personal tastes and preferences of a user.

[0245] "Optimizing operation" means adjusting the operation of an electronic device to the optimum state according to the user's needs and circumstances.

[0246] The system for carrying out the present invention is mainly composed of a speech recognition device linked to a generative AI. A specific embodiment of this system will be described below.

[0247] Hardware and Software Configuration

[0248] Hardware:

[0249] Voice recognition devices (e.g. smartphones, smart speakers)

[0250] Server (a server for hosting generative AI models)

[0251] Electronic devices (e.g. smart appliances, smart plugs)

[0252] software:

[0253] OpenAI API (API for using generative AI models)

[0254] Python (registered trademark) (programming language used to implement programs)

[0255] Data processing and calculation

[0256] server:

[0257] The server hosts a generative AI model that generates optimal product and electronic device combinations based on user requests. User requests are sent to the server in text format via a voice recognition device. The server uses the OpenAI API to send prompts to the generative AI model, which then generates optimal recommendations.

[0258] Terminal (voice recognition device):

[0259] The speech recognizer recognizes the user's speech and converts it into text, which is then sent to a server where it is processed by a generative AI model, which then sends the generated suggestions back to the speech recognizer and presents them to the user in audio or text form.

[0260] User:

[0261] A user speaks their request to a speech recognizer. For example, if they say, "I want new kitchen appliances," the speech recognizer converts this into text and sends it to the server. The server uses a generative AI model to generate optimal suggestions and presents them to the user through the speech recognizer.

[0262] Specific examples

[0263] If a user says, "I want new kitchen appliances," the speech recognizer converts this request into text and sends it to the server, which then sends the following prompt to the generative AI model:

[0264] Example prompt sentence:

[0265] If a user says, "I want new kitchen appliances," suggest the optimal combination of smart appliances.

[0266] Based on this prompt, the generative AI model generates suggestions such as, "Combining a smart oven and a smart refrigerator will improve cooking efficiency." These suggestions are sent from the server to a speech recognition device and presented to the user as voice or text.

[0267] In this way, users can enjoy an efficient and satisfying shopping experience in physical stores.

[0268] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0269] Step 1:

[0270] A user speaks a request to a speech recognition device. For example, the user speaks, "I want new kitchen appliances." The input is the user's voice, and the output is speech data from the speech recognition device.

[0271] Step 2:

[0272] A speech recognizer converts a user's speech into text format. The input is speech data and the output is text data. This conversion is performed using speech recognition technology.

[0273] Step 3:

[0274] The speech recognition device sends the converted text data to the server. The input is text data, and the output is data sent to the server.

[0275] Step 4:

[0276] The server analyzes the received text data and generates a prompt to send to the generative AI model. The input is text data, and the output is a prompt. An example of a specific prompt would be, "If the user says, 'I want new kitchen appliances,' please suggest the optimal combination of smart appliances."

[0277] Step 5:

[0278] The server generates a prompt and sends it to the generative AI model using the OpenAI API. The input is the prompt, and the output is the proposed data from the generative AI model.

[0279] Step 6:

[0280] The generative AI model generates optimal suggestions based on the prompt. The input is the prompt, and the output is the suggested text. For example, a suggestion might be, "Combining a smart oven and a smart refrigerator will improve your cooking efficiency."

[0281] Step 7:

[0282] The server receives the generated suggested text and sends it to the speech recognizer, where the input is the suggested text and the output is the data sent to the speech recognizer.

[0283] Step 8:

[0284] The speech recognition device receives the suggested text and presents it to the user as voice or text. The input is the suggested text, and the output is what is presented to the user. In concrete terms, the speech recognition device tells the user by voice, "Combining a smart oven and a smart refrigerator will improve your cooking efficiency."

[0285] In this way, users can enjoy efficient and satisfying shopping at physical stores.

[0286] Example 3

[0287] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0288] Conventional smart home appliance systems have the problem of not providing optimal operation based on the user's lifestyle and preferences, which means that user convenience is not sufficiently improved. Additionally, users must manually configure the settings, which makes operation cumbersome.

[0289] The specification process by the specification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes a voice recognition device linked with a generative AI, means for the voice recognition device to suggest, purchase, and assist in setting up products that can be linked with the user in an interactive format, means for the voice recognition device to automatically connect with other electronic devices, means for collecting user lifestyle habit data, means for analyzing the collected data and learning the user's lifestyle habits and preferences, and means for optimizing the operation of the electronic device based on the learned data. This enables optimal operation based on the user's lifestyle habits and preferences to be automatically performed, significantly improving user convenience.

[0290] "Generative AI" is an artificial intelligence system that learns based on user data and makes optimal suggestions and operations.

[0291] A "voice recognizer" is a device that recognizes a user's voice and understands and processes instructions and commands.

[0292] "Dialogue style" refers to the way in which users and systems communicate with each other through natural conversation.

[0293] "Product suggestion" is the process of recommending appropriate products based on a user's needs and preferences.

[0294] "Purchase" refers to the user taking steps to acquire a suggested product.

[0295] "Setting assistance" is a function that assists users when setting up electronic devices or systems.

[0296] "Electronic devices" refers to devices that can be connected to the Internet and operated, such as smart speakers and smart lights.

[0297] "Lifestyle data" refers to information about a user's daily behavior and preferences.

[0298] "Analyzing data" refers to processing collected data and extracting meaningful information and patterns.

[0299] "Operation optimization" refers to adjusting the operation of electronic devices to the optimal state based on the user's lifestyle and preferences.

[0300] MODE FOR CARRYING OUT THE INVENTION

[0301] This invention is a system that uses a voice recognition device linked to generative AI to optimize the operation of electronic devices based on the user's lifestyle and preferences. Specific embodiments of this system are described below.

[0302] 1. System Configuration

[0303] The server is responsible for building a generative AI model and collecting and analyzing the user's lifestyle data. The server trains the generative AI model using a machine learning library (e.g., TENSORFLOW (registered trademark), PyTorch (registered trademark)).

[0304] The devices include voice recognition devices (e.g., smart speakers) and other electronic devices (e.g., smart lights). These devices record the user's operation history and send it to the server.

[0305] Users operate the voice recognition device on a daily basis to provide lifestyle data. For example, data is collected when a user says "good night" to a smart speaker at 10 p.m.

[0306] 2. Program Processing

[0307] The server collects the user's operation history sent from the voice recognition device. The collected data is analyzed within the server to learn the user's lifestyle and preferences. For example, it may detect a pattern that the user goes to bed at 10 p.m. every night.

[0308] The server trains a generative AI model based on the analysis results, which then inputs the user's lifestyle data and suggests optimal ways to operate electronic devices.

[0309] The server then uses the learned model to optimize the operation of electronic devices, for example by instructing a voice recognition device to automatically turn off smart lights at 10 p.m.

[0310] The terminal operates according to instructions from the server, allowing the user to enjoy an optimized environment without any special operations.

[0311] 3. Specific Examples

[0312] For example, if a user has a habit of going to bed at 10pm every night, the following actions will occur:

[0313] The user says "good night" to their smart speaker at 10pm.

[0314] The device (smart speaker) records the time and command and sends it to the server.

[0315] The server analyzes data from the past month and detects that the user goes to bed at 10pm every night.

[0316] The server trains a generative AI model to learn this pattern.

[0317] The server instructs the voice recognition device to automatically turn off the smart light at 10pm.

[0318] The device (smart speaker) receives instructions from the server and automatically turns off the smart light.

[0319] 4. Examples of prompts

[0320] An example of a prompt to input to a generative AI model is as follows:

[0321] Enter the user's lifestyle data. For example, if the user goes to bed at 10pm every day, enter "User goes to bed at 10pm every day."

[0322] Using this prompt, the generative AI model can input the user's lifestyle data and suggest optimal electronic device operations.

[0323] As described above, the present invention is a system that significantly improves user convenience by optimizing the operation of an electronic device based on the user's lifestyle and preferences. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[0324] Step 1: Data collection

[0325] The server cooperates with a voice recognition device to collect lifestyle data of the user.

[0326] The terminal (voice recognition device) records the user's operation history and transmits it to the server.

[0327] A user operates a voice recognition device on a daily basis, for example, by saying "good night" at 10 p.m.

[0328] Input: The user's voice command and its duration.

[0329] Output: Operation history data sent to the server.

[0330] Specific operation: When a user says "good night" to the voice recognition device at 10 p.m., the voice recognition device records the time and command and sends it to the server.

[0331] Step 2: Data analysis

[0332] The server analyzes the collected data and extracts the user's lifestyle habits and preferences.

[0333] The server uses machine learning algorithms to detect patterns in the data.

[0334] Input: Collected operation history data.

[0335] Output: Analysis results about the user's lifestyle and preferences.

[0336] Specific operation: The server analyzes data from the past month and detects that the user says "good night" every day at 10 p.m.

[0337] Step 3: Model training

[0338] The server trains a generative AI model based on the analysis results.

[0339] The server builds the model using machine learning libraries such as TensorFlow and PyTorch.

[0340] Input: Analysis results of the user's lifestyle and preferences.

[0341] Output: A trained generative AI model.

[0342] How it works: The server uses the user's lifestyle data as input to train a generative AI model. For example, the model learns the pattern that "the user goes to bed at 10 p.m. every night."

[0343] Step 4: Operation optimization

[0344] The server optimizes the operation of the electronic device based on the learned model.

[0345] The server automates the settings of electronic devices to suit the user's lifestyle.

[0346] Input: A trained generative AI model.

[0347] Output: Optimized electronic device operating instructions.

[0348] Specific operation: The server instructs the voice recognition device to automatically turn off the smart light at 10 p.m.

[0349] Step 5: Execution

[0350] The terminal (voice recognition device or smart light) operates according to instructions from the server.

[0351] The user can enjoy the optimized environment without any special operation.

[0352] Input: Operation instructions from the server.

[0353] Output: The electronic action that is performed.

[0354] Specific operation: At 10pm, the voice recognition device receives instructions from the server and automatically turns off the smart light.

[0355] (Application example 3)

[0356] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] While conventional smart home appliance systems can optimize the operation of appliances based on a user's lifestyle and preferences, they lack the functionality to optimize the shopping experience in physical stores. As a result, users are unable to receive sale information or recommended products at the right time when shopping in physical stores, making it difficult to shop efficiently. Furthermore, in-store navigation and product suggestions are lacking, which hinders the user's shopping experience.

[0358] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0359] In this invention, the server includes a smart speaker linked to a generative AI, means for the smart speaker to suggest, purchase, and set up products that can be linked through a conversation with the user, means for the smart speaker to automatically connect to other smart home appliances, means for the generative AI to learn the user's shopping habits and preferences and optimize the shopping experience at a physical store based on the learning, means for the generative AI to notify the user of sale information and recommended products according to the user's shopping time, and means for the generative AI to navigate within the store and suggest products based on the user's preferences, thereby enabling the user to have an efficient and comfortable shopping experience at a physical store.

[0360] "Generative AI" is an artificial intelligence that learns the user's lifestyle habits and preferences and makes optimal suggestions and operations based on that.

[0361] A "smart speaker" is a device that has voice recognition capabilities and provides information and controls home appliances through conversation with the user.

[0362] "A means of suggesting, purchasing, and assisting with setting up products that can be linked to the user in a conversational format" refers to a function in which a smart speaker suggests products to the user through voice dialogue, assists with the purchase process, and assists with setting up.

[0363] "Means for automatically connecting with other smart home appliances" refers to a function that enables the smart speaker to automatically communicate with other smart home appliances and operate them in coordination.

[0364] "A means of learning users' shopping habits and preferences and optimizing the in-store shopping experience based on that" refers to a function in which generative AI analyzes users' shopping patterns and preferences and, based on that, makes shopping in physical stores more efficient and comfortable.

[0365] "A means of notifying users of sale information and recommended products according to their shopping time" is a function in which generative AI predicts users' shopping times and notifies them of sale information and recommended products on their smartphones at those times.

[0366] "Means for in-store navigation and product suggestions based on user preferences" refers to a function in which generative AI assists users in moving around in physical stores and suggests products that suit the user's preferences.

[0367] To implement this invention, we use a smart speaker, a smartphone, and a navigation system for a physical store that are linked to generative AI. The specific system configuration and its operation are described below.

[0368] System Configuration

[0369] 1. Smart Speaker:

[0370] It is a device with voice recognition capabilities that provides information and controls home appliances through conversation with the user.

[0371] It works in conjunction with generative AI to learn the user's lifestyle habits and preferences.

[0372] 2. Generative AI:

[0373] It is an artificial intelligence that learns the user's shopping habits and preferences and makes optimal suggestions and operations based on them.

[0374] To optimize the in-store shopping experience, we notify users of special offers and recommended products at the time of their shopping.

[0375] 3. Smartphone:

[0376] Users receive notifications from generative AI on devices they carry with them.

[0377] Display in-store navigation and product suggestions.

[0378] 4. In-store navigation system:

[0379] This system assists users in moving around the store and makes product suggestions based on the user's preferences.

[0380] Operation explanation

[0381] 1. User habits:

[0382] The smart speaker works with generative AI to learn a user's lifestyle and shopping habits, for example, learning that the user shops at a specific supermarket every Saturday at 10 a.m.

[0383] 2. Special Offers and Recommended Product Notifications:

[0384] Generative AI notifies users of sale information and recommended products on their smartphones according to their shopping time, allowing them to shop efficiently.

[0385] 3. In-store navigation and product suggestions:

[0386] When a user arrives at a physical store, a smartphone application will begin navigating the store, and generative AI will suggest products based on the user's preferences and help them navigate the store.

[0387] Specific examples

[0388] For example, if a user has a habit of shopping at the supermarket every Saturday at 10 a.m., the generative AI will notify the user of sales information and recommended products on their smartphone at that time. When the user arrives at the supermarket, the smartphone application will begin navigating the store, and the generative AI will suggest products based on the user's preferences.

[0389] Prompt Sentence Examples

[0390] "Build an application that learns your grocery shopping habits every Saturday at 10 AM and notifies you of special sales and product recommendations at that time."

[0391] The above is a specific embodiment for carrying out the present invention.

[0392] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0393] Step 1:

[0394] User habits learning

[0395] The server collects the user's lifestyle and shopping habits through the smart speaker. Specifically, it collects data that the user shops at a specific supermarket every Saturday at 10:00 AM. The input is the user's behavioral data, and the output is the user's habit patterns. The server inputs this data into a generative AI model to learn the user's habits.

[0396] Step 2:

[0397] Special offers and recommended product notifications

[0398] The server uses a generative AI model to generate sale information and recommended products tailored to the user's shopping time. The input is the user's habit patterns and store sale information, and the output is the notification content. The server sends this notification content to the user's smartphone and notifies them. Specifically, the server sends sale information and recommended products via push notification to coincide with the user's shopping time.

[0399] Step 3:

[0400] Start in-store navigation

[0401] When a user arrives at a physical store, the smartphone application starts in-store navigation. The input is the user's location information, and the output is the navigation route. The server receives the user's location information, calculates the optimal route within the store, and sends it to the smartphone. Specifically, it displays the route to the location of the product the user is looking for.

[0402] Step 4:

[0403] Product suggestion

[0404] The server uses a generative AI model to make product suggestions based on the user's preferences. The input is the user's preference data and store inventory information, and the output is a list of suggested products. The server sends this list of suggested products to the smartphone and displays it to the user. Specifically, it lists products that the user is likely to be interested in and displays them on the application.

[0405] Step 5:

[0406] Collecting user feedback

[0407] After the user finishes shopping, the smartphone application collects feedback from the user. The input is the user's feedback, and the output is improvements and new habit data. The server inputs this feedback into the generative AI model to improve the accuracy of next suggestions and navigation. Specifically, it evaluates whether the user purchased the suggested products and whether the navigation was helpful.

[0408] The above are the specific processing steps for carrying out the present invention.

[0409] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0410] "Example 1"

[0411] One embodiment of the present invention is a system that combines a smart speaker linked to a generative AI with an emotion engine. In this system, the emotion engine recognizes the user's emotions through conversation with the user. For example, if the user says, "I'm tired today," the emotion engine recognizes that the user is tired. It then conveys this information to the generative AI. Based on this information, the generative AI suggests a combination of smart home appliances that best suits the user's emotions, such as a smart speaker that plays relaxing music or a smart light that emits a warm glow.

[0412] "Example 2"

[0413] Another embodiment of the present invention is a system in which an emotion engine optimizes the operation of smart home appliances based on a user's emotions. In this system, the emotion engine recognizes the user's emotions and instructs a generative AI to operate smart home appliances in a way that is appropriate for that emotion. For example, if the generative AI recognizes that the user is feeling happy, it instructs a smart speaker to play enjoyable music that the user likes.

[0414] "Example 3"

[0415] Another embodiment of the present invention is a system in which an emotion engine proposes an optimal combination of smart home appliances based on a user's emotions. In this system, the emotion engine recognizes the user's emotions and instructs a generative AI to select a combination of smart home appliances appropriate for that emotion. For example, if the emotion engine recognizes that the user is sad, the generative AI instructs a smart light to emit a calming light and simultaneously instructs a smart speaker to play calming music that the user prefers.

[0416] The processing flow of each embodiment will be described below.

[0417] "Example 1"

[0418] Step 1: The emotion engine recognizes the user's emotions through conversation with the user.

[0419] Step 2: The emotion engine conveys the recognized emotion information to the generative AI.

[0420] Step 3: Based on this information, the generative AI suggests a combination of smart home appliances that best suits the user's emotions.

[0421] "Example 2"

[0422] Step 1: The emotion engine recognizes the user's emotion.

[0423] Step 2: Based on the recognized emotions, the emotion engine issues instructions to the generative AI to optimize the operation of smart home appliances.

[0424] Step 3: The generative AI follows the instructions and operates the smart appliances.

[0425] "Example 3"

[0426] Step 1: The emotion engine recognizes the user's emotion.

[0427] Step 2: Based on the recognized emotions, the emotion engine instructs the generative AI on the optimal combination of smart home appliances.

[0428] Step 3: The generative AI follows the instructions and suggests combinations of smart appliances.

[0429] Example 1

[0430] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0431] Conventional smart speakers can perform simple operations and make suggestions based on the user's requests, but they are unable to make optimal suggestions or settings based on the user's emotions and lifestyle. Furthermore, they lack the ability to link with multiple electronic devices, preventing them from fully improving user convenience. This makes it difficult to provide the highly personalized experience that users desire.

[0432] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0433] In this invention, the server includes a voice recognition device linked to a generative artificial intelligence, means for the voice recognition device to suggest, purchase, and assist in setting up products that can be linked with the user in a conversational format, means for the voice recognition device to automatically connect to other electronic devices, an emotion analysis device for the voice recognition device to analyze the user's emotions, means for transmitting the emotion information analyzed by the emotion analysis device to the generative artificial intelligence, and means for the generative artificial intelligence to suggest an optimal combination of electronic devices based on the emotion information, thereby enabling highly personalized suggestions and settings based on the user's emotions and lifestyle habits.

[0434] "Generative AI" is an AI technology that analyzes user input data and generates appropriate responses and suggestions.

[0435] A "voice recognition device" is a device that converts a user's voice into text data.

[0436] "Electronic devices" include devices that can be connected to the Internet and operated, such as smart heaters and smart lights.

[0437] An "emotion analysis device" is a device that analyzes emotions from a user's comments and actions and provides that information to other systems.

[0438] "Suggestion" is the act of recommending optimal products and settings based on the user's requests and feelings.

[0439] "Settings assistance" is a support function that automatically configures electronic devices based on suggestions selected by the user.

[0440] "Conversational" refers to the way in which a user and a system exchange information through dialogue.

[0441] "Automatic connection" refers to the ability of the system to establish communication with other electronic devices without user intervention.

[0442] "Emotion information" is data that indicates the emotional state of the user analyzed by the emotion analysis device.

[0443] "Personalized" refers to providing services and suggestions that are optimized to suit each user's individual preferences and circumstances.

[0444] This invention is a system that uses a voice recognition device linked to generative artificial intelligence to suggest products, assist with purchasing, and assist with setup through a conversation with the user. It also has the function of analyzing the user's emotions and proposing the optimal combination of electronic devices based on that information.

[0445] Hardware and software used

[0446] 1. Speech Recognition Device: Uses speech recognition technology to convert the user's voice into text data. Specifically, it uses the Google Cloud Speech-to-Text API.

[0447] 2. Generative AI: OpenAI's GPT-4 is used to analyze user requests and emotional information and generate appropriate suggestions.

[0448] 3. Sentiment analyzer: Uses IBM Watson® Tone Analyzer to analyze emotions from user utterances.

[0449] 4. Speech synthesis technology: Amazon Polly is used to communicate the generated suggestions to the user aloud.

[0450] 5. Electronics: This includes household appliances that can be connected to the internet and controlled, such as smart heaters and smart lights.

[0451] Specific operation of the system

[0452] Collaboration between smart speakers and generative AI

[0453] 1. The user says, "I want to heat the room."

[0454] A user speaks to a voice recognition device, saying, "I want to heat up the room."

[0455] 2. The device recognizes the voice and converts it into text data.

[0456] The device uses the Google Cloud Speech-to-Text API to convert speech into text data.

[0457] 3. The device sends the text data to the server.

[0458] The terminal transmits the converted text data to a server via the Internet.

[0459] 4. The server generates suggestions using generative artificial intelligence.

[0460] The server inputs the received text data into OpenAI's GPT-4 and generates a suggestion such as, "A smart heater is suitable. Would you like help setting it up?"

[0461] 5. The server sends the generated proposal to the device.

[0462] The server sends the generated proposal to the terminal.

[0463] 6. The device will then audibly communicate the suggestions to the user.

[0464] The device uses Amazon Polly to translate suggestions into speech and communicate them to the user.

[0465] 7. When the user requests assistance, the device connects to the smart heater and configures it.

[0466] If the user answers "yes," the device will use the Wi-Fi module to connect to the smart heater's API and configure it.

[0467] Collaboration between smart speakers, generative AI, and emotion engines

[0468] 1. A user says, "I'm tired today."

[0469] A user speaks to a speech recognition device, saying, "I'm tired today."

[0470] 2. The device recognizes the voice and converts it into text data.

[0471] The device uses the Google Cloud Speech-to-Text API to convert speech into text data.

[0472] 3. The device sends the text data to the server.

[0473] The terminal transmits the converted text data to a server via the Internet.

[0474] 4. The server analyzes the user's emotions using an emotion analyzer.

[0475] The server uses IBM Watson Tone Analyzer to analyze the user's emotions and recognizes them as "tired."

[0476] 5. The server sends the emotional information to the generative AI.

[0477] The server sends the emotion information to OpenAI's GPT-4.

[0478] 6. The server generates suggestions using generative artificial intelligence.

[0479] Based on the emotional information, the server generates suggestions such as a smart speaker that plays relaxing music or a smart light that emits a warm glow.

[0480] 7. The server sends the generated proposal to the device.

[0481] The server sends the generated proposal to the terminal.

[0482] 8. The device will then audibly communicate the suggestions to the user.

[0483] The device uses Amazon Polly to translate suggestions into speech and communicate them to the user.

[0484] Prompt Sentence Examples

[0485] 1. A prompt from the smart speaker to a user who says, "I want to heat up the room":

[0486] The user says, "I want to heat my room." Generate a response that suggests suitable smart appliances and assists with the setup.

[0487] 2. Prompt sentence suggested in cooperation with the emotion analyzer to a user who says "I'm tired today":

[0488] A user says, "I'm tired today." The emotion analyzer recognizes the user's emotion as "tired." Generate smart home appliance recommendations that match the user's emotion.

[0489] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0490] Collaboration between smart speakers and generative AI

[0491] Step 1:

[0492] The user says, "I want to heat the room."

[0493] Input: User voice input

[0494] Output: Audio data

[0495] Specific operation: The user speaks to the smart speaker, saying, "I want to heat up the room."

[0496] Step 2:

[0497] The device recognizes the voice and converts it into text data.

[0498] Input: Audio data

[0499] Output: Text data

[0500] Specific operation: The device uses the Google Cloud Speech-to-Text API to convert speech into text data.

[0501] Step 3:

[0502] The terminal transmits the text data to the server.

[0503] Input: Text data

[0504] Output: Send data to the server

[0505] Specific operation: The device sends the converted text data to a server via the Internet.

[0506] Step 4:

[0507] The server generates suggestions using generative artificial intelligence.

[0508] Input: Text data

[0509] Output: Suggested text

[0510] Specific operation: The server inputs the received text data into OpenAI's GPT-4 and generates a suggestion such as, "A smart heater is suitable. Would you like help setting it up?"

[0511] Step 5:

[0512] The server sends the generated proposal to the terminal.

[0513] Input: Suggested text

[0514] Output: Sending data to the terminal

[0515] Specific operation: The server sends the generated proposal to the terminal.

[0516] Step 6:

[0517] The device will communicate the suggestions to the user aloud.

[0518] Input: Suggested text

[0519] Output: Audio data

[0520] What it does: The device uses Amazon Polly to translate suggestions into speech and communicate them to the user.

[0521] Step 7:

[0522] When the user requests assistance, the device connects to the smart heater and configures it.

[0523] Input: User's voice input (e.g. "Yes")

[0524] Output: Smart heater setup complete

[0525] Specific operation: If the user answers "yes," the device will use the Wi-Fi module to connect to the smart heater's API and perform the configuration.

[0526] Collaboration between smart speakers, generative AI, and emotion engines

[0527] Step 1:

[0528] A user says, "I'm tired today."

[0529] Input: User voice input

[0530] Output: Audio data

[0531] Specific action: The user speaks to the smart speaker, "I'm tired today."

[0532] Step 2:

[0533] The device recognizes the voice and converts it into text data.

[0534] Input: Audio data

[0535] Output: Text data

[0536] Specific operation: The device uses the Google Cloud Speech-to-Text API to convert speech into text data.

[0537] Step 3:

[0538] The terminal transmits the text data to the server.

[0539] Input: Text data

[0540] Output: Send data to the server

[0541] Specific operation: The device sends the converted text data to a server via the Internet.

[0542] Step 4:

[0543] The server uses an emotion analysis device to analyze the user's emotions.

[0544] Input: Text data

[0545] Output: Emotional information

[0546] Specific operation: The server uses IBM Watson Tone Analyzer to analyze the user's emotions and recognizes them as "tired."

[0547] Step 5:

[0548] The server sends the emotional information to the generative artificial intelligence.

[0549] Input: Emotion information

[0550] Output: Sending data to generative AI

[0551] Specific operation: The server sends the emotion information to OpenAI's GPT-4.

[0552] Step 6:

[0553] The server generates suggestions using generative artificial intelligence.

[0554] Input: Emotion information

[0555] Output: Suggested text

[0556] Specific operation: Based on the emotional information, the server generates suggestions such as a smart speaker that plays relaxing music or a smart light that emits warm light.

[0557] Step 7:

[0558] The server sends the generated proposal to the terminal.

[0559] Input: Suggested text

[0560] Output: Sending data to the terminal

[0561] Specific operation: The server sends the generated proposal to the terminal.

[0562] Step 8:

[0563] The device will communicate the suggestions to the user aloud.

[0564] Input: Suggested text

[0565] Output: Audio data

[0566] What it does: The device uses Amazon Polly to translate suggestions into speech and communicate them to the user.

[0567] (Application example 1)

[0568] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0569] While conventional smart speakers and smart home appliances can provide suggestions and setup assistance based on user needs, they face the challenge of being unable to provide optimal suggestions based on the user's emotions and circumstances. Furthermore, there has been a lack of concrete measures to improve the shopping experience in physical stores.

[0570] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a smart speaker linked with a generative AI, means for the smart speaker to suggest, purchase, and assist in setting up products that can be linked with the user in a conversational format, means for the smart speaker to automatically connect with other smart home appliances, an emotion engine that recognizes the user's emotions, and means for the generative AI to suggest optimal products and services based on the emotions recognized by the emotion engine. This makes it possible to make optimal suggestions based on the user's emotions and situation.

[0571] "Generative AI" is artificial intelligence that generates appropriate responses and suggestions based on user input and the situation.

[0572] A "smart speaker" is a device that has voice recognition capabilities and can perform various operations and provide information through conversations with the user.

[0573] "Means for suggesting, purchasing, and assisting with setting up products" refers to a function that suggests appropriate products based on the user's request, assists with the purchasing process, and also supports post-purchase setup.

[0574] "Means for connecting with other smart home appliances" is a function that allows the smart speaker to communicate with other smart home appliances and operate them in cooperation with each other.

[0575] An "emotion engine" is a technology that analyzes emotions from a user's statements and actions and recognizes those emotions.

[0576] "Means of suggesting optimal products and services" refers to a function in which generative AI suggests the most suitable products and services to users based on the user's emotions recognized by the emotion engine.

[0577] To implement this invention, a smart speaker, emotion engine, and smartphone linked with generative AI are used. The specific system configuration and operation are described below.

[0578] System configuration

[0579] 1. Smart speaker: A device with voice recognition capabilities that can be used to operate or provide information through conversation with the user.

[0580] 2. Generative AI: This is artificial intelligence that generates appropriate responses and suggestions based on user input and context. It uses the OpenAI API.

[0581] 3. Emotion Engine: This technology analyzes emotions from user comments and actions and recognizes those emotions. It uses the Emotion Engine API.

[0582] 4. Smartphone: This is the device that users use in physical stores and works in conjunction with the smart speaker.

[0583] How it works

[0584] 1. Getting User Input

[0585] When a user speaks to their smartphone, the voice input is sent to the smart speaker, which converts the speech into text and sends it to the emotion engine.

[0586] 2. Emotional Recognition

[0587] The emotion engine analyzes the user's emotions from the received text and sends the results to the generative AI. For example, if a user says, "I'm tired today," the emotion engine will recognize this as "tired."

[0588] 3. Proposal Generation

[0589] Based on the emotional information received from the emotion engine, generative AI can suggest optimal products and services to users, such as "relaxation drinks and massage chairs."

[0590] 4. Presenting the proposal

[0591] The suggestions generated by the generative AI are presented to the user via a voice message through the smart speaker. If the user accepts the suggestions, the smart speaker will automatically connect to other smart home appliances and configure them.

[0592] Specific examples

[0593] When a user says "I'm tired today" to their smartphone in a physical store, the application uses an emotion engine to recognize "fatigue" and sends a prompt to the generative AI model saying, "What products or services would you suggest when the user is tired?" The generative AI model then suggests "relaxation drinks and massage chairs."

[0594] Prompt Sentence Examples

[0595] "What products or services would you suggest to users when they're feeling tired?"

[0596] "What smart home appliances would you suggest when a user says, 'I want to heat up my room.'"

[0597] In this way, a system is realized that can make optimal suggestions according to the user's emotions and situation.

[0598] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0599] Step 1:

[0600] A user speaks into a smartphone. The user's voice input is captured by the smartphone's microphone and obtained as voice data. The input is the user's voice, and the output is voice data.

[0601] Step 2:

[0602] The smartphone converts the voice data into text data. Speech recognition software (e.g., Google Speech-to-Text API) is used to convert the voice data into text data. The input is voice data, and the output is text data.

[0603] Step 3:

[0604] The smartphone sends text data to an emotion engine. The emotion engine (e.g., Emotion Engine API) analyzes the text data and recognizes the user's emotion. The input is text data, and the output is emotion data (e.g., "tired").

[0605] Step 4:

[0606] The emotion engine sends the recognized emotion data to the generative AI. The generative AI (e.g., OpenAI API) generates prompts based on the emotion data and suggests appropriate products or services. The input is emotion data, and the output is suggestion data (e.g., "relaxation drinks and massage chairs").

[0607] Step 5:

[0608] The generative AI sends the generated proposal data to a smartphone. The smartphone receives the proposal data and presents it to the user via voice. The input is the proposal data, and the output is voice data.

[0609] Step 6:

[0610] If the user accepts the suggestion, the smartphone sends instructions to the smart speaker, which then connects to and configures other smart appliances. The input is the user's approval data, and the output is the configuration data for the smart appliances.

[0611] In this way, a system is realized that can make optimal suggestions according to the user's emotions and situation.

[0612] Example 2

[0613] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0614] Conventional smart home appliance systems have difficulty proposing optimal combinations and operations of home appliances based on user needs and emotions. Furthermore, the lack of conversational interaction with users and automatic connection functions limited the user experience. This made it difficult for users to effectively use smart home appliances.

[0615] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0616] In this invention, the server includes a voice recognition device linked to a generative artificial intelligence, means for the voice recognition device to suggest, purchase, and assist in setting up products that can be linked in a conversational format with the user, means for the voice recognition device to automatically connect to other electronic devices, means for acquiring user requests, means for analyzing the requests, means for sending prompts to the generative artificial intelligence based on the analysis results, means for the generative artificial intelligence to suggest an optimal combination of electronic devices, means for transmitting the suggestions to the voice recognition device, means for the voice recognition device to notify the user of the suggestions, means for recognizing the user's emotions, means for instructing the generative artificial intelligence to operate the electronic devices based on the emotions, means for the generative artificial intelligence to suggest an operation of the electronic devices based on the emotions, means for transmitting the suggestions to the voice recognition device, and means for the voice recognition device to notify the user of the suggestions. This makes it possible to suggest optimal combinations and operations of home appliances based on the user's requests and emotions.

[0617] "Generative AI" is an AI technology that generates optimal suggestions and operations based on the user's requests and emotions.

[0618] A "voice recognition device" is a device that recognizes the user's voice and gives instructions or suggestions based on that voice.

[0619] "A means for suggesting, purchasing, and assisting with setting up products through a conversational format with the user" is a function that suggests, purchases, and assists with setting up products through a voice conversation with the user.

[0620] The "means for automatically connecting with other electronic devices" is a function for the voice recognition device to automatically connect with other electronic devices.

[0621] The "means for acquiring a user's request" is a function for acquiring a request uttered by a user to a voice recognition device.

[0622] "Means for analyzing requests" refers to a function for analyzing acquired user requests and understanding their contents.

[0623] "Means for sending prompt sentences to the generative AI" is a function for sending instructions to the generative AI based on the analysis results.

[0624] The "means for proposing the optimal combination of electronic devices" is a function in which generative artificial intelligence proposes the optimal combination of electronic devices based on the user's requests.

[0625] The "means for transmitting the proposal content to the speech recognition device" is a function for transmitting the proposal content from the generative artificial intelligence to the speech recognition device.

[0626] The "means for the voice recognition device to notify the user of the content of the suggestion" is a function by which the voice recognition device notifies the user of the content of the suggestion.

[0627] "Means for recognizing the user's emotions" refers to a function that recognizes emotions by analyzing the user's facial expressions, tone of voice, etc.

[0628] "Means for instructing a generative artificial intelligence to operate an electronic device based on emotions" is a function that instructs a generative artificial intelligence to operate an electronic device based on recognized emotions.

[0629] "Means for suggesting operations for electronic devices based on emotions" is a function in which generative artificial intelligence suggests operations for electronic devices based on the user's emotions.

[0630] This invention is a system that uses a voice recognition device linked to generative artificial intelligence to suggest optimal combinations and operations of electronic devices based on the user's requests and emotions. Specific embodiments of this system are described below.

[0631] Hardware and Software Configuration

[0632] Hardware

[0633] Speech recognition device: A device that recognizes a user's voice and acquires their requests and emotions. Specifically, it includes a microphone, speaker, and processor.

[0634] Server: A device that analyzes data sent from the speech recognition device and sends prompts to the generative AI. It is equipped with a high-performance processor and large-capacity memory.

[0635] Electronics: Smart appliances and other electronic devices, such as smart coffee makers, smart plugs, and smart speakers.

[0636] software

[0637] Generative AI model: An AI model that generates optimal suggestions and operations based on the user's requests and emotions. For example, a model that uses natural language processing (NLP) technology.

[0638] Emotion recognition engine: Software that recognizes emotions by analyzing a user's facial expressions and tone of voice.

[0639] Data analysis software: Software for analyzing user requests and generating prompts for generative artificial intelligence.

[0640] Data processing and calculation

[0641] Acquiring and analyzing user requests

[0642] When a user says to the speech recognition device, "I want to make coffee automatically in the morning," the speech recognition device sends the speech data to the server. The server uses data analysis software to analyze the speech and sends the following prompt to the generative artificial intelligence:

[0643] A user says, "I want to automatically brew coffee in the morning." Please suggest the best combination of smart home appliances.

[0644] Proposals by generative artificial intelligence

[0645] The generative AI model analyzes the prompt and suggests the optimal combination of electronic devices. For example, it might suggest, "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time." The server then sends this suggestion to a voice recognizer, which then notifies the user.

[0646] Emotion recognition and operation suggestions

[0647] The emotion recognition engine analyzes the user's facial expressions and tone of voice and determines that the user is feeling happy. The emotion recognition engine then sends the following prompt to the generative AI:

[0648] The user is feeling happy. Tell your smart speaker to play some fun music they like.

[0649] The generative AI model analyzes this prompt and suggests to the smart speaker, "Instruct it to play enjoyable music that the user likes." The server sends this suggestion to the voice recognizer, which then notifies the user and actually plays the music.

[0650] Specific examples

[0651] If a user says, "I want to make coffee automatically in the morning," the speech recognizer sends the statement to the server. The server analyzes the statement and sends a prompt to the generative AI. The generative AI suggests, "By combining a smart coffee maker with a smart plug, you can make coffee automatically at a specified time." The server sends the suggestion to the speech recognizer, which notifies the user.

[0652] If the user is recognized as feeling joy, the emotion recognition engine sends a prompt to the generative AI. The generative AI suggests, "I'll instruct you to play some fun music that the user likes," and the server sends the suggestion to the voice recognition device. The voice recognition device then notifies the user, "I'll play some fun music," and actually plays the music.

[0653] In this way, a system is realized that suggests optimal combinations and operations of electronic devices based on the user's requests and feelings.

[0654] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0655] Step 1:

[0656] The user speaks a request to the voice recognition device.

[0657] Input: User's voice data

[0658] Output: Audio data sent to the speech recognizer

[0659] What happens: A user says, "I want to have coffee brewed automatically in the morning." The voice recognition device captures this speech with a microphone.

[0660] Step 2:

[0661] The voice recognition device sends the user's voice data to the server.

[0662] Input: Audio data captured by a speech recognizer

[0663] Output: Audio data sent to the server

[0664] Specific operation: The voice recognition device converts the captured voice data into a digital signal and sends it to the server.

[0665] Step 3:

[0666] The server analyzes the voice data and converts the user's request into text format.

[0667] Input: Audio data sent to the server

[0668] Output: User requests in text format

[0669] Specific operation: The server uses voice recognition software to analyze the voice data and generate text data such as "I want to make coffee automatically this morning."

[0670] Step 4:

[0671] The server sends a prompt to the generative artificial intelligence.

[0672] Input: User request in text format

[0673] Output: Prompt sent to the generative AI

[0674] Specific operation: The server generates the following prompt sentence based on the text data and sends it to the generative artificial intelligence.

[0675] A user says, "I want to automatically brew coffee in the morning." Please suggest the best combination of smart home appliances.

[0676] Step 5:

[0677] Generative artificial intelligence analyzes the prompt text and suggests the optimal combination of electronic devices.

[0678] Input: Prompt sent to the generative AI

[0679] Output: Proposal

[0680] Specific operation: The generative AI analyzes the prompt and generates a suggestion such as, "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time."

[0681] Step 6:

[0682] The server sends the suggestions to the speech recognition device.

[0683] Input: Proposal from generative artificial intelligence

[0684] Output: Suggestions sent to the speech recognizer

[0685] Specific operation: The server sends the suggestions received from the generative AI to the speech recognition device.

[0686] Step 7:

[0687] A voice recognition device notifies the user of the suggestions.

[0688] Input: Proposal sent from the server

[0689] Output: The suggestion that will be sent to the user

[0690] Specific operation: The voice recognition device notifies the user, "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at the specified time."

[0691] Step 8:

[0692] The emotion recognition engine recognizes the user's emotions.

[0693] Input: User facial expressions and tone of voice

[0694] Output: Perceived user sentiment

[0695] Specific behavior: The emotion recognition engine analyzes the user's facial expressions and tone of voice and recognizes that the user is feeling joy.

[0696] Step 9:

[0697] The emotion recognition engine sends prompts to the generative AI.

[0698] Input: Perceived user emotion

[0699] Output: Prompt sent to the generative AI

[0700] Specific operation: The emotion recognition engine generates the following prompt sentence and sends it to the generative artificial intelligence.

[0701] The user is feeling happy. Tell your smart speaker to play some fun music they like.

[0702] Step 10:

[0703] Generative AI analyzes prompts and suggests emotion-based electronic device controls.

[0704] Input: Prompt sent to the generative AI

[0705] Output: Proposal

[0706] Specific operation: The generative AI analyzes the prompt and generates a suggestion such as, "Instruct the system to play enjoyable music that the user likes."

[0707] Step 11:

[0708] The server sends the suggestions to the speech recognition device.

[0709] Input: Proposal from generative artificial intelligence

[0710] Output: Suggestions sent to the speech recognizer

[0711] Specific operation: The server sends the suggestions received from the generative AI to the speech recognition device.

[0712] Step 12:

[0713] A voice recognition device notifies the user of the suggestions.

[0714] Input: Proposal sent from the server

[0715] Output: The suggestion that will be sent to the user

[0716] Specific operation: The voice recognition device notifies the user, "I'm going to play some fun music," and then actually plays the music.

[0717] (Application example 2)

[0718] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0719] Conventional smart home appliance systems have few functions that suggest the optimal combination of home appliances based on the user's requests or that optimize the operation of home appliances according to the user's emotions. Furthermore, there have been no systems in physical stores that suggest the optimal combination of products and services based on the user's requests, or adjust the in-store environment according to the user's emotions. This has led to the issue of not being able to sufficiently improve user convenience and satisfaction.

[0720] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0721] In this invention, the server includes a smart speaker linked to a generative AI, a means for the smart speaker to suggest, purchase, and assist with the setup of products that can be linked through a conversation with the user, a means for the smart speaker to automatically connect to other smart home appliances, a means for the smart speaker to suggest an optimal combination of products and services based on the user's requests in a physical store, and a means for the smart speaker to recognize the user's emotions and adjust the in-store environment accordingly. This makes it possible to suggest optimal products and services based on the user's requests, and further, by realizing environmental adjustments according to the user's emotions, it is possible to significantly improve user convenience and satisfaction.

[0722] "Generative AI" refers to artificial intelligence that generates optimal suggestions and answers based on user requests and input.

[0723] A "smart speaker" is a speaker that has voice recognition capabilities, can connect to the Internet to provide various information, and can operate other smart home appliances.

[0724] "Conversational" refers to a form of interaction in which the user asks questions or gives instructions by voice, and the system responds by voice.

[0725] "Product proposal" refers to the act of selecting the most suitable product based on the user's requests and conditions and presenting it to the user.

[0726] "Purchase" refers to the act of a user actually buying a suggested product.

[0727] "Setup assistance" refers to the act of providing assistance to users in properly setting up the products or services they have purchased.

[0728] "Means for automatically connecting with other smart home appliances" refers to a function that enables a smart speaker to automatically communicate with other smart home appliances and operate in cooperation with them.

[0729] A "brick and mortar store" is a store that exists in a physical location and where users can visit in person to purchase goods or services.

[0730] "User requirements" refer to the specific needs and desires that users have for the system.

[0731] "Optimal combination of products and services" means selecting multiple products and services that best meet the user's needs and combining them to propose them.

[0732] "User emotion" refers to the psychological state that a user feels, such as happiness, sadness, or surprise.

[0733] "In-store environment" refers to physical elements such as music, lighting, and temperature within a physical store.

[0734] "Means to adjust the environment" refers to the function of changing the music, lighting, etc. in the store according to the user's emotions.

[0735] The system for implementing this invention is centered around a smart speaker linked to generative AI. The smart speaker suggests, purchases, and assists with the setup of compatible products through a conversation with the user. The smart speaker also automatically connects with other smart home appliances and suggests optimal combinations of products and services based on the user's requests in a physical store. Furthermore, the smart speaker recognizes the user's emotions and adjusts the in-store environment accordingly.

[0736] Hardware and software used

[0737] Hardware: Smartphones, smart speakers, in-store smart speakers, smart lighting systems

[0738] Software: OpenAI API, EmotionRecognizer library, HTTP request library (requests)

[0739] System Operation

[0740] The server uses generative AI to propose optimal combinations of products and services based on the user's needs. When a user inputs their needs into a smartphone app, the generative AI generates optimal suggestions based on those needs. For example, if a user inputs "I'm looking for a new smartphone," the generative AI will suggest "a combination of the latest smartphone and recommended accessories."

[0741] The server also uses the EmotionRecognizer library to recognize the user's emotions. Once the server recognizes the user's emotions, it sends instructions to the store's smart speakers and smart lighting system via HTTP requests. For example, if the server recognizes that the user is happy, it will play cheerful music and brighten the lights in the store.

[0742] Specific examples

[0743] When a user uses their smartphone in a physical store and types "I'm looking for a new smartphone," the generative AI generates the best suggestions using the following prompt:

[0744] Prompt Sentence Examples

[0745] Recommend the best combination of products and services based on the user's needs: I'm looking for a new smartphone

[0746] Based on this prompt, the generative AI will suggest a combination of the latest smartphone and recommended accessories. Furthermore, if the EmotionRecognizer library recognizes the user's emotion as "joy," the server will instruct the store's smart speakers to play cheerful music and the smart lighting system to brighten the lights.

[0747] In this way, it is possible to propose optimal products and services based on the user's needs and adjust the environment to suit the user's emotions, which will significantly improve user convenience and satisfaction.

[0748] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0749] Step 1:

[0750] The user enters their request into the smartphone app.

[0751] Input: User's request (e.g. "I'm looking for a new smartphone")

[0752] Output: User request data

[0753] Specific operation: The user opens the smartphone app, enters a request in the text box, and presses the send button.

[0754] Step 2:

[0755] The terminal transmits the user's request data to the server.

[0756] Input: User's desired data

[0757] Output: Request data sent to the server

[0758] Specific operation: The smartphone app sends the user's requested data to the server via an HTTP request.

[0759] Step 3:

[0760] The server uses the generative AI model to generate the optimal combination of products and services.

[0761] Input: User's desired data

[0762] Output: Optimal product and service combinations

[0763] Specific operation: The server calls the OpenAI API, sends the prompt sentence to the generative AI model, and receives the generated suggestions.

[0764] Example prompt: "Suggest the best combination of products and services based on the user's request: I'm looking for a new smartphone."

[0765] Step 4:

[0766] The server sends the generated proposal to the terminal.

[0767] Input: Optimal product and service combinations

[0768] Output: Proposal data sent to the device

[0769] Specific operation: The server sends the generated proposal to the smartphone app as an HTTP response.

[0770] Step 5:

[0771] The terminal displays the proposed data to the user.

[0772] Input: Proposal data

[0773] Output: The proposal displayed to the user

[0774] Specific operation: The smartphone app receives the proposed data and displays it on the screen.

[0775] Step 6:

[0776] The device uses the EmotionRecognizer library to recognize the user's emotions.

[0777] Input: User's voice and facial expression data

[0778] Output: User emotion data

[0779] Specific operation: The smartphone app captures the user's voice and facial expressions using the camera and microphone, and recognizes emotions using the EmotionRecognizer library.

[0780] Step 7:

[0781] The device transmits the emotion data to the server.

[0782] Input: User emotion data

[0783] Output: Emotion data sent to the server

[0784] Specific operation: The smartphone app sends emotion data to the server via an HTTP request.

[0785] Step 8:

[0786] The server generates instructions to adjust the environment in the store based on the emotion data.

[0787] Input: User emotion data

[0788] Output: Environmental adjustment instruction data

[0789] Specific operation: The server analyzes the emotional data, determines appropriate music and lighting settings, and generates instruction data.

[0790] Step 9:

[0791] The server sends environmental adjustment instruction data to the smart speakers and smart lighting systems in the store.

[0792] Input: Environmental adjustment instruction data

[0793] Output: Instruction data sent to smart speakers and smart lighting systems in the store

[0794] Specific operation: The server sends instruction data to the smart speakers and smart lighting systems in the store via HTTP requests.

[0795] Step 10:

[0796] Smart speakers and smart lighting systems in the store will adjust the environment according to instructions.

[0797] Input: Instruction data

[0798] Output: Calibrated in-store environment

[0799] What it does: A smart speaker plays fun music and a smart lighting system brightens the lights.

[0800] Example 3

[0801] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0802] Conventional smart home appliance systems have difficulty optimizing operation based on the user's lifestyle habits and emotions, and have not been able to fully improve user convenience. Furthermore, they lacked the functionality to suggest optimal combinations of home appliances based on the user's needs, which required the user to configure them themselves. Furthermore, since they were unable to suggest home appliance operation based on emotions, it was difficult to provide a comfortable environment that responded to the user's emotions.

[0803] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0804] In this invention, the server includes a voice recognition device linked to a generative AI, means for the voice recognition device to suggest, purchase, and assist with the setup of products that can be linked to the user in a conversational format, means for the voice recognition device to automatically connect to other electronic devices, means for collecting data on the user's lifestyle habits and preferences, means for the generative AI to learn the collected data and determine optimal electronic device operations, and means for the emotion engine to recognize the user's emotions and suggest optimal electronic device operations based on those emotions. This enables optimal home appliance operation based on the user's lifestyle habits and emotions, significantly improving user convenience. It also allows the server to suggest optimal combinations of home appliances based on the user's requests, eliminating the need for the user to manually configure the appliances. Furthermore, it enables suggesting home appliance operations based on emotions, providing a comfortable environment that responds to the user's emotions.

[0805] "Generative AI" is an artificial intelligence system that learns from user data and performs optimal operations and suggestions based on that data.

[0806] A "voice recognition device" is a device that recognizes a user's voice and processes the voice data.

[0807] "Electronic devices" refers to any electrically operated equipment, including smart appliances and other digital devices.

[0808] An "emotion engine" is a system that recognizes the user's emotions and performs appropriate operations or suggestions based on those emotions.

[0809] "Data collection means" refers to a method or device for collecting data about a user's lifestyle and preferences.

[0810] The "operation determination means" refers to a method or device for determining the optimum operation of an electronic device based on collected data.

[0811] The "operation suggestion means" refers to a method or device for suggesting the optimum operation of an electronic device based on the user's feelings and requests.

[0812] "Conversational" refers to a format in which a user and a speech recognition device exchange information through natural dialogue.

[0813] "Means for suggesting, purchasing, and assisting in setting up products" refers to a method or device for suggesting appropriate products to users, assisting in the purchasing process, and further assisting in setting up the products.

[0814] "Automatic connection means" refers to a method or device for automatically connecting a voice recognition device to another electronic device.

[0815] This invention is a system that uses a voice recognition device linked to generative AI to realize optimal operation of electronic devices based on the user's lifestyle and emotions. Specific embodiments of this system are described below.

[0816] Hardware and software used

[0817] 1. Hardware

[0818] Speech recognition device: A device that recognizes a user's voice and processes the voice data. A specific example is a smart speaker (e.g., a typical voice assistant device).

[0819] Electronics: Smart appliances and other digital devices. Examples include smart lights (e.g., common smart lighting devices) and smart speakers.

[0820] 2. Software

[0821] Generative AI model: An artificial intelligence system that learns from user data and performs optimal operations and suggestions based on that data. A specific example is a general generative AI model.

[0822] Emotion engine: A system that recognizes the user's emotions and performs appropriate operations or suggestions based on those emotions. A specific example is a general emotion recognition API.

[0823] System Operation

[0824] 1. Data Collection

[0825] Records the user's daily actions and preferences. For example, if the user says "good night" at 10 p.m. every day, that time is recorded.

[0826] The terminal (voice recognition device) collects user behavior data and sends it to the server.

[0827] 2. Data training

[0828] The server sends the collected data to the generative AI model.

[0829] The generative AI model learns the user's lifestyle habits and preferences, for example, learning the user's pattern of saying "good night" every day at 10 p.m.

[0830] 3. Operation optimization

[0831] The generative AI model determines the optimal operation of electronic devices based on the information it has learned.

[0832] The server sends operational instructions to the electronic devices. For example, at 10 p.m., the server instructs the voice recognition device to "turn off the smart light."

[0833] 4. Emotion recognition

[0834] The terminal (voice recognition device) captures the user's voice and facial expressions and sends them to the server.

[0835] The emotion engine analyzes the captured data and recognizes the user's emotion. For example, if the user speaks in a sad voice, the emotion engine will recognize the emotion as "sad."

[0836] 5. Emotion-based suggestions

[0837] Based on the emotions recognized by the emotion engine, it instructs the generative AI on how to optimally operate electronic devices.

[0838] The generative AI model determines the appropriate operation for the electronic device based on the user's emotion, and the server sends instructions to the device. For example, if the user is recognized as sad, the server will instruct a smart light to emit a calming light and a smart speaker to play calming music.

[0839] Specific examples

[0840] 1. Learning user lifestyle habits

[0841] The user says "good night" every night at 10pm.

[0842] The terminal records the time and sends it to the server.

[0843] The generative AI model learns the pattern and instructs the server to automatically turn off the smart lights at 10pm.

[0844] 2. Emotion-based recommendations

[0845] The user says in a sad voice, "I'm tired today."

[0846] The device captures the audio and sends it to the server.

[0847] The emotion engine recognizes this as "sad" and sends that information to the generative AI model.

[0848] A generative AI model instructs a smart light to "emit a soothing color of light" and a smart speaker to "play soothing music."

[0849] Prompt Sentence Examples

[0850] Lifestyle learning prompts

[0851] We've learned that the user goes to bed at 10pm every night. Tell the smart speaker to turn off the smart light at 10pm.

[0852] Emotion-based suggestion prompts

[0853] The user is recognized as sad. Instruct the smart light to emit a calming light color and the smart speaker to play the user's preferred calming music.

[0854] The flow of the identification process in the third embodiment will be described with reference to FIG.

[0855] Step 1:

[0856] Data collection

[0857] Recording the user's daily actions and preferences, for example, when the user says "good night" at 10pm every night.

[0858] The terminal (voice recognition device) collects the user's voice data and sends the data to the server.

[0859] Input: User's voice data

[0860] Data processing: Converts audio data into text data and adds a timestamp.

[0861] Output: Text data and timestamp

[0862] Step 2:

[0863] Data training

[0864] The server sends the collected data to the generative AI model.

[0865] The generative AI model learns the user's habits and preferences, for example, learning that the user says "good night" every night at 10 p.m.

[0866] Input: Text data and timestamp

[0867] Data calculation: Using pattern recognition algorithms, user behavior patterns are extracted.

[0868] Output: User behavior patterns

[0869] Step 3:

[0870] Operation optimization

[0871] The generative AI model determines the optimal operation of electronic devices based on the information it has learned.

[0872] The server sends operational instructions to the electronic devices. For example, at 10 p.m., the server instructs the voice recognition device to "turn off the smart light."

[0873] Input: User behavior patterns

[0874] Data calculation: Generate optimal operation sequences based on behavioral patterns.

[0875] Output: Operation instructions

[0876] Step 4:

[0877] emotion recognition

[0878] The terminal (voice recognition device) captures the user's voice and facial expressions and sends them to the server.

[0879] The emotion engine analyzes the captured data and recognizes the user's emotion. For example, if the user speaks in a sad voice, the emotion engine will recognize the emotion as "sad."

[0880] Input: User's voice data and facial expression data

[0881] Data processing: Analyze voice data and facial expression data to identify emotions.

[0882] Output: User's emotional state

[0883] Step 5:

[0884] Emotion-based suggestions

[0885] Based on the emotions recognized by the emotion engine, it instructs the generative AI on how to optimally operate electronic devices.

[0886] The generative AI model determines the appropriate operation for the electronic device based on the user's emotion, and the server sends instructions to the device. For example, if the user is recognized as sad, the server will instruct a smart light to emit a calming light and a smart speaker to play calming music.

[0887] Input: User's emotional state

[0888] Data Computation: Generate optimal operation sequences based on emotional states.

[0889] Output: Operation instructions

[0890] (Application example 3)

[0891] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0892] Conventional smart home appliance systems can optimize operation based on the user's lifestyle and preferences, but they were unable to optimize the environment or suggest products based on the customer's emotions and lifestyle in physical stores. In addition, there was a lack of means to provide an in-store environment based on the customer's emotions, making it difficult to improve the customer experience.

[0893] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0894] In this invention, the server includes a smart speaker linked to a generative AI, means for the smart speaker to suggest, purchase, and assist in setting up products that can be linked through a conversation with the user, means for the smart speaker to automatically connect to other smart home appliances, means for the generative AI to recognize the user's emotions and optimize the operation of the smart home appliances based on those emotions, means for the generative AI to optimize the in-store environment based on the customer's lifestyle and preferences in a physical store, means for the generative AI to provide an in-store environment that corresponds to the customer's emotions, and means for the generative AI to suggest products and services based on the customer's preferences. This makes it possible to provide an optimal environment and suggest products that correspond to the customer's emotions and lifestyle even in a physical store.

[0895] "Generative AI" is an artificial intelligence system that learns the user's lifestyle habits and preferences and then performs optimal operations and makes suggestions based on that.

[0896] A "smart speaker" is a speaker with a voice assistant function that uses voice recognition technology to interact with users and operate other smart home appliances in conjunction with them.

[0897] "A means of suggesting, purchasing, and assisting with setting up products through conversation with the user" is a function in which a smart speaker can converse with the user via voice to suggest products, assist with purchasing procedures, and assist with setting up.

[0898] "Means for automatically connecting with other smart home appliances" refers to a function that enables the smart speaker to automatically communicate with other smart home appliances and operate them in coordination.

[0899] "Means of recognizing user emotions and optimizing the operation of smart home appliances based on those emotions" refers to a function in which generative AI analyzes user emotions and adjusts the settings and operation of smart home appliances based on the results.

[0900] "A means of optimizing the in-store environment based on customers' lifestyle habits and preferences in physical stores" refers to a function in which generative AI learns customers' past behavioral data and preferences, and adjusts the in-store environment, such as lighting and music, based on that.

[0901] "Means of providing an in-store environment that responds to customer emotions" refers to a function in which generative AI recognizes customer emotions in real time and provides an environment, such as lighting and music, that is appropriate for those emotions.

[0902] "Means of suggesting products and services based on customer preferences" refers to a function in which generative AI analyzes a customer's past purchase history and preferences, and then suggests appropriate products and services based on that.

[0903] As an embodiment of the present invention, a system for improving customer experience in a physical store is constructed. A specific embodiment of the system is described below.

[0904] System configuration

[0905] The system includes smart speakers, smart glasses, smartphones, and other smart home appliances that are connected to generative AI. These devices communicate with a server and provide an optimal environment based on the customer's lifestyle and emotions.

[0906] Hardware and software used

[0907] Smart speaker: Uses voice recognition technology to communicate with users and connect with other smart home appliances.

[0908] Smart glasses: Equipped with a camera and display, they analyze customers' facial expressions and recognize their emotions.

[0909] Smartphone: Collects customer data through an application and sends it to a server.

[0910] Server: Runs generative AI models and processes data.

[0911] Generative AI model: Learns the user's lifestyle habits and emotions and provides optimal operations and suggestions.

[0912] Data processing and calculation

[0913] The server receives customer facial expression data sent from smart glasses or smartphones, analyzes the customer's emotions using an emotion recognition model, and then learns the customer's past behavioral data and preferences, optimizing the in-store environment (lighting, music, etc.) based on that data.

[0914] Specific examples

[0915] For example, when a customer enters a store, the smart glasses analyze the customer's facial expression and determine that they are relaxed. The server then uses a generative AI model to instruct the store to soften the lighting and play calming music. The display also shows the customer's most frequently purchased products.

[0916] Prompt Sentence Examples

[0917] "Analyze users' facial expressions and recognize their emotions. Based on the emotions detected, optimize lighting and music in the store and display products that match the user's preferences."

[0918] In this way, even in physical stores, it will be possible to provide the optimal environment and product suggestions that suit customers' emotions and lifestyles.

[0919] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0920] Step 1:

[0921] Acquiring customer facial expression data

[0922] The smart glasses or smartphone camera is used to capture customer facial expression data.

[0923] Input: Customer's face image

[0924] Output: Facial expression data (image data)

[0925] How it works: The camera in the smart glasses or smartphone captures the customer's face and sends the image data to a server.

[0926] Step 2:

[0927] emotion recognition

[0928] The server inputs the received facial expression data into an emotion recognition model to analyze the customer's emotions.

[0929] Input: facial expression data (image data)

[0930] Output: Emotion data (e.g., relaxed, stressed, etc.)

[0931] Specific Operation: The server uses an emotion recognition model (e.g., a machine learning model) to infer the customer's emotion from the facial expression data.

[0932] Step 3:

[0933] Referencing lifestyle data

[0934] The server references the customer's past behavioral data and preferences from a database.

[0935] Input: Customer ID

[0936] Output: Lifestyle data (e.g., visit time, purchase history, etc.)

[0937] Specific operation: The server queries the database using the customer ID as a key to obtain the customer's lifestyle habit data.

[0938] Step 4:

[0939] Environment optimization instructions

[0940] The server generates instructions to optimize the in-store environment (lighting, music, etc.) based on emotional data and lifestyle data.

[0941] Input: Emotion data, lifestyle data

[0942] Output: Environmental settings instructions (e.g. lighting color, type of music, etc.)

[0943] How it works: The server uses a generative AI model to calculate optimal environmental settings based on emotional and lifestyle data, and sends these instructions to smart appliances.

[0944] Step 5:

[0945] Run environment setup

[0946] Smart home appliances set up the environment within the store based on instructions from the server.

[0947] Input: Configuration instructions

[0948] Output: Optimized store environment (e.g., changing lighting, playing music, etc.)

[0949] Specific actions: Smart lights change color and smart speakers play specified music.

[0950] Step 6:

[0951] View product suggestions

[0952] The server suggests products and services based on the customer's preferences and displays them on the display.

[0953] Input: lifestyle data, customer preferences

[0954] Output: Product suggestions (e.g. product list displayed on the screen)

[0955] Specific operation: The server analyzes the customer's past purchase history and preferences, selects appropriate products, and sends instructions to display them on the screen.

[0956] In this way, even in physical stores, it will be possible to provide the optimal environment and product suggestions that suit customers' emotions and lifestyles.

[0957] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0958] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0959] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0960] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0961] [Second embodiment]

[0962] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0963] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0964] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0965] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0966] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0967] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0968] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0969] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0970] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0971] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0972] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0973] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0974] "Example 1"

[0975] One embodiment of the present invention provides a smart speaker that works in conjunction with generative AI. This smart speaker suggests, purchases, and assists with the setup of smart home appliances through a conversation with the user. Specifically, when the user says, "I want to heat my room," the smart speaker works in conjunction with the generative AI to suggest, "A smart heater would be suitable. Would you like help setting it up?" When the user requests assistance, the smart speaker automatically connects to the smart heater and sets it up.

[0976] "Example 2"

[0977] Furthermore, generative AI can suggest optimal combinations of smart home appliances based on user requests. For example, if a user says, "I want to make coffee automatically in the morning," the smart speaker will suggest, "By combining a smart coffee maker with a smart plug, you can make coffee automatically at the specified time."

[0978] "Example 3"

[0979] Generative AI can also learn about a user's lifestyle and preferences and optimize the operation of smart home appliances accordingly. For example, if a user learns that they go to bed at 10 p.m. every night, the smart speaker will automatically turn off the smart lights at 10 p.m.

[0980] The processing flow of each embodiment will be described below.

[0981] "Example 1"

[0982] Step 1: The user says to the smart speaker, "I want to heat up my room."

[0983] Step 2: The smart speaker works with the generative AI to determine the best smart appliance for the user's needs. In this case, it determines that a smart heater is suitable.

[0984] Step 3: The smart speaker will suggest to the user, "A smart heater is suitable. Would you like help setting it up?"

[0985] Step 4: When the user requests assistance, the smart speaker will automatically connect to the smart heater and set it up.

[0986] "Example 2"

[0987] Step 1: The user says to the smart speaker, "I want to automatically brew coffee in the morning."

[0988] Step 2: The smart speaker works with the generative AI to determine the optimal combination of smart appliances for the user's needs. In this case, it determines that a smart coffee maker and a smart plug are the best combination.

[0989] Step 3: The smart speaker suggests to the user, "By combining your smart coffee maker with the smart plug, you can automatically brew coffee at the time you specify."

[0990] "Example 3"

[0991] Step 1: The generative AI learns the user's habits and preferences. In this case, it learns that the user goes to bed at 10pm every night.

[0992] Step 2: The smart speaker will use the learned information to optimize the operation of smart home appliances, in this case automatically turning off the smart lights at 10 o'clock.

[0993] Example 1

[0994] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0995] Conventional smart speakers only respond to users' voice inputs and lack the functionality to suggest appropriate smart home appliances based on the user's specific needs and assist with their setup. They also lack the functionality to optimize the operation of smart home appliances based on the user's lifestyle and preferences. This requires users to manually configure the appliances, reducing convenience.

[0996] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0997] In this invention, the server includes means for receiving a user's voice input, means for converting the voice into text, means for transmitting the text data to the server, means for the server to generate a response using a generative AI model, means for returning the response to the terminal, means for the terminal to convert the response into voice and suggest it to the user, means for receiving the user's response, and means for connecting to and configuring smart home appliances. This makes it possible to suggest appropriate smart home appliances based on the user's specific needs and automatically configure them. It also makes it possible to optimize the operation of smart home appliances based on the user's lifestyle and preferences.

[0998] "Means for receiving user voice input" refers to devices or software that capture voices uttered by the user and input them into the system.

[0999] "Speech-to-text means" refers to speech recognition technology for analyzing and converting captured speech data into corresponding text data.

[1000] The "means for transmitting text data to a server" refers to a communication protocol or interface for transmitting the converted text data to a server via a network.

[1001] "Means by which the server generates a response using a generative AI model" refers to the process by which the server generates an appropriate response using a generative AI model based on the text data received by the server.

[1002] The "means for returning a response to the terminal" refers to a communication protocol or interface for returning the generated response to the terminal via the network.

[1003] The "means for the terminal to convert the response into voice and make a suggestion to the user" refers to a device or software that converts the text response received from the server into voice using speech synthesis technology and makes a suggestion to the user.

[1004] The "means for receiving user responses" refers to a device or software for capturing responses made by users to suggestions and inputting them into the system.

[1005] "Means for connecting to and configuring smart home appliances" refers to devices or software that connect to the proposed smart home appliances via a communication protocol and automatically perform the necessary settings.

[1006] This invention is a system that uses a smart speaker linked to a generative AI model to suggest, purchase, and assist with the setup of smart home appliances based on user voice input. Specific embodiments of this system are described below.

[1007] Hardware and software used

[1008] Hardware

[1009] Smart speaker: A device that accepts a user's voice input and plays audio.

[1010] Server: A computer system for hosting generative AI models and performing data processing.

[1011] Smart appliances: Home appliances that can be controlled over a network, such as a smart heater.

[1012] software

[1013] Speech Recognition Software: Technology for converting speech to text, such as the Google Cloud Speech-to-Text API.

[1014] Generative AI models: Artificial intelligence models, such as OpenAI's GPT-3, that generate appropriate responses based on user requests.

[1015] Text-to-speech software: Technology for converting text to speech, such as Amazon Polly.

[1016] Communication Protocol: The protocol used to send and receive data, such as HTTPS.

[1017] System Operation

[1018] User voice input

[1019] The user says to the smart speaker, "I want to heat up the room," and the smart speaker captures the voice through its built-in microphone.

[1020] Speech-to-text

[1021] The smart speaker uses voice recognition software to convert the captured voice data into text data, which includes the phrase "I want to heat my room."

[1022] Sending text data

[1023] The smart speaker then sends the converted text data to the server using the HTTPS protocol to ensure data security.

[1024] Generating a response

[1025] The server inputs the received text data into the generative AI model, which generates an appropriate response based on the user's request. In this case, it generates the response, "A smart heater is suitable. Would you like help setting it up?"

[1026] Returning a response

[1027] The server then sends the generated response back to the smart speaker, again using the HTTPS protocol.

[1028] Playing audio

[1029] The smart speaker converts the text response received from the server into speech using speech synthesis software, and plays the converted speech from the speaker, suggesting to the user, "A smart heater is suitable. Would you like help setting it up?"

[1030] User Response

[1031] If the user responds "yes," the smart speaker again captures the audio and converts it to text using speech recognition software. This text data is then analyzed to confirm that the user is requesting assistance.

[1032] Smart appliance settings

[1033] The smart speaker automatically connects to the smart heater using communication protocols such as Wi-Fi or Bluetooth. Once the connection is established, the smart speaker configures the smart heater, such as setting the temperature and timer.

[1034] Examples and prompts

[1035] Specific examples

[1036] When a user says, "I want to warm up the room," the smart speaker will suggest, "A smart heater would be suitable. Would you like help setting it up?" If the user answers "yes," the smart speaker will automatically connect to the smart heater and complete the setup. For example, you can set the heater temperature to 22 degrees and set a timer to automatically turn it off after one hour.

[1037] Prompt Sentence Examples

[1038] "When a user says they want to heat their room, explain the system that suggests appropriate smart appliances and assists them in setting them up."

[1039] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1040] Step 1:

[1041] Accepting user voice input

[1042] A user says to a smart speaker, "I want to heat the room." The device (smart speaker) captures the voice through its built-in microphone. The input is the user's voice data, and the output is the captured voice data.

[1043] Step 2:

[1044] Convert speech to text

[1045] The device converts the captured voice data into text data using speech recognition software (e.g., Google Cloud Speech-to-Text API). The input is the captured voice data, and the output is the text data "I want to heat the room."

[1046] Step 3:

[1047] Send text data to the server

[1048] The terminal sends the converted text data to the server. The HTTPS protocol is used for transmission to ensure data security. The input is text data, and the output is the text data sent to the server.

[1049] Step 4:

[1050] The server generates a response using the generative AI model

[1051] The server inputs the received text data into a generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates an appropriate response based on the user's request. The input is text data, and the output is the response, "A smart heater is suitable. Would you like help setting it up?"

[1052] Step 5:

[1053] Sends the response back to the terminal

[1054] The server generates a response and sends it back to the device, again using the HTTPS protocol. The input is the generated response, and the output is the response sent back to the device.

[1055] Step 6:

[1056] The device converts the response into speech and suggests it to the user.

[1057] The device converts the text response received from the server into speech using speech synthesis software (e.g., Amazon Polly). The converted speech is played back through the speaker, suggesting to the user, "This smart heater is suitable. Would you like help setting it up?" The input is the text response, and the output is the speech response.

[1058] Step 7:

[1059] Receive the user's response

[1060] If the user responds "yes," the device again captures the voice and converts it to text using speech recognition software. This text data is then parsed to determine that the user is requesting assistance. The input is the user's voice response, and the output is text data.

[1061] Step 8:

[1062] Connect to smart home appliances and set them up

[1063] The terminal automatically connects to the smart heater using communication protocols such as Wi-Fi or Bluetooth. Once the connection is established, the terminal configures the smart heater, such as setting the temperature and timer. The input is the user's configuration request, and the output is the smart heater after the configuration is complete.

[1064] (Application example 1)

[1065] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1066] While existing smart speakers have the ability to suggest smart home appliances and assist with setup based on user needs, they lack functionality to improve the shopping experience in physical stores. As a result, users have difficulty finding the appropriate product section when searching for products in physical stores, and there is a lack of in-store navigation.

[1067] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1068] In this invention, the server includes a smart speaker linked with a generative AI, a means for the smart speaker to suggest, purchase, and set up products that can be linked with the user through a conversation, a means for the smart speaker to automatically connect with other smart home appliances, and a means for the smart speaker to suggest an appropriate product section based on the user's request in a physical store and provide in-store navigation, thereby improving the user's shopping experience in the physical store and enabling them to quickly find the appropriate product section.

[1069] "Generative AI" is an artificial intelligence technology that generates responses in natural language based on user input.

[1070] A "smart speaker" is a voice assistant device that has voice recognition capabilities and can perform various operations through conversations with the user.

[1071] "Conversational" refers to a form of interaction in which the user asks questions or gives instructions by voice, and the device responds by voice.

[1072] "Product suggestion" is the act of selecting appropriate products based on the user's requests and introducing them to the user.

[1073] "Purchase" refers to the user's action of taking steps to acquire the suggested product.

[1074] "Setup assistance" refers to the act of assisting users with initial setup and operation so that they can properly use the product they have purchased.

[1075] "Automatically connect to other smart home appliances" is a function that allows the smart speaker to communicate with and operate other smart home appliances via wireless communication based on user instructions.

[1076] "Suggesting the appropriate product section in a physical store based on the user's request" refers to the act of guiding a user to the section where a specific product is located when the user is searching for that product in a physical store.

[1077] "Providing in-store navigation" refers to a function that provides route guidance for users to reach the desired product section within a physical store.

[1078] As an embodiment of this invention, we will explain a system that uses a smart speaker linked to generative AI. This system provides product suggestions, purchasing assistance, and setup assistance through a conversation with the user, and also provides navigation within a physical store.

[1079] First, the smart speaker has a voice recognition function and converts the user's speech into text data. This text data is sent to a generative AI, which generates an appropriate response based on the user's request. The generative AI is implemented using, for example, OpenAI's API.

[1080] Generative AI suggests appropriate products based on user input. For example, if a user says, "I want warm clothes," the generative AI responds, "We have warm clothes in this section. Would you like me to show you?" This response is transmitted to the user through a smart speaker.

[1081] Next, if the user requests directions, the smart speaker will activate the in-store navigation function. This navigation function is linked to a database containing information about the sections in the store and will guide the user to the desired section. Navigation information is provided, for example, using Wi-Fi or Bluetooth beacons in the store.

[1082] As a concrete example, by inputting the following prompt sentence into a generative AI, a response that meets the user's request can be generated.

[1083] Example prompt sentence:

[1084] "If a user says, 'I want warm clothes,' generate an Assistant response that suggests appropriate products and provides navigation within the store."

[1085] The system will enhance the user shopping experience by integrating smart speakers, generative AI, and navigation functions. Users can quickly find the product section they are looking for in a physical store, and can also receive suggestions and setup assistance for smart home appliances, greatly improving user convenience.

[1086] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1087] Step 1:

[1088] A user says to a smart speaker, "I want warm clothes." The smart speaker uses its voice recognition function to convert this statement into text data. The input is the user's voice data, and the output is text data.

[1089] Step 2:

[1090] The smart speaker sends the converted text data to the generative AI, which takes this text data as input and creates a prompt to generate an appropriate response. The input is text data, and the output is a prompt.

[1091] Step 3:

[1092] The generative AI generates a response based on the prompt text according to the user's request. Specifically, the generative AI generates the response, "There are warm clothes in this section. Can you show me?" The input is the prompt text, and the output is the generated response text.

[1093] Step 4:

[1094] The smart speaker converts the generated response text into speech and transmits it to the user. The user listens to the smart speaker's response and decides whether to ask for guidance. The input is the generated response text, and the output is speech data.

[1095] Step 5:

[1096] When a user requests guidance, the smart speaker activates the navigation function, which connects to a database containing information about sections in the store and generates route information to guide the user to the desired section. The input is the user's instructions, and the output is route information.

[1097] Step 6:

[1098] The smart speaker then communicates the generated route information to the user by voice and guides the user to the desired section. The input is the route information and the output is voice data.

[1099] Step 7:

[1100] The user follows the guidance of the smart speaker to move around the store and reach the desired section. The input is voice data, and the output is the user's actions.

[1101] Example 2

[1102] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1103] Conventional smart home appliance systems required users to individually configure and operate each appliance, resulting in low convenience. Furthermore, they lacked the functionality to suggest optimal combinations of appliances based on the user's needs, requiring the user to spend time and effort finding the optimal combination. Furthermore, they lacked the functionality to optimize the operation of appliances based on the user's lifestyle and preferences. To address these issues, a system is needed that can suggest optimal combinations of appliances based on the user's needs and further optimize their operation.

[1104] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1105] In this invention, the server includes a voice recognition device linked to a generative artificial intelligence, means for the voice recognition device to suggest, purchase, and assist with setting up products that can be linked in a conversational format with the user, means for the voice recognition device to automatically connect to other electronic devices, means for converting the user's voice into text data, means for generating prompt sentences for inputting the text data to the generative artificial intelligence, means for the generative artificial intelligence to suggest an optimal combination of electronic devices, and means for communicating the suggestion to the user by voice. This makes it possible to automatically suggest an optimal combination of home appliances based on the user's requests and further optimize their operation.

[1106] "Generative AI" is an AI system that generates appropriate responses and suggestions based on user input.

[1107] A "voice recognition device" is a device that converts a user's voice into text data.

[1108] "User" refers to a person who uses this system.

[1109] "Conversational" is a format in which the user and the system interact, with the system responding to what the user says.

[1110] "Product suggestion" is the act of recommending appropriate products based on the user's requests.

[1111] "Purchase" is the act of a user acquiring a suggested product.

[1112] "Setting assistance" refers to the act of providing assistance to a user in properly setting up a proposed product.

[1113] "Electronic devices" refers to digital devices such as smart home appliances.

[1114] "Text data" is character information converted by a voice recognition device.

[1115] A "prompt sentence" is an instruction sentence to be input into generative artificial intelligence.

[1116] A "suggestion" is the optimal solution provided by generative artificial intelligence based on the user's request.

[1117] The "audio means" is a method for transmitting the generated suggestions to the user as audio.

[1118] This invention is a system that uses a voice recognition device linked to generative artificial intelligence to propose the optimal combination of electronic devices based on the user's request and optimize their operation. This system is implemented using the following hardware and software.

[1119] Hardware and software used

[1120] Hardware:

[1121] 1. Voice recognition devices (e.g. smart speakers)

[1122] 2. Server (a server for hosting generative AI)

[1123] software:

[1124] 1. Speech recognition software (e.g., software that converts speech into text data)

[1125] 2. Generative AI models (e.g., OpenAI's GPT-4)

[1126] Data processing and calculation

[1127] Voice recognition device:

[1128] When a user speaks a request to the voice recognition device, the voice recognition device uses built-in voice recognition software to convert the speech into text data. For example, if the user says, "I want to make coffee automatically in the morning," this speech is converted into text data that says, "I want to make coffee automatically in the morning."

[1129] server:

[1130] The server receives the text data sent from the speech recognition device. The server then generates a prompt to be input to the generative AI model. For example, this prompt might be in the form of, "The user wants to automatically make coffee in the morning. Which electronic device should I pair it with?"

[1131] The generative AI model analyzes this prompt and suggests optimal combinations of electronic devices, such as "Pairing a smart coffee maker with a smart plug will allow you to automatically brew coffee at a specified time."

[1132] The server sends the generated suggestions to a speech recognizer, which converts the suggestions into speech using speech synthesis software. Ultimately, the speech recognizer tells the user, "By combining your smart coffee maker with your smart plug, you can automatically brew coffee at the time you specify."

[1133] Specific examples

[1134] While getting ready in the morning, a user might say to the smart speaker, "I want to have coffee brewed automatically this morning." This triggers the system. The smart speaker uses voice recognition software to convert this speech into text data and send it to the server.

[1135] The server generates a prompt based on the received text data to be input to the generative AI model. An example of a prompt is as follows:

[1136] A user says he wants to automatically brew coffee in the morning. What electronic device would be a good combination?

[1137] A generative AI model analyzes this prompt and generates the following suggestions:

[1138] By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time.

[1139] The server sends the suggestion to the smart speaker, which uses speech synthesis software to convert the suggestion into speech and convey it to the user.

[1140] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1141] Step 1:

[1142] The user speaks their request to the smart speaker.

[1143] Specific behavior:

[1144] The user speaks to the smart speaker, "I want to have coffee brewed automatically in the morning."

[1145] Input: User's voice

[1146] Output: Audio data

[1147] Step 2:

[1148] The device (smart speaker) converts the voice into text data.

[1149] Specific behavior:

[1150] The device uses built-in voice recognition software to convert the user's speech into text data.

[1151] Input: Audio data

[1152] Output: Text data "I want to make coffee automatically in the morning."

[1153] Step 3:

[1154] The server receives the text data and generates a prompt sentence for the generative AI model.

[1155] Specific behavior:

[1156] The server receives the text data sent from the terminal and generates a prompt sentence to be input to the generative artificial intelligence model.

[1157] Input: Text data "I want to automatically brew coffee in the morning"

[1158] Output: Prompt: "The user wants to automatically brew coffee in the morning. Which electronic device should they pair with?"

[1159] Step 4:

[1160] The server inputs prompts into the generative AI model, which then suggests the optimal combination of electronic devices.

[1161] Specific behavior:

[1162] The server inputs the generated prompt sentence into a generative artificial intelligence model, which then suggests the optimal combination of electronic devices.

[1163] Input: Prompt: "The user wants to automatically brew coffee in the morning. Which electronic device would be best to pair it with?"

[1164] Output: Suggestion: "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time."

[1165] Step 5:

[1166] The server receives the generated proposal and sends it to the terminal.

[1167] Specific behavior:

[1168] The server transmits the proposals received from the generative artificial intelligence model to the terminal.

[1169] Input: "Pair your smart coffee maker with a smart plug to automatically brew coffee at a specified time."

[1170] Output: Text data of the proposal

[1171] Step 6:

[1172] The device will communicate the suggestions to the user by voice.

[1173] Specific behavior:

[1174] The device converts the received suggestions into speech using speech synthesis software and conveys them to the user.

[1175] Input: Proposal text data

[1176] Output: "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time."

[1177] (Application example 2)

[1178] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1179] Conventional systems using voice recognition devices and generative AI have limited functionality for suggesting optimal product and electronic device combinations based on user requests, and lack specific suggestions for improving the shopping experience, particularly in brick-and-mortar stores. Furthermore, they lack the functionality to optimize electronic device operation based on the user's lifestyle and preferences. This makes it difficult for users to shop efficiently and satisfactorily in brick-and-mortar stores.

[1180] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition device linked with a generative AI, means for the voice recognition device to suggest, purchase, and assist with setting up products that can be linked with the user in a conversational format, means for the voice recognition device to automatically connect with other electronic devices, and means for the voice recognition device to suggest optimal products and combinations based on the user's requests in order to improve the shopping experience in physical stores. This enables users to shop efficiently and satisfactorily in physical stores.

[1181] "Generative AI" is a type of artificial intelligence that generates optimal suggestions and answers based on user requests and input.

[1182] A "voice recognition device" is a device that recognizes a user's voice and processes instructions or information based on that voice.

[1183] "Conversational" means that the user and the system exchange information through natural dialogue.

[1184] "Product suggestion" is the act of recommending the most suitable product based on the user's requests and needs.

[1185] "Purchase" refers to the user's action of taking steps to acquire the suggested product.

[1186] "Setup assistance" refers to the act of assisting users in setting up purchased products or electronic devices.

[1187] "Electronic devices" is a general term for machines and devices that operate using electricity.

[1188] A "brick and mortar store" is a store that exists in a physical location and where customers can visit in person to purchase products.

[1189] "Shopping experience" refers to the overall experience and feelings a user has while selecting and purchasing a product.

[1190] "Lifestyle habits" refers to the user's behavioral patterns and habits in their daily lives.

[1191] "Preferences" refer to the personal tastes and preferences of a user.

[1192] "Optimizing operation" means adjusting the operation of an electronic device to the optimum state according to the user's needs and circumstances.

[1193] The system for carrying out the present invention is mainly composed of a speech recognition device linked to a generative AI. A specific embodiment of this system will be described below.

[1194] Hardware and Software Configuration

[1195] Hardware:

[1196] Voice recognition devices (e.g. smartphones, smart speakers)

[1197] Server (a server for hosting generative AI models)

[1198] Electronic devices (e.g. smart appliances, smart plugs)

[1199] software:

[1200] OpenAI API (API for using generative AI models)

[1201] Python (the programming language used to implement the program)

[1202] Data processing and calculation

[1203] server:

[1204] The server hosts a generative AI model that generates optimal product and electronic device combinations based on user requests. User requests are sent to the server in text format via a voice recognition device. The server uses the OpenAI API to send prompts to the generative AI model, which then generates optimal recommendations.

[1205] Terminal (voice recognition device):

[1206] The speech recognizer recognizes the user's speech and converts it into text, which is then sent to a server where it is processed by a generative AI model, which then sends the generated suggestions back to the speech recognizer and presents them to the user in audio or text form.

[1207] User:

[1208] A user speaks their request to a speech recognizer. For example, if they say, "I want new kitchen appliances," the speech recognizer converts this into text and sends it to the server. The server uses a generative AI model to generate optimal suggestions and presents them to the user through the speech recognizer.

[1209] Specific examples

[1210] If a user says, "I want new kitchen appliances," the speech recognizer converts this request into text and sends it to the server, which then sends the following prompt to the generative AI model:

[1211] Example prompt sentence:

[1212] If a user says, "I want new kitchen appliances," suggest the optimal combination of smart appliances.

[1213] Based on this prompt, the generative AI model generates suggestions such as, "Combining a smart oven and a smart refrigerator will improve cooking efficiency." These suggestions are sent from the server to a speech recognition device and presented to the user as voice or text.

[1214] In this way, users can enjoy an efficient and satisfying shopping experience in physical stores.

[1215] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1216] Step 1:

[1217] A user speaks a request to a speech recognition device. For example, the user speaks, "I want new kitchen appliances." The input is the user's voice, and the output is speech data from the speech recognition device.

[1218] Step 2:

[1219] A speech recognizer converts a user's speech into text format. The input is speech data and the output is text data. This conversion is performed using speech recognition technology.

[1220] Step 3:

[1221] The speech recognition device sends the converted text data to the server. The input is text data, and the output is data sent to the server.

[1222] Step 4:

[1223] The server analyzes the received text data and generates a prompt to send to the generative AI model. The input is text data, and the output is a prompt. An example of a specific prompt would be, "If the user says, 'I want new kitchen appliances,' please suggest the optimal combination of smart appliances."

[1224] Step 5:

[1225] The server generates a prompt and sends it to the generative AI model using the OpenAI API. The input is the prompt, and the output is the proposed data from the generative AI model.

[1226] Step 6:

[1227] The generative AI model generates optimal suggestions based on the prompt. The input is the prompt, and the output is the suggested text. For example, a suggestion might be, "Combining a smart oven and a smart refrigerator will improve your cooking efficiency."

[1228] Step 7:

[1229] The server receives the generated suggested text and sends it to the speech recognizer, where the input is the suggested text and the output is the data sent to the speech recognizer.

[1230] Step 8:

[1231] The speech recognition device receives the suggested text and presents it to the user as voice or text. The input is the suggested text, and the output is what is presented to the user. In concrete terms, the speech recognition device tells the user by voice, "Combining a smart oven and a smart refrigerator will improve your cooking efficiency."

[1232] In this way, users can enjoy efficient and satisfying shopping at physical stores.

[1233] Example 3

[1234] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1235] Conventional smart home appliance systems have the problem of not providing optimal operation based on the user's lifestyle and preferences, which means that user convenience is not sufficiently improved. Additionally, users must manually configure the settings, which makes operation cumbersome.

[1236] The specification process by the specification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes a voice recognition device linked with a generative AI, means for the voice recognition device to suggest, purchase, and assist in setting up products that can be linked with the user in an interactive format, means for the voice recognition device to automatically connect with other electronic devices, means for collecting user lifestyle habit data, means for analyzing the collected data and learning the user's lifestyle habits and preferences, and means for optimizing the operation of the electronic device based on the learned data. This enables optimal operation based on the user's lifestyle habits and preferences to be automatically performed, significantly improving user convenience.

[1237] "Generative AI" is an artificial intelligence system that learns based on user data and makes optimal suggestions and operations.

[1238] A "voice recognizer" is a device that recognizes a user's voice and understands and processes instructions and commands.

[1239] "Dialogue style" refers to the way in which users and systems communicate with each other through natural conversation.

[1240] "Product suggestion" is the process of recommending appropriate products based on a user's needs and preferences.

[1241] "Purchase" refers to the user taking steps to acquire a suggested product.

[1242] "Setting assistance" is a function that assists users when setting up electronic devices or systems.

[1243] "Electronic devices" refers to devices that can be connected to the Internet and operated, such as smart speakers and smart lights.

[1244] "Lifestyle data" refers to information about a user's daily behavior and preferences.

[1245] "Analyzing data" refers to processing collected data and extracting meaningful information and patterns.

[1246] "Operation optimization" refers to adjusting the operation of electronic devices to the optimal state based on the user's lifestyle and preferences.

[1247] MODE FOR CARRYING OUT THE INVENTION

[1248] This invention is a system that uses a voice recognition device linked to generative AI to optimize the operation of electronic devices based on the user's lifestyle and preferences. Specific embodiments of this system are described below.

[1249] 1. System Configuration

[1250] The server is responsible for building a generative AI model and collecting and analyzing the user's lifestyle data. The server trains the generative AI model using machine learning libraries (e.g., TensorFlow, PyTorch).

[1251] The devices include voice recognition devices (e.g., smart speakers) and other electronic devices (e.g., smart lights). These devices record the user's operation history and send it to the server.

[1252] Users operate the voice recognition device on a daily basis to provide lifestyle data. For example, data is collected when a user says "good night" to a smart speaker at 10 p.m.

[1253] 2. Program Processing

[1254] The server collects the user's operation history sent from the voice recognition device. The collected data is analyzed within the server to learn the user's lifestyle and preferences. For example, it may detect a pattern that the user goes to bed at 10 p.m. every night.

[1255] The server trains a generative AI model based on the analysis results, which then inputs the user's lifestyle data and suggests optimal ways to operate electronic devices.

[1256] The server then uses the learned model to optimize the operation of electronic devices, for example by instructing a voice recognition device to automatically turn off smart lights at 10 p.m.

[1257] The terminal operates according to instructions from the server, allowing the user to enjoy an optimized environment without any special operations.

[1258] 3. Specific Examples

[1259] For example, if a user has a habit of going to bed at 10pm every night, the following actions will occur:

[1260] The user says "good night" to their smart speaker at 10pm.

[1261] The device (smart speaker) records the time and command and sends it to the server.

[1262] The server analyzes data from the past month and detects that the user goes to bed at 10pm every night.

[1263] The server trains a generative AI model to learn this pattern.

[1264] The server instructs the voice recognition device to automatically turn off the smart light at 10pm.

[1265] The device (smart speaker) receives instructions from the server and automatically turns off the smart light.

[1266] 4. Examples of prompts

[1267] An example of a prompt to input to a generative AI model is as follows:

[1268] Enter the user's lifestyle data. For example, if the user goes to bed at 10pm every day, enter "User goes to bed at 10pm every day."

[1269] Using this prompt, the generative AI model can input the user's lifestyle data and suggest optimal electronic device operations.

[1270] As described above, the present invention is a system that significantly improves user convenience by optimizing the operation of an electronic device based on the user's lifestyle and preferences. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[1271] Step 1: Data collection

[1272] The server cooperates with a voice recognition device to collect lifestyle data of the user.

[1273] The terminal (voice recognition device) records the user's operation history and transmits it to the server.

[1274] A user operates a voice recognition device on a daily basis, for example, by saying "good night" at 10 p.m.

[1275] Input: The user's voice command and its duration.

[1276] Output: Operation history data sent to the server.

[1277] Specific operation: When a user says "good night" to the voice recognition device at 10 p.m., the voice recognition device records the time and command and sends it to the server.

[1278] Step 2: Data analysis

[1279] The server analyzes the collected data and extracts the user's lifestyle habits and preferences.

[1280] The server uses machine learning algorithms to detect patterns in the data.

[1281] Input: Collected operation history data.

[1282] Output: Analysis results about the user's lifestyle and preferences.

[1283] Specific operation: The server analyzes data from the past month and detects that the user says "good night" every day at 10 p.m.

[1284] Step 3: Model training

[1285] The server trains a generative AI model based on the analysis results.

[1286] The server builds the model using machine learning libraries such as TensorFlow and PyTorch.

[1287] Input: Analysis of the user's lifestyle and preferences.

[1288] Output: A trained generative AI model.

[1289] How it works: The server uses the user's lifestyle data as input to train a generative AI model. For example, the model learns the pattern that "the user goes to bed at 10 p.m. every night."

[1290] Step 4: Operation optimization

[1291] The server optimizes the operation of the electronic device based on the learned model.

[1292] The server automates the settings of electronic devices to suit the user's lifestyle.

[1293] Input: A trained generative AI model.

[1294] Output: Optimized electronic device operating instructions.

[1295] Specific operation: The server instructs the voice recognition device to automatically turn off the smart light at 10 p.m.

[1296] Step 5: Execution

[1297] The terminal (voice recognition device or smart light) operates according to instructions from the server.

[1298] The user can enjoy the optimized environment without any special operation.

[1299] Input: Operation instructions from the server.

[1300] Output: The electronic action that is performed.

[1301] Specific operation: At 10pm, the voice recognition device receives instructions from the server and automatically turns off the smart light.

[1302] (Application example 3)

[1303] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1304] While conventional smart home appliance systems can optimize the operation of appliances based on a user's lifestyle and preferences, they lack the functionality to optimize the shopping experience in physical stores. As a result, users are unable to receive sale information or recommended products at the right time when shopping in physical stores, making it difficult to shop efficiently. Furthermore, in-store navigation and product suggestions are lacking, which hinders the user's shopping experience.

[1305] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1306] In this invention, the server includes a smart speaker linked to a generative AI, means for the smart speaker to suggest, purchase, and set up products that can be linked through a conversation with the user, means for the smart speaker to automatically connect to other smart home appliances, means for the generative AI to learn the user's shopping habits and preferences and optimize the shopping experience at a physical store based on the learning, means for the generative AI to notify the user of sale information and recommended products according to the user's shopping time, and means for the generative AI to navigate within the store and suggest products based on the user's preferences, thereby enabling the user to have an efficient and comfortable shopping experience at a physical store.

[1307] "Generative AI" is an artificial intelligence that learns the user's lifestyle habits and preferences and makes optimal suggestions and operations based on that.

[1308] A "smart speaker" is a device that has voice recognition capabilities and provides information and controls home appliances through conversation with the user.

[1309] "A means of suggesting, purchasing, and assisting with setting up products that can be linked to the user in a conversational format" refers to a function in which a smart speaker suggests products to the user through voice dialogue, assists with the purchase process, and assists with setting up.

[1310] "Means for automatically connecting with other smart home appliances" refers to a function that enables the smart speaker to automatically communicate with other smart home appliances and operate them in coordination.

[1311] "A means of learning users' shopping habits and preferences and optimizing the in-store shopping experience based on that" refers to a function in which generative AI analyzes users' shopping patterns and preferences and, based on that, makes shopping in physical stores more efficient and comfortable.

[1312] "A means of notifying users of sale information and recommended products according to their shopping time" is a function in which generative AI predicts users' shopping times and notifies them of sale information and recommended products on their smartphones at those times.

[1313] "Means for in-store navigation and product suggestions based on user preferences" refers to a function in which generative AI assists users in moving around in physical stores and suggests products that suit the user's preferences.

[1314] To implement this invention, we use a smart speaker, a smartphone, and a navigation system for a physical store that are linked to generative AI. The specific system configuration and its operation are described below.

[1315] System Configuration

[1316] 1. Smart Speaker:

[1317] It is a device with voice recognition capabilities that provides information and controls home appliances through conversation with the user.

[1318] It works in conjunction with generative AI to learn the user's lifestyle habits and preferences.

[1319] 2. Generative AI:

[1320] It is an artificial intelligence that learns the user's shopping habits and preferences and makes optimal suggestions and operations based on them.

[1321] To optimize the in-store shopping experience, we notify users of special offers and recommended products at the time of their shopping.

[1322] 3. Smartphone:

[1323] Users receive notifications from generative AI on devices they carry with them.

[1324] Display in-store navigation and product suggestions.

[1325] 4. In-store navigation system:

[1326] This system assists users in moving around the store and makes product suggestions based on the user's preferences.

[1327] Operation explanation

[1328] 1. User habits:

[1329] The smart speaker works with generative AI to learn a user's lifestyle and shopping habits, for example, learning that the user shops at a specific supermarket every Saturday at 10 a.m.

[1330] 2. Special Offers and Recommended Product Notifications:

[1331] Generative AI notifies users of sale information and recommended products on their smartphones according to their shopping time, allowing them to shop efficiently.

[1332] 3. In-store navigation and product suggestions:

[1333] When a user arrives at a physical store, a smartphone application will begin navigating the store, and generative AI will suggest products based on the user's preferences and help them navigate the store.

[1334] Specific examples

[1335] For example, if a user has a habit of shopping at the supermarket every Saturday at 10 a.m., the generative AI will notify the user of sales information and recommended products on their smartphone at that time. When the user arrives at the supermarket, the smartphone application will begin navigating the store, and the generative AI will suggest products based on the user's preferences.

[1336] Prompt Sentence Examples

[1337] "Build an application that learns your grocery shopping habits every Saturday at 10 AM and notifies you of special offers and product recommendations at that time."

[1338] The above is a specific embodiment for carrying out the present invention.

[1339] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1340] Step 1:

[1341] User habits learning

[1342] The server collects the user's lifestyle and shopping habits through the smart speaker. Specifically, it collects data that the user shops at a specific supermarket every Saturday at 10:00 AM. The input is the user's behavioral data, and the output is the user's habit patterns. The server inputs this data into a generative AI model to learn the user's habits.

[1343] Step 2:

[1344] Special offers and recommended product notifications

[1345] The server uses a generative AI model to generate sale information and recommended products tailored to the user's shopping time. The input is the user's habit patterns and store sale information, and the output is the notification content. The server sends this notification content to the user's smartphone and notifies them. Specifically, the server sends sale information and recommended products via push notification to coincide with the user's shopping time.

[1346] Step 3:

[1347] Start in-store navigation

[1348] When a user arrives at a physical store, the smartphone application starts in-store navigation. The input is the user's location information, and the output is the navigation route. The server receives the user's location information, calculates the optimal route within the store, and sends it to the smartphone. Specifically, it displays the route to the location of the product the user is looking for.

[1349] Step 4:

[1350] Product suggestion

[1351] The server uses a generative AI model to make product suggestions based on the user's preferences. The input is the user's preference data and store inventory information, and the output is a list of suggested products. The server sends this list of suggested products to the smartphone and displays it to the user. Specifically, it lists products that the user is likely to be interested in and displays them on the application.

[1352] Step 5:

[1353] Collecting user feedback

[1354] After the user finishes shopping, the smartphone application collects feedback from the user. The input is the user's feedback, and the output is improvements and new habit data. The server inputs this feedback into the generative AI model to improve the accuracy of next suggestions and navigation. Specifically, it evaluates whether the user purchased the suggested products and whether the navigation was helpful.

[1355] The above are the specific processing steps for carrying out the present invention.

[1356] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1357] "Example 1"

[1358] One embodiment of the present invention is a system that combines a smart speaker linked to a generative AI with an emotion engine. In this system, the emotion engine recognizes the user's emotions through conversation with the user. For example, if the user says, "I'm tired today," the emotion engine recognizes that the user is tired. It then conveys this information to the generative AI. Based on this information, the generative AI suggests a combination of smart home appliances that best suits the user's emotions, such as a smart speaker that plays relaxing music or a smart light that emits a warm glow.

[1359] "Example 2"

[1360] Another embodiment of the present invention is a system in which an emotion engine optimizes the operation of smart home appliances based on a user's emotions. In this system, the emotion engine recognizes the user's emotions and instructs a generative AI to operate smart home appliances in a way that is appropriate for that emotion. For example, if the generative AI recognizes that the user is feeling happy, it instructs a smart speaker to play enjoyable music that the user likes.

[1361] "Example 3"

[1362] Another embodiment of the present invention is a system in which an emotion engine proposes an optimal combination of smart home appliances based on a user's emotions. In this system, the emotion engine recognizes the user's emotions and instructs a generative AI to select a combination of smart home appliances appropriate for that emotion. For example, if the emotion engine recognizes that the user is sad, the generative AI instructs a smart light to emit a calming light and simultaneously instructs a smart speaker to play calming music that the user prefers.

[1363] The processing flow of each embodiment will be described below.

[1364] "Example 1"

[1365] Step 1: The emotion engine recognizes the user's emotions through conversation with the user.

[1366] Step 2: The emotion engine conveys the recognized emotion information to the generative AI.

[1367] Step 3: Based on this information, the generative AI suggests a combination of smart home appliances that best suits the user's emotions.

[1368] "Example 2"

[1369] Step 1: The emotion engine recognizes the user's emotion.

[1370] Step 2: Based on the recognized emotions, the emotion engine issues instructions to the generative AI to optimize the operation of smart home appliances.

[1371] Step 3: The generative AI follows the instructions and operates the smart appliances.

[1372] "Example 3"

[1373] Step 1: The emotion engine recognizes the user's emotion.

[1374] Step 2: Based on the recognized emotions, the emotion engine instructs the generative AI on the optimal combination of smart home appliances.

[1375] Step 3: The generative AI follows the instructions and suggests combinations of smart appliances.

[1376] Example 1

[1377] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1378] Conventional smart speakers can perform simple operations and make suggestions based on the user's requests, but they are unable to make optimal suggestions or settings based on the user's emotions and lifestyle. Furthermore, they lack the ability to link with multiple electronic devices, preventing them from fully improving user convenience. This makes it difficult to provide the highly personalized experience that users desire.

[1379] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1380] In this invention, the server includes a voice recognition device linked to a generative artificial intelligence, means for the voice recognition device to suggest, purchase, and assist in setting up products that can be linked with the user in a conversational format, means for the voice recognition device to automatically connect to other electronic devices, an emotion analysis device for the voice recognition device to analyze the user's emotions, means for transmitting the emotion information analyzed by the emotion analysis device to the generative artificial intelligence, and means for the generative artificial intelligence to suggest an optimal combination of electronic devices based on the emotion information, thereby enabling highly personalized suggestions and settings based on the user's emotions and lifestyle habits.

[1381] "Generative AI" is an AI technology that analyzes user input data and generates appropriate responses and suggestions.

[1382] A "voice recognition device" is a device that converts a user's voice into text data.

[1383] "Electronic devices" include devices that can be connected to the Internet and operated, such as smart heaters and smart lights.

[1384] An "emotion analysis device" is a device that analyzes emotions from a user's comments and actions and provides that information to other systems.

[1385] "Suggestion" is the act of recommending optimal products and settings based on the user's requests and feelings.

[1386] "Settings assistance" is a support function that automatically configures electronic devices based on suggestions selected by the user.

[1387] "Conversational" refers to the way in which a user and a system exchange information through dialogue.

[1388] "Automatic connection" refers to the ability of the system to establish communication with other electronic devices without user intervention.

[1389] "Emotion information" is data that indicates the emotional state of the user analyzed by the emotion analysis device.

[1390] "Personalized" refers to providing services and suggestions that are optimized to suit each user's individual preferences and circumstances.

[1391] This invention is a system that uses a voice recognition device linked to generative artificial intelligence to suggest products, assist with purchasing, and assist with setup through a conversation with the user. It also has the function of analyzing the user's emotions and proposing the optimal combination of electronic devices based on that information.

[1392] Hardware and software used

[1393] 1. Speech Recognition Device: Uses speech recognition technology to convert the user's voice into text data. Specifically, it uses the Google Cloud Speech-to-Text API.

[1394] 2. Generative AI: OpenAI's GPT-4 is used to analyze user requests and emotional information and generate appropriate suggestions.

[1395] 3. Sentiment analyzer: IBM Watson Tone Analyzer is used to analyze emotions from user speech.

[1396] 4. Speech synthesis technology: Amazon Polly is used to communicate the generated suggestions to the user aloud.

[1397] 5. Electronics: This includes household appliances that can be connected to the internet and controlled, such as smart heaters and smart lights.

[1398] Specific operation of the system

[1399] Collaboration between smart speakers and generative AI

[1400] 1. The user says, "I want to heat the room."

[1401] A user speaks to a voice recognition device, saying, "I want to heat up the room."

[1402] 2. The device recognizes the voice and converts it into text data.

[1403] The device uses the Google Cloud Speech-to-Text API to convert speech into text data.

[1404] 3. The device sends the text data to the server.

[1405] The terminal transmits the converted text data to a server via the Internet.

[1406] 4. The server generates suggestions using generative artificial intelligence.

[1407] The server inputs the received text data into OpenAI's GPT-4 and generates a suggestion such as, "A smart heater is suitable. Would you like help setting it up?"

[1408] 5. The server sends the generated proposal to the device.

[1409] The server sends the generated proposal to the terminal.

[1410] 6. The device will then audibly communicate the suggestions to the user.

[1411] The device uses Amazon Polly to translate suggestions into speech and communicate them to the user.

[1412] 7. When the user requests assistance, the device connects to the smart heater and configures it.

[1413] If the user answers "yes," the device will use the Wi-Fi module to connect to the smart heater's API and configure it.

[1414] Collaboration between smart speakers, generative AI, and emotion engines

[1415] 1. A user says, "I'm tired today."

[1416] A user speaks to a speech recognition device, saying, "I'm tired today."

[1417] 2. The device recognizes the voice and converts it into text data.

[1418] The device uses the Google Cloud Speech-to-Text API to convert speech into text data.

[1419] 3. The device sends the text data to the server.

[1420] The terminal transmits the converted text data to a server via the Internet.

[1421] 4. The server analyzes the user's emotions using an emotion analyzer.

[1422] The server uses IBM Watson Tone Analyzer to analyze the user's emotions and recognizes them as "tired."

[1423] 5. The server sends the emotional information to the generative AI.

[1424] The server sends the emotion information to OpenAI's GPT-4.

[1425] 6. The server generates suggestions using generative artificial intelligence.

[1426] Based on the emotional information, the server generates suggestions such as a smart speaker that plays relaxing music or a smart light that emits a warm glow.

[1427] 7. The server sends the generated proposal to the device.

[1428] The server sends the generated proposal to the terminal.

[1429] 8. The device will then audibly communicate the suggestions to the user.

[1430] The device uses Amazon Polly to translate suggestions into speech and communicate them to the user.

[1431] Prompt Sentence Examples

[1432] 1. A prompt sentence suggested by the smart speaker to a user who says, "I want to heat up the room":

[1433] The user says, "I want to heat my room." Generate a response that suggests suitable smart appliances and assists with the setup.

[1434] 2. Prompt sentence suggested in cooperation with the emotion analyzer to a user who says "I'm tired today":

[1435] A user says, "I'm tired today." The emotion analyzer recognizes the user's emotion as "tired." Generate smart home appliance recommendations that match the user's emotion.

[1436] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1437] Collaboration between smart speakers and generative AI

[1438] Step 1:

[1439] The user says, "I want to heat the room."

[1440] Input: User voice input

[1441] Output: Audio data

[1442] Specific operation: The user speaks to the smart speaker, saying, "I want to heat up the room."

[1443] Step 2:

[1444] The device recognizes the voice and converts it into text data.

[1445] Input: Audio data

[1446] Output: Text data

[1447] Specific operation: The device uses the Google Cloud Speech-to-Text API to convert speech into text data.

[1448] Step 3:

[1449] The terminal transmits the text data to the server.

[1450] Input: Text data

[1451] Output: Send data to the server

[1452] Specific operation: The device sends the converted text data to a server via the Internet.

[1453] Step 4:

[1454] The server generates suggestions using generative artificial intelligence.

[1455] Input: Text data

[1456] Output: Suggested text

[1457] Specific operation: The server inputs the received text data into OpenAI's GPT-4 and generates a suggestion such as, "A smart heater is suitable. Would you like help setting it up?"

[1458] Step 5:

[1459] The server sends the generated proposal to the terminal.

[1460] Input: Suggested text

[1461] Output: Sending data to the terminal

[1462] Specific operation: The server sends the generated proposal to the terminal.

[1463] Step 6:

[1464] The device will communicate the suggestions to the user aloud.

[1465] Input: Suggested text

[1466] Output: Audio data

[1467] What it does: The device uses Amazon Polly to translate suggestions into speech and communicate them to the user.

[1468] Step 7:

[1469] When the user requests assistance, the device connects to the smart heater and configures it.

[1470] Input: User's voice input (e.g. "Yes")

[1471] Output: Smart heater setup complete

[1472] Specific operation: If the user answers "yes," the device will use the Wi-Fi module to connect to the smart heater's API and perform the configuration.

[1473] Collaboration between smart speakers, generative AI, and emotion engines

[1474] Step 1:

[1475] A user says, "I'm tired today."

[1476] Input: User voice input

[1477] Output: Audio data

[1478] Specific action: The user speaks to the smart speaker, "I'm tired today."

[1479] Step 2:

[1480] The device recognizes the voice and converts it into text data.

[1481] Input: Audio data

[1482] Output: Text data

[1483] Specific operation: The device uses the Google Cloud Speech-to-Text API to convert speech into text data.

[1484] Step 3:

[1485] The terminal transmits the text data to the server.

[1486] Input: Text data

[1487] Output: Send data to the server

[1488] Specific operation: The device sends the converted text data to a server via the Internet.

[1489] Step 4:

[1490] The server uses an emotion analysis device to analyze the user's emotions.

[1491] Input: Text data

[1492] Output: Emotional information

[1493] Specific operation: The server uses IBM Watson Tone Analyzer to analyze the user's emotions and recognizes them as "tired."

[1494] Step 5:

[1495] The server sends the emotional information to the generative artificial intelligence.

[1496] Input: Emotion information

[1497] Output: Sending data to generative AI

[1498] Specific operation: The server sends the emotion information to OpenAI's GPT-4.

[1499] Step 6:

[1500] The server generates suggestions using generative artificial intelligence.

[1501] Input: Emotion information

[1502] Output: Suggested text

[1503] Specific operation: Based on the emotional information, the server generates suggestions such as a smart speaker that plays relaxing music or a smart light that emits warm light.

[1504] Step 7:

[1505] The server sends the generated proposal to the terminal.

[1506] Input: Suggested text

[1507] Output: Sending data to the terminal

[1508] Specific operation: The server sends the generated proposal to the terminal.

[1509] Step 8:

[1510] The device will communicate the suggestions to the user aloud.

[1511] Input: Suggested text

[1512] Output: Audio data

[1513] What it does: The device uses Amazon Polly to translate suggestions into speech and communicate them to the user.

[1514] (Application example 1)

[1515] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1516] While conventional smart speakers and smart home appliances can provide suggestions and setup assistance based on user needs, they face the challenge of being unable to provide optimal suggestions based on the user's emotions and circumstances. Furthermore, there has been a lack of concrete measures to improve the shopping experience in physical stores.

[1517] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a smart speaker linked with a generative AI, means for the smart speaker to suggest, purchase, and assist in setting up products that can be linked with the user in a conversational format, means for the smart speaker to automatically connect with other smart home appliances, an emotion engine that recognizes the user's emotions, and means for the generative AI to suggest optimal products and services based on the emotions recognized by the emotion engine. This makes it possible to make optimal suggestions based on the user's emotions and situation.

[1518] "Generative AI" is artificial intelligence that generates appropriate responses and suggestions based on user input and the situation.

[1519] A "smart speaker" is a device that has voice recognition capabilities and can perform various operations and provide information through conversations with the user.

[1520] "Means for suggesting, purchasing, and assisting with setting up products" refers to a function that suggests appropriate products based on the user's request, assists with the purchasing process, and also supports post-purchase setup.

[1521] "Means for connecting with other smart home appliances" is a function that allows the smart speaker to communicate with other smart home appliances and operate them in cooperation with each other.

[1522] An "emotion engine" is a technology that analyzes emotions from a user's statements and actions and recognizes those emotions.

[1523] "Means of suggesting optimal products and services" refers to a function in which generative AI suggests the most suitable products and services to users based on the user's emotions recognized by the emotion engine.

[1524] To implement this invention, a smart speaker, emotion engine, and smartphone linked with generative AI are used. The specific system configuration and operation are described below.

[1525] System configuration

[1526] 1. Smart speaker: A device with voice recognition capabilities that can be used to operate or provide information through conversation with the user.

[1527] 2. Generative AI: This is artificial intelligence that generates appropriate responses and suggestions based on user input and context. It uses the OpenAI API.

[1528] 3. Emotion Engine: This technology analyzes emotions from user comments and actions and recognizes those emotions. It uses the Emotion Engine API.

[1529] 4. Smartphone: This is the device that users use in physical stores and works in conjunction with the smart speaker.

[1530] How it works

[1531] 1. Getting User Input

[1532] When a user speaks to their smartphone, the voice input is sent to the smart speaker, which converts the speech into text and sends it to the emotion engine.

[1533] 2. Emotional Recognition

[1534] The emotion engine analyzes the user's emotions from the received text and sends the results to the generative AI. For example, if a user says, "I'm tired today," the emotion engine will recognize this as "tired."

[1535] 3. Proposal Generation

[1536] Based on the emotional information received from the emotion engine, generative AI can suggest optimal products and services to users, such as "relaxation drinks and massage chairs."

[1537] 4. Presenting the proposal

[1538] The suggestions generated by the generative AI are presented to the user via a voice message through the smart speaker. If the user accepts the suggestions, the smart speaker will automatically connect to other smart home appliances and configure them.

[1539] Specific examples

[1540] When a user says "I'm tired today" to their smartphone in a physical store, the application uses an emotion engine to recognize "fatigue" and sends a prompt to the generative AI model saying, "What products or services would you suggest when the user is tired?" The generative AI model then suggests "relaxation drinks and massage chairs."

[1541] Prompt Sentence Examples

[1542] "What products or services would you suggest to users when they're feeling tired?"

[1543] "What smart home appliances would you suggest when a user says, 'I want to heat up my room.'"

[1544] In this way, a system is realized that can make optimal suggestions according to the user's emotions and situation.

[1545] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1546] Step 1:

[1547] A user speaks into a smartphone. The user's voice input is captured by the smartphone's microphone and obtained as voice data. The input is the user's voice, and the output is voice data.

[1548] Step 2:

[1549] The smartphone converts the voice data into text data. Speech recognition software (e.g., Google Speech-to-Text API) is used to convert the voice data into text data. The input is voice data, and the output is text data.

[1550] Step 3:

[1551] The smartphone sends text data to an emotion engine. The emotion engine (e.g., Emotion Engine API) analyzes the text data and recognizes the user's emotion. The input is text data, and the output is emotion data (e.g., "tired").

[1552] Step 4:

[1553] The emotion engine sends the recognized emotion data to the generative AI. The generative AI (e.g., OpenAI API) generates prompts based on the emotion data and suggests appropriate products or services. The input is emotion data, and the output is suggestion data (e.g., "relaxation drinks and massage chairs").

[1554] Step 5:

[1555] The generative AI sends the generated proposal data to a smartphone. The smartphone receives the proposal data and presents it to the user via voice. The input is the proposal data, and the output is voice data.

[1556] Step 6:

[1557] If the user accepts the suggestion, the smartphone sends instructions to the smart speaker, which then connects to and configures other smart appliances. The input is the user's approval data, and the output is the configuration data for the smart appliances.

[1558] In this way, a system is realized that can make optimal suggestions according to the user's emotions and situation.

[1559] Example 2

[1560] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1561] Conventional smart home appliance systems have difficulty proposing optimal combinations and operations of home appliances based on user needs and emotions. Furthermore, the lack of conversational interaction with users and automatic connection functions limited the user experience. This made it difficult for users to effectively use smart home appliances.

[1562] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1563] In this invention, the server includes a voice recognition device linked to a generative artificial intelligence, means for the voice recognition device to suggest, purchase, and assist in setting up products that can be linked in a conversational format with the user, means for the voice recognition device to automatically connect to other electronic devices, means for acquiring user requests, means for analyzing the requests, means for sending prompts to the generative artificial intelligence based on the analysis results, means for the generative artificial intelligence to suggest an optimal combination of electronic devices, means for transmitting the suggestions to the voice recognition device, means for the voice recognition device to notify the user of the suggestions, means for recognizing the user's emotions, means for instructing the generative artificial intelligence to operate the electronic devices based on the emotions, means for the generative artificial intelligence to suggest an operation of the electronic devices based on the emotions, means for transmitting the suggestions to the voice recognition device, and means for the voice recognition device to notify the user of the suggestions. This makes it possible to suggest optimal combinations and operations of home appliances based on the user's requests and emotions.

[1564] "Generative AI" is an AI technology that generates optimal suggestions and operations based on the user's requests and emotions.

[1565] A "voice recognition device" is a device that recognizes the user's voice and gives instructions or suggestions based on that voice.

[1566] "A means for suggesting, purchasing, and assisting with setting up products through a conversational format with the user" is a function that suggests, purchases, and assists with setting up products through a voice conversation with the user.

[1567] The "means for automatically connecting with other electronic devices" is a function for the voice recognition device to automatically connect with other electronic devices.

[1568] The "means for acquiring a user's request" is a function for acquiring a request uttered by a user to a voice recognition device.

[1569] "Means for analyzing requests" refers to a function for analyzing acquired user requests and understanding their contents.

[1570] "Means for sending prompt sentences to the generative AI" is a function for sending instructions to the generative AI based on the analysis results.

[1571] The "means for proposing the optimal combination of electronic devices" is a function in which generative artificial intelligence proposes the optimal combination of electronic devices based on the user's requests.

[1572] The "means for transmitting the proposal content to the speech recognition device" is a function for transmitting the proposal content from the generative artificial intelligence to the speech recognition device.

[1573] The "means for the voice recognition device to notify the user of the content of the suggestion" is a function by which the voice recognition device notifies the user of the content of the suggestion.

[1574] "Means for recognizing the user's emotions" refers to a function that recognizes emotions by analyzing the user's facial expressions, tone of voice, etc.

[1575] "Means for instructing a generative artificial intelligence to operate an electronic device based on emotions" is a function that instructs a generative artificial intelligence to operate an electronic device based on recognized emotions.

[1576] "Means for suggesting operations for electronic devices based on emotions" is a function in which generative artificial intelligence suggests operations for electronic devices based on the user's emotions.

[1577] This invention is a system that uses a voice recognition device linked to generative artificial intelligence to suggest optimal combinations and operations of electronic devices based on the user's requests and emotions. Specific embodiments of this system are described below.

[1578] Hardware and Software Configuration

[1579] Hardware

[1580] Speech recognition device: A device that recognizes a user's voice and acquires their requests and emotions. Specifically, it includes a microphone, speaker, and processor.

[1581] Server: A device that analyzes data sent from the speech recognition device and sends prompts to the generative AI. It is equipped with a high-performance processor and large-capacity memory.

[1582] Electronics: Smart appliances and other electronic devices, such as smart coffee makers, smart plugs, and smart speakers.

[1583] software

[1584] Generative AI model: An AI model that generates optimal suggestions and operations based on the user's requests and emotions. For example, a model that uses natural language processing (NLP) technology.

[1585] Emotion recognition engine: Software that recognizes emotions by analyzing a user's facial expressions and tone of voice.

[1586] Data analysis software: Software for analyzing user requests and generating prompts for generative artificial intelligence.

[1587] Data processing and calculation

[1588] Acquiring and analyzing user requests

[1589] When a user says to the speech recognition device, "I want to make coffee automatically in the morning," the speech recognition device sends the speech data to the server. The server uses data analysis software to analyze the speech and sends the following prompt to the generative artificial intelligence:

[1590] A user says, "I want to automatically brew coffee in the morning." Please suggest the best combination of smart home appliances.

[1591] Proposals by generative artificial intelligence

[1592] The generative AI model analyzes the prompt and suggests the optimal combination of electronic devices. For example, it might suggest, "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time." The server then sends this suggestion to a voice recognizer, which then notifies the user.

[1593] Emotion recognition and operation suggestions

[1594] The emotion recognition engine analyzes the user's facial expressions and tone of voice and determines that the user is feeling happy. The emotion recognition engine then sends the following prompt to the generative AI:

[1595] The user is feeling happy. Tell your smart speaker to play some fun music they like.

[1596] The generative AI model analyzes this prompt and suggests to the smart speaker, "Instruct it to play enjoyable music that the user likes." The server sends this suggestion to the voice recognizer, which then notifies the user and actually plays the music.

[1597] Specific examples

[1598] If a user says, "I want to make coffee automatically in the morning," the speech recognizer sends the statement to the server. The server analyzes the statement and sends a prompt to the generative AI. The generative AI suggests, "By combining a smart coffee maker with a smart plug, you can make coffee automatically at a specified time." The server sends the suggestion to the speech recognizer, which notifies the user.

[1599] If the user is recognized as feeling joy, the emotion recognition engine sends a prompt to the generative AI. The generative AI suggests, "I'll instruct you to play some fun music that the user likes," and the server sends the suggestion to the voice recognition device. The voice recognition device then notifies the user, "I'll play some fun music," and actually plays the music.

[1600] In this way, a system is realized that suggests optimal combinations and operations of electronic devices based on the user's requests and feelings.

[1601] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1602] Step 1:

[1603] The user speaks a request to the voice recognition device.

[1604] Input: User's voice data

[1605] Output: Audio data sent to the speech recognizer

[1606] What happens: A user says, "I want to have coffee brewed automatically in the morning." The voice recognition device captures this speech with a microphone.

[1607] Step 2:

[1608] The voice recognition device sends the user's voice data to the server.

[1609] Input: Audio data captured by a speech recognizer

[1610] Output: Audio data sent to the server

[1611] Specific operation: The voice recognition device converts the captured voice data into a digital signal and sends it to the server.

[1612] Step 3:

[1613] The server analyzes the voice data and converts the user's request into text format.

[1614] Input: Audio data sent to the server

[1615] Output: User requests in text format

[1616] Specific operation: The server uses voice recognition software to analyze the voice data and generate text data such as "I want to make coffee automatically this morning."

[1617] Step 4:

[1618] The server sends a prompt to the generative artificial intelligence.

[1619] Input: User request in text format

[1620] Output: Prompt sent to the generative AI

[1621] Specific operation: The server generates the following prompt sentence based on the text data and sends it to the generative artificial intelligence.

[1622] A user says, "I want to automatically brew coffee in the morning." Please suggest the best combination of smart home appliances.

[1623] Step 5:

[1624] Generative artificial intelligence analyzes the prompt text and suggests the optimal combination of electronic devices.

[1625] Input: Prompt sent to the generative AI

[1626] Output: Proposal

[1627] Specific operation: The generative AI analyzes the prompt and generates a suggestion such as, "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time."

[1628] Step 6:

[1629] The server sends the suggestions to the speech recognition device.

[1630] Input: Proposal from generative artificial intelligence

[1631] Output: Suggestions sent to the speech recognizer

[1632] Specific operation: The server sends the suggestions received from the generative AI to the speech recognition device.

[1633] Step 7:

[1634] A voice recognition device notifies the user of the suggestions.

[1635] Input: Proposal sent from the server

[1636] Output: The suggestion that will be sent to the user

[1637] Specific operation: The voice recognition device notifies the user, "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at the specified time."

[1638] Step 8:

[1639] The emotion recognition engine recognizes the user's emotions.

[1640] Input: User facial expressions and tone of voice

[1641] Output: Perceived user sentiment

[1642] Specific behavior: The emotion recognition engine analyzes the user's facial expressions and tone of voice and recognizes that the user is feeling joy.

[1643] Step 9:

[1644] The emotion recognition engine sends prompts to the generative AI.

[1645] Input: Perceived user emotion

[1646] Output: Prompt sent to the generative AI

[1647] Specific operation: The emotion recognition engine generates the following prompt sentence and sends it to the generative artificial intelligence.

[1648] The user is feeling happy. Tell your smart speaker to play some fun music they like.

[1649] Step 10:

[1650] Generative AI analyzes prompts and suggests emotion-based electronic device controls.

[1651] Input: Prompt sent to the generative AI

[1652] Output: Proposal

[1653] Specific operation: The generative AI analyzes the prompt and generates a suggestion such as, "Instruct the system to play enjoyable music that the user likes."

[1654] Step 11:

[1655] The server sends the suggestions to the speech recognition device.

[1656] Input: Proposal from generative artificial intelligence

[1657] Output: Suggestions sent to the speech recognizer

[1658] Specific operation: The server sends the suggestions received from the generative AI to the speech recognition device.

[1659] Step 12:

[1660] A voice recognition device notifies the user of the suggestions.

[1661] Input: Proposal sent from the server

[1662] Output: The suggestion that will be sent to the user

[1663] Specific operation: The voice recognition device notifies the user, "I'm going to play some fun music," and then actually plays the music.

[1664] (Application example 2)

[1665] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1666] Conventional smart home appliance systems have few functions that suggest the optimal combination of home appliances based on the user's requests or that optimize the operation of home appliances according to the user's emotions. Furthermore, there have been no systems in physical stores that suggest the optimal combination of products and services based on the user's requests, or adjust the in-store environment according to the user's emotions. This has led to the issue of not being able to sufficiently improve user convenience and satisfaction.

[1667] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1668] In this invention, the server includes a smart speaker linked to a generative AI, a means for the smart speaker to suggest, purchase, and assist with the setup of products that can be linked through a conversation with the user, a means for the smart speaker to automatically connect to other smart home appliances, a means for the smart speaker to suggest an optimal combination of products and services based on the user's requests in a physical store, and a means for the smart speaker to recognize the user's emotions and adjust the in-store environment accordingly. This makes it possible to suggest optimal products and services based on the user's requests, and further, by realizing environmental adjustments according to the user's emotions, it is possible to significantly improve user convenience and satisfaction.

[1669] "Generative AI" refers to artificial intelligence that generates optimal suggestions and answers based on user requests and input.

[1670] A "smart speaker" is a speaker that has voice recognition capabilities, can connect to the Internet to provide various information, and can operate other smart home appliances.

[1671] "Conversational" refers to a form of interaction in which the user asks questions or gives instructions by voice, and the system responds by voice.

[1672] "Product proposal" refers to the act of selecting the most suitable product based on the user's requests and conditions and presenting it to the user.

[1673] "Purchase" refers to the act of a user actually buying a suggested product.

[1674] "Setup assistance" refers to the act of providing assistance to users in properly setting up the products or services they have purchased.

[1675] "Means for automatically connecting with other smart home appliances" refers to a function that enables a smart speaker to automatically communicate with other smart home appliances and operate in cooperation with them.

[1676] A "brick and mortar store" is a store that exists in a physical location and where users can visit in person to purchase goods or services.

[1677] "User requirements" refer to the specific needs and desires that users have for the system.

[1678] "Optimal combination of products and services" means selecting multiple products and services that best meet the user's needs and combining them to propose them.

[1679] "User emotion" refers to the psychological state that a user feels, such as happiness, sadness, or surprise.

[1680] "In-store environment" refers to physical elements such as music, lighting, and temperature within a physical store.

[1681] "Means to adjust the environment" refers to the function of changing the music, lighting, etc. in the store according to the user's emotions.

[1682] The system for implementing this invention is centered around a smart speaker linked to generative AI. The smart speaker suggests, purchases, and assists with the setup of compatible products through a conversation with the user. The smart speaker also automatically connects with other smart home appliances and suggests optimal combinations of products and services based on the user's requests in a physical store. Furthermore, the smart speaker recognizes the user's emotions and adjusts the in-store environment accordingly.

[1683] Hardware and software used

[1684] Hardware: Smartphones, smart speakers, in-store smart speakers, smart lighting systems

[1685] Software: OpenAI API, EmotionRecognizer library, HTTP request library (requests)

[1686] System Operation

[1687] The server uses generative AI to propose optimal combinations of products and services based on the user's needs. When a user inputs their needs into a smartphone app, the generative AI generates optimal suggestions based on those needs. For example, if a user inputs "I'm looking for a new smartphone," the generative AI will suggest "a combination of the latest smartphone and recommended accessories."

[1688] The server also uses the EmotionRecognizer library to recognize the user's emotions. Once the server recognizes the user's emotions, it sends instructions to the store's smart speakers and smart lighting system via HTTP requests. For example, if the server recognizes that the user is happy, it will play cheerful music and brighten the lights in the store.

[1689] Specific examples

[1690] When a user uses their smartphone in a physical store and types "I'm looking for a new smartphone," the generative AI generates the best suggestions using the following prompt:

[1691] Prompt Sentence Examples

[1692] Recommend the best combination of products and services based on the user's needs: I'm looking for a new smartphone

[1693] Based on this prompt, the generative AI will suggest a combination of the latest smartphone and recommended accessories. Furthermore, if the EmotionRecognizer library recognizes the user's emotion as "joy," the server will instruct the store's smart speakers to play cheerful music and the smart lighting system to brighten the lights.

[1694] In this way, it is possible to propose optimal products and services based on the user's needs and adjust the environment to suit the user's emotions, which will significantly improve user convenience and satisfaction.

[1695] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1696] Step 1:

[1697] The user enters their request into the smartphone app.

[1698] Input: User's request (e.g. "I'm looking for a new smartphone")

[1699] Output: User request data

[1700] Specific operation: The user opens the smartphone app, enters a request in the text box, and presses the send button.

[1701] Step 2:

[1702] The terminal transmits the user's request data to the server.

[1703] Input: User's desired data

[1704] Output: Request data sent to the server

[1705] Specific operation: The smartphone app sends the user's requested data to the server via an HTTP request.

[1706] Step 3:

[1707] The server uses the generative AI model to generate the optimal combination of products and services.

[1708] Input: User's desired data

[1709] Output: Optimal product and service combinations

[1710] Specific operation: The server calls the OpenAI API, sends the prompt sentence to the generative AI model, and receives the generated suggestions.

[1711] Example prompt: "Suggest the best combination of products and services based on the user's request: I'm looking for a new smartphone."

[1712] Step 4:

[1713] The server sends the generated proposal to the terminal.

[1714] Input: Optimal product and service combinations

[1715] Output: Proposal data sent to the device

[1716] Specific operation: The server sends the generated proposal to the smartphone app as an HTTP response.

[1717] Step 5:

[1718] The terminal displays the proposed data to the user.

[1719] Input: Proposal data

[1720] Output: The proposal displayed to the user

[1721] Specific operation: The smartphone app receives the proposed data and displays it on the screen.

[1722] Step 6:

[1723] The device uses the EmotionRecognizer library to recognize the user's emotions.

[1724] Input: User's voice and facial expression data

[1725] Output: User emotion data

[1726] Specific operation: The smartphone app captures the user's voice and facial expressions using the camera and microphone, and recognizes emotions using the EmotionRecognizer library.

[1727] Step 7:

[1728] The device transmits the emotion data to the server.

[1729] Input: User emotion data

[1730] Output: Emotion data sent to the server

[1731] Specific operation: The smartphone app sends emotion data to the server via an HTTP request.

[1732] Step 8:

[1733] The server generates instructions to adjust the environment in the store based on the emotion data.

[1734] Input: User emotion data

[1735] Output: Environmental adjustment instruction data

[1736] Specific operation: The server analyzes the emotional data, determines appropriate music and lighting settings, and generates instruction data.

[1737] Step 9:

[1738] The server sends environmental adjustment instruction data to the smart speakers and smart lighting systems in the store.

[1739] Input: Environmental adjustment instruction data

[1740] Output: Instruction data sent to smart speakers and smart lighting systems in the store

[1741] Specific operation: The server sends instruction data to the smart speakers and smart lighting systems in the store via HTTP requests.

[1742] Step 10:

[1743] Smart speakers and smart lighting systems in the store will adjust the environment according to instructions.

[1744] Input: Instruction data

[1745] Output: Calibrated in-store environment

[1746] What it does: A smart speaker plays fun music and a smart lighting system brightens the lights.

[1747] Example 3

[1748] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1749] Conventional smart home appliance systems have difficulty optimizing operation based on the user's lifestyle habits and emotions, and have not been able to fully improve user convenience. Furthermore, they lacked the functionality to suggest optimal combinations of home appliances based on the user's needs, which required the user to configure them themselves. Furthermore, since they were unable to suggest home appliance operation based on emotions, it was difficult to provide a comfortable environment that responded to the user's emotions.

[1750] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1751] In this invention, the server includes a voice recognition device linked to a generative AI, means for the voice recognition device to suggest, purchase, and assist with the setup of products that can be linked to the user in a conversational format, means for the voice recognition device to automatically connect to other electronic devices, means for collecting data on the user's lifestyle habits and preferences, means for the generative AI to learn the collected data and determine optimal electronic device operations, and means for the emotion engine to recognize the user's emotions and suggest optimal electronic device operations based on those emotions. This enables optimal home appliance operation based on the user's lifestyle habits and emotions, significantly improving user convenience. It also allows the server to suggest optimal combinations of home appliances based on the user's requests, eliminating the need for the user to manually configure the appliances. Furthermore, it enables suggesting home appliance operations based on emotions, providing a comfortable environment that responds to the user's emotions.

[1752] "Generative AI" is an artificial intelligence system that learns from user data and performs optimal operations and suggestions based on that data.

[1753] A "voice recognition device" is a device that recognizes a user's voice and processes the voice data.

[1754] "Electronic devices" refers to any electrically operated equipment, including smart appliances and other digital devices.

[1755] An "emotion engine" is a system that recognizes the user's emotions and performs appropriate operations or suggestions based on those emotions.

[1756] "Data collection means" refers to a method or device for collecting data about a user's lifestyle and preferences.

[1757] The "operation determination means" refers to a method or device for determining the optimum operation of an electronic device based on collected data.

[1758] The "operation suggestion means" refers to a method or device for suggesting the optimum operation of an electronic device based on the user's feelings and requests.

[1759] "Conversational" refers to a format in which a user and a speech recognition device exchange information through natural dialogue.

[1760] "Means for suggesting, purchasing, and assisting in setting up products" refers to a method or device for suggesting appropriate products to users, assisting in the purchasing process, and further assisting in setting up the products.

[1761] "Automatic connection means" refers to a method or device for automatically connecting a voice recognition device to another electronic device.

[1762] This invention is a system that uses a voice recognition device linked to generative AI to realize optimal operation of electronic devices based on the user's lifestyle and emotions. Specific embodiments of this system are described below.

[1763] Hardware and software used

[1764] 1. Hardware

[1765] Speech recognition device: A device that recognizes a user's voice and processes the voice data. A specific example is a smart speaker (e.g., a typical voice assistant device).

[1766] Electronics: Smart appliances and other digital devices. Examples include smart lights (e.g., common smart lighting devices) and smart speakers.

[1767] 2. Software

[1768] Generative AI model: An artificial intelligence system that learns from user data and performs optimal operations and suggestions based on that data. A specific example is a general generative AI model.

[1769] Emotion engine: A system that recognizes the user's emotions and performs appropriate operations or suggestions based on those emotions. A specific example is a general emotion recognition API.

[1770] System Operation

[1771] 1. Data Collection

[1772] Records the user's daily actions and preferences. For example, if the user says "good night" at 10 p.m. every day, that time is recorded.

[1773] The terminal (voice recognition device) collects user behavior data and sends it to the server.

[1774] 2. Data training

[1775] The server sends the collected data to the generative AI model.

[1776] The generative AI model learns the user's lifestyle habits and preferences, for example, learning the user's pattern of saying "good night" every day at 10 p.m.

[1777] 3. Operation optimization

[1778] The generative AI model determines the optimal operation of electronic devices based on the information it has learned.

[1779] The server sends operational instructions to the electronic devices. For example, at 10 p.m., the server instructs the voice recognition device to "turn off the smart light."

[1780] 4. Emotion recognition

[1781] The terminal (voice recognition device) captures the user's voice and facial expressions and sends them to the server.

[1782] The emotion engine analyzes the captured data and recognizes the user's emotion. For example, if the user speaks in a sad voice, the emotion engine will recognize the emotion as "sad."

[1783] 5. Emotion-based suggestions

[1784] Based on the emotions recognized by the emotion engine, it instructs the generative AI on how to optimally operate electronic devices.

[1785] The generative AI model determines the appropriate operation for the electronic device based on the user's emotion, and the server sends instructions to the device. For example, if the user is recognized as sad, the server will instruct a smart light to emit a calming light and a smart speaker to play calming music.

[1786] Specific examples

[1787] 1. Learning user lifestyle habits

[1788] The user says "good night" every night at 10pm.

[1789] The terminal records the time and sends it to the server.

[1790] The generative AI model learns the pattern and instructs the server to automatically turn off the smart lights at 10pm.

[1791] 2. Emotion-based recommendations

[1792] The user says in a sad voice, "I'm tired today."

[1793] The device captures the audio and sends it to the server.

[1794] The emotion engine recognizes this as "sad" and sends that information to the generative AI model.

[1795] A generative AI model instructs a smart light to "emit a soothing color of light" and a smart speaker to "play soothing music."

[1796] Prompt Sentence Examples

[1797] Lifestyle learning prompts

[1798] We've learned that the user goes to bed at 10pm every night. Tell the smart speaker to turn off the smart light at 10pm.

[1799] Emotion-based suggestion prompts

[1800] The user is recognized as sad. Instruct the smart light to emit a calming light color and the smart speaker to play the user's preferred calming music.

[1801] The flow of the identification process in the third embodiment will be described with reference to FIG.

[1802] Step 1:

[1803] Data collection

[1804] Recording the user's daily actions and preferences, for example, when the user says "good night" at 10pm every night.

[1805] The terminal (voice recognition device) collects the user's voice data and sends the data to the server.

[1806] Input: User's voice data

[1807] Data processing: Converts audio data into text data and adds a timestamp.

[1808] Output: Text data and timestamp

[1809] Step 2:

[1810] Data training

[1811] The server sends the collected data to the generative AI model.

[1812] The generative AI model learns the user's habits and preferences, for example, learning that the user says "good night" every night at 10 p.m.

[1813] Input: Text data and timestamp

[1814] Data calculation: Using pattern recognition algorithms, user behavior patterns are extracted.

[1815] Output: User behavior patterns

[1816] Step 3:

[1817] Operation optimization

[1818] The generative AI model determines the optimal operation of electronic devices based on the information it has learned.

[1819] The server sends operational instructions to the electronic devices. For example, at 10 p.m., the server instructs the voice recognition device to "turn off the smart light."

[1820] Input: User behavior patterns

[1821] Data calculation: Generate optimal operation sequences based on behavioral patterns.

[1822] Output: Operation instructions

[1823] Step 4:

[1824] emotion recognition

[1825] The terminal (voice recognition device) captures the user's voice and facial expressions and sends them to the server.

[1826] The emotion engine analyzes the captured data and recognizes the user's emotion. For example, if the user speaks in a sad voice, the emotion engine will recognize the emotion as "sad."

[1827] Input: User's voice data and facial expression data

[1828] Data processing: Analyze voice data and facial expression data to identify emotions.

[1829] Output: User's emotional state

[1830] Step 5:

[1831] Emotion-based suggestions

[1832] Based on the emotions recognized by the emotion engine, it instructs the generative AI on how to optimally operate electronic devices.

[1833] The generative AI model determines the appropriate operation for the electronic device based on the user's emotion, and the server sends instructions to the device. For example, if the user is recognized as sad, the server will instruct a smart light to emit a calming light and a smart speaker to play calming music.

[1834] Input: User's emotional state

[1835] Data Computation: Generate optimal operation sequences based on emotional states.

[1836] Output: Operation instructions

[1837] (Application example 3)

[1838] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1839] Conventional smart home appliance systems can optimize operation based on the user's lifestyle and preferences, but they were unable to optimize the environment or suggest products based on the customer's emotions and lifestyle in physical stores. In addition, there was a lack of means to provide an in-store environment based on the customer's emotions, making it difficult to improve the customer experience.

[1840] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1841] In this invention, the server includes a smart speaker linked to a generative AI, means for the smart speaker to suggest, purchase, and assist in setting up products that can be linked through a conversation with the user, means for the smart speaker to automatically connect to other smart home appliances, means for the generative AI to recognize the user's emotions and optimize the operation of the smart home appliances based on those emotions, means for the generative AI to optimize the in-store environment based on the customer's lifestyle and preferences in a physical store, means for the generative AI to provide an in-store environment that corresponds to the customer's emotions, and means for the generative AI to suggest products and services based on the customer's preferences. This makes it possible to provide an optimal environment and suggest products that correspond to the customer's emotions and lifestyle even in a physical store.

[1842] "Generative AI" is an artificial intelligence system that learns the user's lifestyle habits and preferences and then performs optimal operations and makes suggestions based on that.

[1843] A "smart speaker" is a speaker with a voice assistant function that uses voice recognition technology to interact with users and operate other smart home appliances in conjunction with them.

[1844] "A means of suggesting, purchasing, and assisting with setting up products through conversation with the user" is a function in which a smart speaker can converse with the user via voice to suggest products, assist with purchasing procedures, and assist with setting up.

[1845] "Means for automatically connecting with other smart home appliances" refers to a function that enables the smart speaker to automatically communicate with other smart home appliances and operate them in coordination.

[1846] "Means of recognizing user emotions and optimizing the operation of smart home appliances based on those emotions" refers to a function in which generative AI analyzes user emotions and adjusts the settings and operation of smart home appliances based on the results.

[1847] "A means of optimizing the in-store environment based on customers' lifestyle habits and preferences in physical stores" refers to a function in which generative AI learns customers' past behavioral data and preferences, and adjusts the in-store environment, such as lighting and music, based on that.

[1848] "Means of providing an in-store environment that responds to customer emotions" refers to a function in which generative AI recognizes customer emotions in real time and provides an environment, such as lighting and music, that is appropriate for those emotions.

[1849] "Means of suggesting products and services based on customer preferences" refers to a function in which generative AI analyzes a customer's past purchase history and preferences, and then suggests appropriate products and services based on that.

[1850] As an embodiment of the present invention, a system for improving customer experience in a physical store is constructed. A specific embodiment of the system is described below.

[1851] System configuration

[1852] The system includes smart speakers, smart glasses, smartphones, and other smart home appliances that are connected to generative AI. These devices communicate with a server and provide an optimal environment based on the customer's lifestyle and emotions.

[1853] Hardware and software used

[1854] Smart speaker: Uses voice recognition technology to communicate with users and connect with other smart home appliances.

[1855] Smart glasses: Equipped with a camera and display, they analyze customers' facial expressions and recognize their emotions.

[1856] Smartphone: Collects customer data through an application and sends it to a server.

[1857] Server: Runs generative AI models and processes data.

[1858] Generative AI model: Learns the user's lifestyle habits and emotions and provides optimal operations and suggestions.

[1859] Data processing and calculation

[1860] The server receives customer facial expression data sent from smart glasses or smartphones, analyzes the customer's emotions using an emotion recognition model, and then learns the customer's past behavioral data and preferences, optimizing the in-store environment (lighting, music, etc.) based on that data.

[1861] Specific examples

[1862] For example, when a customer enters a store, the smart glasses analyze the customer's facial expression and determine that they are relaxed. The server then uses a generative AI model to instruct the store to soften the lighting and play calming music. The display also shows the customer's most frequently purchased products.

[1863] Prompt Sentence Examples

[1864] "Analyze users' facial expressions and recognize their emotions. Based on the emotions detected, optimize lighting and music in the store and display products that match the user's preferences."

[1865] In this way, even in physical stores, it will be possible to provide the optimal environment and product suggestions that suit customers' emotions and lifestyles.

[1866] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1867] Step 1:

[1868] Acquiring customer facial expression data

[1869] The smart glasses or smartphone camera is used to capture customer facial expression data.

[1870] Input: Customer's face image

[1871] Output: Facial expression data (image data)

[1872] How it works: The camera in the smart glasses or smartphone captures the customer's face and sends the image data to a server.

[1873] Step 2:

[1874] emotion recognition

[1875] The server inputs the received facial expression data into an emotion recognition model to analyze the customer's emotions.

[1876] Input: facial expression data (image data)

[1877] Output: Emotion data (e.g., relaxed, stressed, etc.)

[1878] Specific Operation: The server uses an emotion recognition model (e.g., a machine learning model) to infer the customer's emotion from the facial expression data.

[1879] Step 3:

[1880] Referencing lifestyle data

[1881] The server references the customer's past behavioral data and preferences from a database.

[1882] Input: Customer ID

[1883] Output: Lifestyle data (e.g., visit time, purchase history, etc.)

[1884] Specific operation: The server queries the database using the customer ID as a key to obtain the customer's lifestyle habit data.

[1885] Step 4:

[1886] Environment optimization instructions

[1887] The server generates instructions to optimize the in-store environment (lighting, music, etc.) based on emotional data and lifestyle data.

[1888] Input: Emotion data, lifestyle data

[1889] Output: Environmental settings instructions (e.g. lighting color, type of music, etc.)

[1890] How it works: The server uses a generative AI model to calculate optimal environmental settings based on emotional and lifestyle data, and sends these instructions to smart appliances.

[1891] Step 5:

[1892] Run environment setup

[1893] Smart home appliances set up the environment within the store based on instructions from the server.

[1894] Input: Environment setting instructions

[1895] Output: Optimized store environment (e.g., changing lighting, playing music, etc.)

[1896] Specific actions: Smart lights change color and smart speakers play specified music.

[1897] Step 6:

[1898] View product suggestions

[1899] The server suggests products and services based on the customer's preferences and displays them on the display.

[1900] Input: lifestyle data, customer preferences

[1901] Output: Product suggestions (e.g. product list displayed on the screen)

[1902] Specific operation: The server analyzes the customer's past purchase history and preferences, selects appropriate products, and sends instructions to display them on the screen.

[1903] In this way, even in physical stores, it will be possible to provide the optimal environment and product suggestions that suit customers' emotions and lifestyles.

[1904] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1905] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1906] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1907] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1908] [Third embodiment]

[1909] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1910] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1911] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1912] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1913] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1914] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1915] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1916] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1917] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1918] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1919] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1920] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1921] "Example 1"

[1922] One embodiment of the present invention provides a smart speaker that works in conjunction with generative AI. This smart speaker suggests, purchases, and assists with the setup of smart home appliances through a conversation with the user. Specifically, when the user says, "I want to heat my room," the smart speaker works in conjunction with the generative AI to suggest, "A smart heater would be suitable. Would you like help setting it up?" When the user requests assistance, the smart speaker automatically connects to the smart heater and sets it up.

[1923] "Example 2"

[1924] Furthermore, generative AI can suggest optimal combinations of smart home appliances based on user requests. For example, if a user says, "I want to make coffee automatically in the morning," the smart speaker will suggest, "By combining a smart coffee maker with a smart plug, you can make coffee automatically at the specified time."

[1925] "Example 3"

[1926] Generative AI can also learn about a user's lifestyle and preferences and optimize the operation of smart home appliances accordingly. For example, if a user learns that they go to bed at 10 p.m. every night, the smart speaker will automatically turn off the smart lights at 10 p.m.

[1927] The processing flow of each embodiment will be described below.

[1928] "Example 1"

[1929] Step 1: The user says to the smart speaker, "I want to heat up my room."

[1930] Step 2: The smart speaker works with the generative AI to determine the best smart appliance for the user's needs. In this case, it determines that a smart heater is suitable.

[1931] Step 3: The smart speaker will suggest to the user, "A smart heater is suitable. Would you like help setting it up?"

[1932] Step 4: When the user requests assistance, the smart speaker will automatically connect to the smart heater and set it up.

[1933] "Example 2"

[1934] Step 1: The user says to the smart speaker, "I want to automatically brew coffee in the morning."

[1935] Step 2: The smart speaker works with the generative AI to determine the optimal combination of smart appliances for the user's needs. In this case, it determines that a smart coffee maker and a smart plug are the best combination.

[1936] Step 3: The smart speaker suggests to the user, "By combining your smart coffee maker with the smart plug, you can automatically brew coffee at the time you specify."

[1937] "Example 3"

[1938] Step 1: The generative AI learns the user's habits and preferences. In this case, it learns that the user goes to bed at 10pm every night.

[1939] Step 2: The smart speaker will use the learned information to optimize the operation of smart home appliances, in this case automatically turning off the smart lights at 10 o'clock.

[1940] Example 1

[1941] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1942] Conventional smart speakers only respond to users' voice inputs and lack the functionality to suggest appropriate smart home appliances based on the user's specific needs and assist with their setup. They also lack the functionality to optimize the operation of smart home appliances based on the user's lifestyle and preferences. This requires users to manually configure the appliances, reducing convenience.

[1943] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1944] In this invention, the server includes means for receiving a user's voice input, means for converting the voice into text, means for transmitting the text data to the server, means for the server to generate a response using a generative AI model, means for returning the response to the terminal, means for the terminal to convert the response into voice and suggest it to the user, means for receiving the user's response, and means for connecting to and configuring smart home appliances. This makes it possible to suggest appropriate smart home appliances based on the user's specific needs and automatically configure them. It also makes it possible to optimize the operation of smart home appliances based on the user's lifestyle and preferences.

[1945] "Means for receiving user voice input" refers to devices or software that capture voices uttered by the user and input them into the system.

[1946] "Speech-to-text means" refers to speech recognition technology for analyzing and converting captured speech data into corresponding text data.

[1947] The "means for transmitting text data to a server" refers to a communication protocol or interface for transmitting the converted text data to a server via a network.

[1948] "Means by which the server generates a response using a generative AI model" refers to the process by which the server generates an appropriate response using a generative AI model based on the text data received by the server.

[1949] The "means for returning a response to the terminal" refers to a communication protocol or interface for returning the generated response to the terminal via the network.

[1950] The "means for the terminal to convert the response into voice and make a suggestion to the user" refers to a device or software that converts the text response received from the server into voice using speech synthesis technology and makes a suggestion to the user.

[1951] The "means for receiving user responses" refers to a device or software for capturing responses made by users to suggestions and inputting them into the system.

[1952] "Means for connecting to and configuring smart home appliances" refers to devices or software that connect to the proposed smart home appliances via a communication protocol and automatically perform the necessary settings.

[1953] This invention is a system that uses a smart speaker linked to a generative AI model to suggest, purchase, and assist with the setup of smart home appliances based on user voice input. Specific embodiments of this system are described below.

[1954] Hardware and software used

[1955] Hardware

[1956] Smart speaker: A device that accepts a user's voice input and plays audio.

[1957] Server: A computer system for hosting generative AI models and performing data processing.

[1958] Smart appliances: Home appliances that can be controlled over a network, such as a smart heater.

[1959] software

[1960] Speech Recognition Software: Technology for converting speech to text, such as the Google Cloud Speech-to-Text API.

[1961] Generative AI models: Artificial intelligence models, such as OpenAI's GPT-3, that generate appropriate responses based on user requests.

[1962] Text-to-speech software: Technology for converting text to speech, such as Amazon Polly.

[1963] Communication Protocol: The protocol used to send and receive data, such as HTTPS.

[1964] System Operation

[1965] User voice input

[1966] The user says to the smart speaker, "I want to heat up the room," and the smart speaker captures the voice through its built-in microphone.

[1967] Speech-to-text

[1968] The smart speaker uses voice recognition software to convert the captured voice data into text data, which includes the phrase "I want to heat my room."

[1969] Sending text data

[1970] The smart speaker then sends the converted text data to the server using the HTTPS protocol to ensure data security.

[1971] Generating a response

[1972] The server inputs the received text data into the generative AI model, which generates an appropriate response based on the user's request. In this case, it generates the response, "A smart heater is suitable. Would you like help setting it up?"

[1973] Returning a response

[1974] The server then sends the generated response back to the smart speaker, again using the HTTPS protocol.

[1975] Playing audio

[1976] The smart speaker converts the text response received from the server into speech using speech synthesis software, and plays the converted speech from the speaker, suggesting to the user, "A smart heater is suitable. Would you like help setting it up?"

[1977] User Response

[1978] If the user responds "yes," the smart speaker again captures the audio and converts it to text using speech recognition software. This text data is then analyzed to confirm that the user is requesting assistance.

[1979] Smart appliance settings

[1980] The smart speaker automatically connects to the smart heater using communication protocols such as Wi-Fi or Bluetooth. Once the connection is established, the smart speaker configures the smart heater, such as setting the temperature and timer.

[1981] Examples and prompts

[1982] Specific examples

[1983] When a user says, "I want to warm up the room," the smart speaker will suggest, "A smart heater would be suitable. Would you like help setting it up?" If the user answers "yes," the smart speaker will automatically connect to the smart heater and complete the setup. For example, you can set the heater temperature to 22 degrees and set a timer to automatically turn it off after one hour.

[1984] Prompt Sentence Examples

[1985] "When a user says they want to heat their room, explain the system that suggests appropriate smart appliances and assists them in setting them up."

[1986] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1987] Step 1:

[1988] Accepting user voice input

[1989] A user says to a smart speaker, "I want to heat the room." The device (smart speaker) captures the voice through its built-in microphone. The input is the user's voice data, and the output is the captured voice data.

[1990] Step 2:

[1991] Convert speech to text

[1992] The device converts the captured voice data into text data using speech recognition software (e.g., Google Cloud Speech-to-Text API). The input is the captured voice data, and the output is the text data "I want to heat the room."

[1993] Step 3:

[1994] Send text data to the server

[1995] The terminal sends the converted text data to the server. The HTTPS protocol is used for transmission to ensure data security. The input is text data, and the output is the text data sent to the server.

[1996] Step 4:

[1997] The server generates a response using the generative AI model

[1998] The server inputs the received text data into a generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates an appropriate response based on the user's request. The input is text data, and the output is the response, "A smart heater is suitable. Would you like help setting it up?"

[1999] Step 5:

[2000] Sends the response back to the terminal

[2001] The server generates a response and sends it back to the device, again using the HTTPS protocol. The input is the generated response, and the output is the response sent back to the device.

[2002] Step 6:

[2003] The device converts the response into speech and suggests it to the user.

[2004] The device converts the text response received from the server into speech using speech synthesis software (e.g., Amazon Polly). The converted speech is played back through the speaker, suggesting to the user, "This smart heater is suitable. Would you like help setting it up?" The input is the text response, and the output is the speech response.

[2005] Step 7:

[2006] Receive the user's response

[2007] If the user responds "yes," the device again captures the voice and converts it to text using speech recognition software. This text data is then parsed to determine that the user is requesting assistance. The input is the user's voice response, and the output is text data.

[2008] Step 8:

[2009] Connect to smart home appliances and set them up

[2010] The terminal automatically connects to the smart heater using communication protocols such as Wi-Fi or Bluetooth. Once the connection is established, the terminal configures the smart heater, such as setting the temperature and timer. The input is the user's configuration request, and the output is the smart heater after the configuration is complete.

[2011] (Application example 1)

[2012] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2013] While existing smart speakers have the ability to suggest smart home appliances and assist with setup based on user needs, they lack functionality to improve the shopping experience in physical stores. As a result, users have difficulty finding the appropriate product section when searching for products in physical stores, and there is a lack of in-store navigation.

[2014] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2015] In this invention, the server includes a smart speaker linked with a generative AI, a means for the smart speaker to suggest, purchase, and set up products that can be linked with the user through a conversation, a means for the smart speaker to automatically connect with other smart home appliances, and a means for the smart speaker to suggest an appropriate product section based on the user's request in a physical store and provide in-store navigation, thereby improving the user's shopping experience in the physical store and enabling them to quickly find the appropriate product section.

[2016] "Generative AI" is an artificial intelligence technology that generates responses in natural language based on user input.

[2017] A "smart speaker" is a voice assistant device that has voice recognition capabilities and can perform various operations through conversations with the user.

[2018] "Conversational" refers to a form of interaction in which the user asks questions or gives instructions by voice, and the device responds by voice.

[2019] "Product suggestion" is the act of selecting appropriate products based on the user's requests and introducing them to the user.

[2020] "Purchase" refers to the user's action of taking steps to acquire the suggested product.

[2021] "Setup assistance" refers to the act of assisting users with initial setup and operation so that they can properly use the product they have purchased.

[2022] "Automatically connect to other smart home appliances" is a function that allows the smart speaker to communicate with and operate other smart home appliances via wireless communication based on user instructions.

[2023] "Suggesting the appropriate product section in a physical store based on the user's request" refers to the act of guiding a user to the section where a specific product is located when the user is searching for that product in a physical store.

[2024] "Providing in-store navigation" refers to a function that provides route guidance for users to reach the desired product section within a physical store.

[2025] As an embodiment of this invention, we will explain a system that uses a smart speaker linked to generative AI. This system provides product suggestions, purchasing assistance, and setup assistance through a conversation with the user, and also provides navigation within a physical store.

[2026] First, the smart speaker has a voice recognition function and converts the user's speech into text data. This text data is sent to a generative AI, which generates an appropriate response based on the user's request. The generative AI is implemented using, for example, OpenAI's API.

[2027] Generative AI suggests appropriate products based on user input. For example, if a user says, "I want warm clothes," the generative AI responds, "We have warm clothes in this section. Would you like me to show you?" This response is transmitted to the user through a smart speaker.

[2028] Next, if the user requests directions, the smart speaker will activate the in-store navigation function. This navigation function is linked to a database containing information about the sections in the store and will guide the user to the desired section. Navigation information is provided, for example, using Wi-Fi or Bluetooth beacons in the store.

[2029] As a concrete example, by inputting the following prompt sentence into a generative AI, a response that meets the user's request can be generated.

[2030] Example prompt sentence:

[2031] "If a user says, 'I want warm clothes,' generate an Assistant response that suggests appropriate products and provides navigation within the store."

[2032] The system will enhance the user shopping experience by integrating smart speakers, generative AI, and navigation functions. Users can quickly find the product section they are looking for in a physical store, and can also receive suggestions and setup assistance for smart home appliances, greatly improving user convenience.

[2033] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2034] Step 1:

[2035] A user says to a smart speaker, "I want warm clothes." The smart speaker uses its voice recognition function to convert this statement into text data. The input is the user's voice data, and the output is text data.

[2036] Step 2:

[2037] The smart speaker sends the converted text data to the generative AI, which takes this text data as input and creates a prompt to generate an appropriate response. The input is text data, and the output is a prompt.

[2038] Step 3:

[2039] The generative AI generates a response based on the prompt text according to the user's request. Specifically, the generative AI generates the response, "There are warm clothes in this section. Can you show me?" The input is the prompt text, and the output is the generated response text.

[2040] Step 4:

[2041] The smart speaker converts the generated response text into speech and transmits it to the user. The user listens to the smart speaker's response and decides whether to ask for guidance. The input is the generated response text, and the output is speech data.

[2042] Step 5:

[2043] When a user requests guidance, the smart speaker activates the navigation function, which connects to a database containing information about sections in the store and generates route information to guide the user to the desired section. The input is the user's instructions, and the output is route information.

[2044] Step 6:

[2045] The smart speaker then communicates the generated route information to the user by voice and guides the user to the desired section. The input is the route information and the output is voice data.

[2046] Step 7:

[2047] The user follows the guidance of the smart speaker to move around the store and reach the desired section. The input is voice data, and the output is the user's actions.

[2048] Example 2

[2049] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2050] Conventional smart home appliance systems required users to individually configure and operate each appliance, resulting in low convenience. Furthermore, they lacked the functionality to suggest optimal combinations of appliances based on the user's needs, requiring the user to spend time and effort finding the optimal combination. Furthermore, they lacked the functionality to optimize the operation of appliances based on the user's lifestyle and preferences. To address these issues, a system is needed that can suggest optimal combinations of appliances based on the user's needs and further optimize their operation.

[2051] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2052] In this invention, the server includes a voice recognition device linked to a generative artificial intelligence, means for the voice recognition device to suggest, purchase, and assist with setting up products that can be linked in a conversational format with the user, means for the voice recognition device to automatically connect to other electronic devices, means for converting the user's voice into text data, means for generating prompt sentences for inputting the text data to the generative artificial intelligence, means for the generative artificial intelligence to suggest an optimal combination of electronic devices, and means for communicating the suggestion to the user by voice. This makes it possible to automatically suggest an optimal combination of home appliances based on the user's requests and further optimize their operation.

[2053] "Generative AI" is an AI system that generates appropriate responses and suggestions based on user input.

[2054] A "voice recognition device" is a device that converts a user's voice into text data.

[2055] "User" refers to a person who uses this system.

[2056] "Conversational" is a format in which the user and the system interact, with the system responding to what the user says.

[2057] "Product suggestion" is the act of recommending appropriate products based on the user's requests.

[2058] "Purchase" is the act of a user acquiring a suggested product.

[2059] "Setting assistance" refers to the act of providing assistance to a user in properly setting up a proposed product.

[2060] "Electronic devices" refers to digital devices such as smart home appliances.

[2061] "Text data" is character information converted by a voice recognition device.

[2062] A "prompt sentence" is an instruction sentence to be input into generative artificial intelligence.

[2063] A "suggestion" is the optimal solution provided by generative artificial intelligence based on the user's request.

[2064] The "audio means" is a method for transmitting the generated suggestions to the user as audio.

[2065] This invention is a system that uses a voice recognition device linked to generative artificial intelligence to propose the optimal combination of electronic devices based on the user's request and optimize their operation. This system is implemented using the following hardware and software.

[2066] Hardware and software used

[2067] Hardware:

[2068] 1. Voice recognition devices (e.g. smart speakers)

[2069] 2. Server (a server for hosting generative AI)

[2070] software:

[2071] 1. Speech recognition software (e.g., software that converts speech into text data)

[2072] 2. Generative AI models (e.g., OpenAI's GPT-4)

[2073] Data processing and calculation

[2074] Voice recognition device:

[2075] When a user speaks a request to the voice recognition device, the voice recognition device uses built-in voice recognition software to convert the speech into text data. For example, if the user says, "I want to make coffee automatically in the morning," this speech is converted into text data that says, "I want to make coffee automatically in the morning."

[2076] server:

[2077] The server receives the text data sent from the speech recognition device. The server then generates a prompt to be input to the generative AI model. For example, this prompt might be in the form of, "The user wants to automatically make coffee in the morning. Which electronic device should I pair it with?"

[2078] The generative AI model analyzes this prompt and suggests optimal combinations of electronic devices, such as "Pairing a smart coffee maker with a smart plug will allow you to automatically brew coffee at a specified time."

[2079] The server sends the generated suggestions to a speech recognizer, which converts the suggestions into speech using speech synthesis software. Ultimately, the speech recognizer tells the user, "By combining your smart coffee maker with your smart plug, you can automatically brew coffee at the time you specify."

[2080] Specific examples

[2081] While getting ready in the morning, a user might say to the smart speaker, "I want to have coffee brewed automatically this morning." This triggers the system. The smart speaker uses voice recognition software to convert this speech into text data and send it to the server.

[2082] The server generates a prompt based on the received text data to be input to the generative AI model. An example of a prompt is as follows:

[2083] A user says he wants to automatically brew coffee in the morning. What electronic device would be a good combination?

[2084] A generative AI model analyzes this prompt and generates the following suggestions:

[2085] By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time.

[2086] The server sends the suggestion to the smart speaker, which uses speech synthesis software to convert the suggestion into speech and convey it to the user.

[2087] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2088] Step 1:

[2089] The user speaks their request to the smart speaker.

[2090] Specific behavior:

[2091] The user speaks to the smart speaker, "I want to have coffee brewed automatically in the morning."

[2092] Input: User's voice

[2093] Output: Audio data

[2094] Step 2:

[2095] The device (smart speaker) converts the voice into text data.

[2096] Specific behavior:

[2097] The device uses built-in voice recognition software to convert the user's speech into text data.

[2098] Input: Audio data

[2099] Output: Text data "I want to make coffee automatically in the morning."

[2100] Step 3:

[2101] The server receives the text data and generates a prompt sentence for the generative AI model.

[2102] Specific behavior:

[2103] The server receives the text data sent from the terminal and generates a prompt sentence to be input to the generative artificial intelligence model.

[2104] Input: Text data "I want to make coffee automatically in the morning"

[2105] Output: Prompt: "The user wants to automatically brew coffee in the morning. Which electronic device should they pair with?"

[2106] Step 4:

[2107] The server inputs prompts into the generative AI model, which then suggests the optimal combination of electronic devices.

[2108] Specific behavior:

[2109] The server inputs the generated prompt sentence into a generative artificial intelligence model, which then suggests the optimal combination of electronic devices.

[2110] Input: Prompt: "The user wants to automatically brew coffee in the morning. Which electronic device would be best to pair it with?"

[2111] Output: Suggestion: "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time."

[2112] Step 5:

[2113] The server receives the generated proposal and sends it to the terminal.

[2114] Specific behavior:

[2115] The server transmits the proposals received from the generative artificial intelligence model to the terminal.

[2116] Input: "Pair your smart coffee maker with a smart plug to automatically brew coffee at a specified time."

[2117] Output: Text data of the proposal

[2118] Step 6:

[2119] The device will communicate the suggestions to the user by voice.

[2120] Specific behavior:

[2121] The device converts the received suggestions into speech using speech synthesis software and conveys them to the user.

[2122] Input: Proposal text data

[2123] Output: "By combining a smart coffee maker with a smart plug, you can automatically brew coffee at a specified time."

[2124] (Application example 2)

[2125] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2126] Conventional systems using voice recognition devices and generative AI have limited functionality for suggesting optimal product and electronic device combinations based on user requests, and lack specific suggestions for improving the shopping experience, particularly in brick-and-mortar stores. Furthermore, they lack the functionality to optimize electronic device operation based on the user's lifestyle and preferences. This makes it difficult for users to shop efficiently and satisfactorily in brick-and-mortar stores.

[2127] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition device linked with a generative AI, means for the voice recognition device to suggest, purchase, and assist with setting up products that can be linked with the user in a conversational format, means for the voice recognition device to automatically connect with other electronic devices, and means for the voice recognition device to suggest optimal products and combinations based on the user's requests in order to improve the shopping experience in physical stores. This enables users to shop efficiently and satisfactorily in physical stores.

[2128] "Generative AI" is a type of artificial intelligence that generates optimal suggestions and answers based on user requests and input.

[2129] A "voice recognition device" is a device that recognizes a user's voice and processes instructions or information based on that voice.

[2130] "Conversational" means that the user and the system exchange information through natural dialogue.

[2131] "Product suggestion" is the act of recommending the most suitable product based on the user's requests and needs.

[2132] "Purchase" refers to the user's action of taking steps to acquire the suggested product.

[2133] "Setup assistance" refers to the act of assisting users in setting up purchased products or electronic devices.

[2134] "Electronic devices" is a general term for machines and devices that operate using electricity.

[2135] A "brick and mortar store" is a store that exists in a physical location and where customers can visit in person to purchase products.

[2136] "Shopping experience" refers to the overall experience and feelings a user has while selecting and purchasing a product.

[2137] "Lifestyle habits" refers to the user's behavioral patterns and habits in their daily lives.

[2138] "Preferences" refer to the personal tastes and preferences of a user.

[2139] "Optimizing operation" means adjusting the operation of an electronic device to the optimum state according to the user's needs and circumstances.

[2140] The system for carrying out the present invention is mainly composed of a speech recognition device linked to a generative AI. A specific embodiment of this system will be described below.

[2141] Hardware and Software Configuration

[2142] Hardware:

[2143] Voice recognition devices (e.g. smartphones, smart speakers)

[2144] Server (a server for hosting generative AI models)

[2145] Electronic devices (e.g. smart appliances, smart plugs)

[2146] software:

[2147] OpenAI API (API for using generative AI models)

[2148] Python (the programming language used to implement the program)

[2149] Data processing and calculation

[2150] server:

[2151] The server hosts a generative AI model that generates optimal product and electronic device combinations based on user requests. User requests are sent to the server in text format via a voice recognition device. The server uses the OpenAI API to send prompts to the generative AI model, which then generates optimal reco...

Claims

1. a first means for receiving a voice input indicative of a request from a user; second means for converting voice data of a voice input indicating a request from the user into first text data; a third means for generating a first prompt sentence for outputting a response to a request from the user based on the first text data; a fourth means for inputting the generated first prompt sentence into a generative AI model to generate the response; and a fifth means for playing the generated response from a smart speaker to suggest it to the user; the first means for receiving a voice input indicating a response of the user to the response played from the smart speaker; The second means converts voice data of a voice input indicating an answer of the user into second text data; and a sixth means for configuring the smart home appliance indicated in the response based on an analysis result of the second text data. system.

2. the first means for receiving a voice input from the user indicating a request related to the electronic device; The second means converts voice data of a voice input indicating a request from the user regarding the electronic device into third text data; the third means generates a second prompt sentence for outputting a combination of electronic devices in response to a request from the user regarding an electronic device, based on the third text data; The fourth means inputs the generated second prompt sentence into a generative AI model to generate a combination of electronic devices. The system of claim 1 .

3. a seventh means for recognizing an emotion of the user from text data obtained by converting voice data of the user's voice using an emotion engine; the third means generates a third prompt sentence for outputting a suggestion of a product or service according to the emotion of the user, based on the emotion of the user; The fourth means inputs the generated third prompt sentence into a generative AI model to generate the suggestion.

3. A system according to claim 1 or 2.

Images