System

Abstract

A system is provided.SOLUTION: The system includes a means for receiving a voice input, a means for analyzing the received voice input, a means for generating a response on the basis of an analysis result, a means for transmitting the generated response to a user terminal, and a means for displaying and voice-transmitting the response by the user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, it has become increasingly important for shopping facility operators to quickly and accurately provide the products and services that customers desire. However, there is a problem with limited staff numbers, making it difficult to respond to all customers immediately. In addition, customers often have difficulty finding the right product among the many options available. This can lead to a poor customer experience and a decrease in purchasing motivation. New technological solutions are needed to solve these issues and increase customer satisfaction. [Means for solving the problem]

[0005] The present invention relates to a system for receiving a voice input, analyzing the voice input, generating a response based on the analysis result, and transmitting the generated response to a user terminal. Specifically, the system includes the following means.

[0006] A means of accepting voice input

[0007] A means of analyzing the received voice input

[0008] A means of generating a response based on the parsed results

[0009] A means for transmitting the generated response to the user terminal.

[0010] A means for displaying and audibly issuing responses on the user's terminal

[0011] Furthermore, the system includes a means for searching for specific product information based on the analysis results and including that information in the generated response, as well as a means for accepting additional questions from the user and performing further analysis and generating a response based on those questions. This allows users to obtain the product information they need efficiently and in real time, improving customer satisfaction and reducing the workload of staff.

[0012] "Voice input" refers to the capture of verbal words or instructions from a user by a digital device.

[0013] "Means for receiving" refers to devices or software that identify voice input and collect it as data.

[0014] "Means for analysis" refers to devices or software that convert collected voice data into text data and understand and interpret its content.

[0015] A "means for generating a response" is a device or software for generating an appropriate response message based on the parsed data.

[0016] The "transmitting means" refers to a device or software for delivering the generated response message to the user's terminal.

[0017] The "display and voice transmission means" refers to a device or software for visually displaying and audibly reproducing a response message on a user terminal.

[0018] "Means for searching for product information" refers to devices or software for searching for related product information from databases, etc. based on the analysis results.

[0019] A "means for accepting follow-up questions" is a device or software for capturing further questions or requests from the user as voice input.

[0020] The "means for re-analyzing and generating a response" refers to a device or software for re-analyzing an additional question received and creating a new response message. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention provides a character avatar that utilizes speech recognition and generation AI to function as a shopping assistant and guidance system. The system includes means for accepting speech input, means for analyzing the speech data, means for generating a response based on the analysis results, means for transmitting the generated response to a user terminal, and means for displaying and audibly transmitting the response on the user terminal.

[0043] System Configuration

[0044] 1. User voice input

[0045] Users launch a dedicated application and voice their question or request, for example, "Which shoes are the most popular in this store?"

[0046] 2. Sending audio data

[0047] The device records the user's voice and sends it as data to the server.

[0048] 3. Analysis of audio data

[0049] The server converts the received voice data into text data using a speech recognition engine. A generative AI model is used to understand the user's intent from this text data. As a result of the analysis, the generative AI generates a query to identify the "most popular shoes."

[0050] 4. Information Search

[0051] The server uses the generated query to search for relevant information from a product database, for example, "the most popular shoes" to identify "XYZ model."

[0052] 5. Generating the Response

[0053] The server generates a response message with the search results, for example, a message containing information such as "Currently, the most popular item is the 'XYZ model' shoe."

[0054] 6. Sending a response to the user

[0055] The server sends the generated response message to the terminal.

[0056] The device conveys the received message to the user audibly and visually through a character avatar, who says, "Here are some popular shoes," while displaying an image of the product.

[0057] 7. Gathering Feedback

[0058] If the user wants more information, they can say something like, "Do these shoes come in other colors?"

[0059] The terminal transmits this additional audio data back to the server.

[0060] The server analyzes the voice data again, generates a new response, and sends it to the terminal.

[0061] The device will then display the received response audibly and visually using a character avatar, for example, "These shoes are available in red, blue, and black."

[0062] Specific examples

[0063] For example, if a user asks "Which shoes are the most popular in this store?" while in a store, the system operates as follows: First, the device accepts the user's voice input and sends it to the server. The server converts the voice data into text and analyzes the content. It then searches the product database based on the analysis results and identifies the "XYZ model." The server generates a response message saying "The XYZ model is the most popular" and sends it to the device. The device then conveys this response message aloud through a character avatar and simultaneously displays an image. If the user then asks, "Are these shoes available in other colors?", the same process is used to provide color information.

[0064] In this way, the present invention utilizes voice recognition and generative AI to provide users with a real-time and efficient shopping assistant service, thereby improving customer satisfaction and reducing the burden on staff.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] To begin interacting with the character avatar, users launch a dedicated application and speak their questions or requests into a microphone, for example, "Which shoes are the most popular in this store?"

[0068] Step 2:

[0069] The device records the user's voice input using a microphone and stores it as digital voice data.

[0070] Step 3:

[0071] The device converts the recorded audio data into an appropriate format (e.g., compresses and encodes it) and sends it to a server via the Internet.

[0072] Step 4:

[0073] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[0074] Step 5:

[0075] The server passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[0076] Step 6:

[0077] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[0078] Step 7:

[0079] The server receives the search results and generates a response message to provide to the user, for example, "The most popular shoe at the moment is the 'XYZ model'."

[0080] Step 8:

[0081] The server generates a response message and sends it to the terminal.

[0082] Step 9:

[0083] The device analyzes the received response message, and the character avatar communicates information to the user audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product.

[0084] Step 10:

[0085] If the user wants more information, they can ask a follow-up question, such as "Do these shoes come in other colors?"

[0086] Step 11:

[0087] The device re-records the additional audio data, converts it, and sends it to the server.

[0088] Step 12:

[0089] The server again analyzes the audio data and generates new text to generate appropriate responses to follow-up questions, such as "These shoes are available in red, blue, and black."

[0090] Step 13:

[0091] The server sends a new response message to the terminal.

[0092] Step 14:

[0093] The device then conveys a new response message to the user via a character avatar, who responds aloud, "These shoes are available in red, blue, and black," and displays the color variations on the screen.

[0094] Through the above series of steps, the system of the present invention responds to user requests in real time and provides necessary information, thereby increasing customer satisfaction.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] Conventional shopping assistant systems have had the problem of being unable to provide a quick and accurate response when users ask questions. In particular, when obtaining information using voice input, the accuracy of voice recognition and analysis is low, making it difficult to accurately understand the user's intent and provide an appropriate response. Furthermore, responses to follow-up questions from users can be slow, resulting in a decline in customer satisfaction.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes a means for converting received voice data into text data, a means for analyzing the user's intent using a generative AI model, and a means for generating a query based on the analysis and searching for information from a database. This increases the accuracy of voice recognition, making it possible to accurately analyze the user's intent and quickly generate an appropriate response.

[0100] "Voice input" refers to the act of recording the user's voice through a microphone on a device.

[0101] "Digital audio data" refers to data obtained by converting an analog audio signal into a digital format.

[0102] The "server" is a central system responsible for analyzing voice data, searching the database, generating responses, etc.

[0103] "Text data" is character information converted from speech by a speech recognition engine.

[0104] A "generative AI model" is an algorithm that uses artificial intelligence to understand user intent and generate a response.

[0105] A "query" is a query command for retrieving specific information from a database.

[0106] A "database" is a collection of data that stores specific product information and other related information.

[0107] A "response message" is a server-generated text answer to a user's question.

[0108] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[0109] A "character avatar" is a virtual character that displays a response message audibly and visually.

[0110] This invention is a shopping assistant and guidance system that utilizes speech recognition and generation AI, allowing users to search for product information through voice input and receive responses in real time.

[0111] The system includes the following components:

[0112] A means of accepting voice input

[0113] A means to record received voice input and send it to a server as digital audio data

[0114] A means of converting received voice data into text data

[0115] A means of analyzing user intent using generative AI models

[0116] A means of generating queries based on analysis and retrieving information from a database

[0117] A means for generating a response message based on the searched information and transmitting it to the user terminal.

[0118] Means for displaying a response message audibly and visually at the user terminal

[0119] As a specific example, we use the Google Cloud Speech-to-Text API to convert voice data into text, and then use a generative AI model such as OpenAI's GPT-3 to analyze the captured text data and generate appropriate responses to user questions.

[0120] For example, if a user asks "Which shoes are the most popular in this store?" by voice, the system works as follows: The user launches a dedicated application and speaks the question. The device records this voice and sends it to the server. The server converts the voice data into text data and analyzes it using a generative AI model. The resulting query is run against the database, and the "XYZ model" is identified as the most popular shoe. The server generates a response message and sends it to the device. The device communicates the response to the user audibly and visually through a character avatar.

[0121] Examples of prompts include "What are the most popular shoes in this store?" or "Do you have these shoes in other colors?"

[0122] In this way, by combining speech recognition and generative AI, the present invention provides highly accurate information search and precise responses in a short time, aiming to improve customer satisfaction and reduce the burden on store staff.

[0123] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0124] Step 1:

[0125] The user starts a dedicated application and performs voice input.

[0126] Input: User-uttered speech (e.g., "Which shoes are the most popular in this store?")

[0127] What it does: Records audio through the microphone on your smartphone or tablet.

[0128] Output: Recorded audio data

[0129] Step 2:

[0130] The device digitizes the recorded audio data and sends it to the server.

[0131] Input: Recorded audio data

[0132] What it does: Converts voice data into digital form and sends it as data packets to a server.

[0133] Output: Digital audio data sent to the server

[0134] Step 3:

[0135] The server converts the received voice data into text data.

[0136] Input: Digital audio data sent to the server

[0137] How it works: Converts audio data into text using the Google Cloud Speech-to-Text API or similar.

[0138] Output: Text data (e.g., "Which shoes are the most popular in this store?")

[0139] Step 4:

[0140] The server inputs the text data into a generative AI model and analyzes the user's intent.

[0141] Input: Text data

[0142] How it works: Text data is fed into a generative AI model such as OpenAI's GPT-3, which analyzes the user's intent (the content of the request).

[0143] Output: Analysis results (e.g., a query to identify "most popular shoes")

[0144] Step 5:

[0145] The server retrieves information from a database based on the analysis results.

[0146] Input: The query generated as a result of the analysis

[0147] What it does: Queries the product database to find information about the corresponding product.

[0148] Output: Search results (e.g., "XYZ model" shoes are the most popular)

[0149] Step 6:

[0150] The server generates a response message based on the search results and transmits it to the user terminal.

[0151] Input: Search results

[0152] What it does: Based on the search results, it generates a text response message to the user's question.

[0153] Output: Response message (e.g., "Currently, the most popular shoe is the XYZ model.")

[0154] Step 7:

[0155] The terminal transmits the response message received from the server to the user through the character avatar.

[0156] Input: Response message sent by the server

[0157] What it does: It analyzes the response message and communicates it to the user audibly and visually through a character avatar. For example, the avatar might say, "Here are some popular shoes," while displaying a product image.

[0158] Output: The response displayed and spoken to the user

[0159] Step 8:

[0160] If the user desires more information, they can ask additional questions by voice and start the process again.

[0161] Input: An additional spoken question from the user (e.g., "Do these shoes come in other colors?")

[0162] How it works: The device records audio and sends the digital audio data to the server, which then converts the audio data back into text, analyzes it with a generative AI model, generates a response message, and sends it to the device.

[0163] Output: An additional response that is displayed and spoken to the user (e.g., "These shoes come in red, blue, and black")

[0164] This allows users to efficiently obtain necessary product information in real time.

[0165] (Application example 1)

[0166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0167] Conventional shopping support systems for brick-and-mortar stores have the problem of not being able to provide efficient support because the product information search and in-store guidance that users face are not performed in real time.In addition, there is a lack of interaction through the user's voice, and there are also issues with familiarity and intuitive operability.

[0168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0169] In this invention, the server includes means for accepting voice input, means for analyzing the accepted voice input, means for generating a response based on the analysis result, means for transmitting the generated response to a user terminal, means for displaying and audibly transmitting the response on the user terminal, means for generating a response based on a specific question using a generative AI model, means for converting the generated response from text to voice, and means for converting voice data to text data using a voice recognition engine. This allows users to easily obtain product information and store guides by voice, enabling a familiar and intuitive shopping experience.

[0170] The "means for accepting voice input" refers to a microphone or peripheral device for capturing the user's voice, and is a mechanism for acquiring voice signals as digital data.

[0171] "Means for analyzing received voice input" refers to software or algorithms that convert voice data into text data and analyze the content to understand the user's intent or question.

[0172] "Means for generating a response based on the analysis results" refers to the process of using algorithms or software, particularly generative AI models, to generate appropriate responses based on the results of voice analysis.

[0173] "Means for transmitting the generated response to the user terminal" refers to a communication means or protocol for transferring the generated response data from the server to the user's device.

[0174] "Means for displaying and audibly transmitting a response on a user terminal" refers to a mechanism with a display and speaker for visually displaying and audibly playing received response data on a device.

[0175] A "means for generating a response based on a specific question using a generative AI model" is a computational model or algorithm for using generative AI technology to generate a meaningful response based on a user's question.

[0176] "Means for converting the generated response from text to speech" refers to speech synthesis technology or software for converting a text response into speech.

[0177] "Means for converting voice data into text data using a voice recognition engine" refers to a voice recognition technology that analyzes voice input and converts the voice into corresponding text data.

[0178] To implement this invention, a system that combines speech recognition, generative AI, speech synthesis, and communication technologies is required. This system is designed to support shopping in stores and uses smartphones as the main user terminal. The specific operation of the system and its components are described below.

[0179] System configuration

[0180] Accepting voice input

[0181] The user launches a dedicated application on their smartphone and inputs a question or request by voice, such as, "What is the most popular camera in this store?" The smartphone's microphone picks up this voice and captures it as digital data.

[0182] Analysis of voice data and conversion to text

[0183] The smartphone sends the captured voice data to a server installed in the cloud. The server converts the voice data into text data using a speech recognition engine (e.g., Google Speech Recognition API). The converted text data is passed to a generative AI model (e.g., GPT-3) to understand the user's intent.

[0184] Generating a response and retrieving product information

[0185] The server's generative AI model generates an appropriate response based on the analyzed text data. Specifically, it understands the user's question and generates the most appropriate answer. For example, a response such as "The most popular camera at present is model ABC" is generated. In this process, a query is sent to a product information database to obtain the latest sales information and popularity data.

[0186] Text-to-speech conversion of responses

[0187] The generated text response is converted to audio using a speech synthesis engine (e.g., Google Text-to-Speech API), allowing the user to hear the answer to their question aloud.

[0188] Sending and displaying a response to the user terminal

[0189] The server then sends the converted response data to the user's smartphone. The dedicated application then displays the response visually and audibly using a character avatar. For example, the character avatar may say, "Here's a popular camera," while also displaying an image of the product.

[0190] Examples of concrete examples and prompts

[0191] As a concrete example, consider a situation where a user is in a store and asks, "What is the most popular camera in this store?" The system accepts voice input, performs analysis and response generation on the server, and finally responds with audio and visual responses such as, "The most popular camera right now is model ABC." An example of a prompt sentence is as follows:

[0192] "What is the most popular camera in this store?"

[0193] This allows users to easily obtain product information and store guides via voice, providing a friendly and intuitive shopping experience.

[0194] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0195] Step 1:

[0196] A user launches a dedicated application installed on their smartphone and speaks a question or request, for example, "What is the most popular camera in this store?" The smartphone's microphone captures this voice and acquires it as digital data, which is the input and becomes the voice data for subsequent processing.

[0197] Step 2:

[0198] The device sends the captured audio data to the server. This is the audio data transmission process. The input from the device is the user's audio data, and the output is to send this data to the server. Specifically, data is sent to the server using an HTTP request or WebSocket.

[0199] Step 3:

[0200] The server converts the received voice data into text data using a speech recognition engine (e.g., Google Speech Recognition API). The input is the user's voice data, and the output is text data. This conversion is performed by the speech recognition engine. Specifically, it includes extracting features from the voice signal and analyzing phonemes.

[0201] Step 4:

[0202] The server uses a generative AI model (e.g., GPT-3) to generate a response based on the converted text data. The input is text data, and the output is the generated response text. This is where the generative AI model analyzes the meaning of the text and generates the optimal response. For example, a response such as "Currently, the most popular camera is the ABC model" is generated.

[0203] Step 5:

[0204] The server queries a product information database to find additional information to complement the generated response. The input is a text-based query, and the output is the corresponding product information. This step uses a database management system (e.g., MySQL) to look up the product information and parse the information.

[0205] Step 6:

[0206] The server converts the generated text response into audio using a speech synthesis engine (e.g., Google Text-to-Speech API). The input is the generated response text, and the output is the audio data. This conversion involves mapping text to audio and generating an audio file.

[0207] Step 7:

[0208] The server sends the generated voice data to the user's smartphone. The input is voice data, and the output is data transmission to the smartphone. This data transfer is performed using communication protocols such as HTTP requests and WebSocket.

[0209] Step 8:

[0210] The device uses a character avatar to display the received audio data visually and audibly. The input is the audio data sent from the server, and the output is audio playback and information display for the user. Specifically, the audio is played back through the smartphone's speaker, related information is displayed on the display, and the character avatar says things like, "Popular cameras are here."

[0211] The above processing flow allows users to easily obtain product information and store guides by voice, providing a familiar and intuitive shopping experience.

[0212] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0213] The present invention provides a character avatar that utilizes speech recognition, generative AI, and an emotion engine to function as a shopping assistant and guidance system. This system includes means for accepting a user's voice input, analyzing the voice input, generating a response based on the analysis results, transmitting the generated response to a user terminal, and displaying and vocalizing the response on the user terminal. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the user experience is further improved.

[0214] System Configuration

[0215] 1. User voice input

[0216] Users launch a dedicated application and voice their question or request, for example, "Which shoes are the most popular in this store?"

[0217] 2. Sending audio data

[0218] The device records the user's voice input with a microphone and stores it as digital audio data, which is then converted into a suitable format and sent to a server over the Internet.

[0219] 3. Analysis of audio data

[0220] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[0221] 4. Response generation based on analysis results

[0222] The server then passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[0223] 5. Searching for Information

[0224] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[0225] 6. Emotion Recognition with Emotion Engine

[0226] The server uses an emotion engine to recognize the user's emotional state from voice input and the user's past interactions, for example, determining whether the user is frustrated or happy.

[0227] 7. Regulating responses based on emotions

[0228] The server adjusts the response message based on the results of the emotion engine. For example, if the user is frustrated, it generates a more polite response or one that provides additional information.

[0229] 8. Sending the Response

[0230] The server generates a response message and sends it to the terminal. For example, the response message might say, "The most popular shoe at the moment is the 'XYZ model'. It comes in red, blue, and black."

[0231] 9. Display and Call to User

[0232] The device analyzes the received response message, and a character avatar communicates information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product and adding a polite comment based on the user's emotions.

[0233] 10. Receiving User Feedback

[0234] If the user wants more information, they can ask a follow-up question, such as "Do you have other colors?"

[0235] 11. Processing of follow-up questions

[0236] The device again records additional audio data and sends it to the server, which again analyzes the audio data and generates a new response, such as a message containing the information, "These shoes are available in red, blue, and black."

[0237] 12. Sending and Displaying Final Responses

[0238] The server sends a new response message to the terminal, which again provides the information to the user audibly and visually via the character avatar.

[0239] Specific examples

[0240] For example, if a user asks, "Which shoes are the most popular in this store?" while in a store, the system operates as follows: First, the user's voice input is recorded and sent to the server. The server converts the voice data into text and analyzes the content. The system then searches the product database based on the analysis results, identifies an "XYZ model," and recognizes the user's emotions using an emotion engine. For example, if the system recognizes that the user is excited, it generates a response containing a competitive expression such as "These shoes are especially recommended." The server then sends the generated response to the device, which uses a character avatar to convey the information both audibly and visually. If the user subsequently expresses interest and asks the follow-up question, "Do you have any other colors?", the system also provides color information through a similar process.

[0241] In this way, the system of the present invention combines speech recognition, generative AI, and an emotion engine to provide efficient real-time shopping support that adapts to the user's emotional state, thereby improving customer satisfaction and reducing the burden on staff.

[0242] The processing flow will be explained below.

[0243] Step 1:

[0244] The user launches the application and speaks a question or request into the microphone, for example, "Which shoes are the most popular in this store?"

[0245] Step 2:

[0246] The device records the user's voice input using a microphone and stores it as digital voice data.

[0247] Step 3:

[0248] The device converts the recorded audio data into an appropriate format (e.g., compresses and encodes it) and sends it to a server via the Internet.

[0249] Step 4:

[0250] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[0251] Step 5:

[0252] The server passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[0253] Step 6:

[0254] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[0255] Step 7:

[0256] The server receives the search results and passes the user's voice input data to the emotion engine, which analyzes the user's emotional state. For example, the emotion engine recognizes the user's voice data as "excited."

[0257] Step 8:

[0258] The server generates a response message based on the emotion engine results, depending on the user's emotions. For example, if the user is excited, the server creates a response such as "These shoes are especially recommended," including additional reasons for the recommendation.

[0259] Step 9:

[0260] The server sends the generated response message and appropriate product information to the terminal.

[0261] Step 10:

[0262] The device analyzes the response message and the character avatar communicates information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product with additional comments based on the user's emotions.

[0263] Step 11:

[0264] If the user wants more information, they can ask a follow-up question, such as "Do these shoes come in other colors?"

[0265] Step 12:

[0266] The device re-records the additional audio data, converts it, and sends it to the server.

[0267] Step 13:

[0268] The server again analyzes the voice data, generates new text, retrieves product color variation information from the database, and generates a new response message.

[0269] Step 14:

[0270] The server uses the emotion engine again to analyze the user's emotional state when asking the follow-up question and tailor an appropriate response message based on that emotional state. For example, if the user asks a question in a dissatisfied tone, the server generates a polite response such as, "We have a wide variety of colors, so please try them out."

[0271] Step 15:

[0272] The server sends a new response message to the terminal.

[0273] Step 16:

[0274] The device then re-prompts the user with a new response message, both audibly and visually, via a character avatar. The avatar responds audibly, saying, "These shoes are available in red, blue, and black," and displays the color variations on the screen. At the same time, it adds a comment that takes the user's feelings into consideration.

[0275] Through the above series of processing steps, the system of the present invention provides information in real time while adapting to the user's emotional state, realizing an efficient and fulfilling customer experience.

[0276] Example 2

[0277] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0278] In today's shopping experience, users want fast and accurate information, but traditional systems face the challenge of being unable to accurately grasp the user's intentions and emotions and provide appropriate responses. Furthermore, impersonal responses often fail to improve the user experience and leave the user feeling unsatisfied. This can lead to lower customer satisfaction and a loss of purchasing motivation.

[0279] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0280] In this invention, the server includes a means for passing voice data to a voice recognition engine and converting it into natural language text, a means for passing the converted text data to a generative AI model and analyzing the content of the text, and a means for recognizing the user's emotions and adjusting the response message based on the results. This makes it possible to accurately grasp the user's intentions and emotions and quickly generate an appropriate response.

[0281] (Definitions of important terms included in the claims)

[0282] "Voice input" is a means of recording the user's spoken words or voice as digital data.

[0283] "Digital audio data" refers to data obtained by converting an analog audio signal into a digital format.

[0284] "Recording" means the act of recording and saving the user's voice.

[0285] A "server" is a central processing unit that analyzes voice data and transmits the processing results to a user terminal.

[0286] A "speech recognition engine" is software or hardware that converts speech data into natural language text.

[0287] "Natural language text" is text data in a language that humans can understand that has been converted from voice data.

[0288] A "generative AI model" is an artificial intelligence algorithm designed to generate appropriate responses to specific inputs.

[0289] "Text content analysis" is the process of understanding the converted natural language text and identifying the user's intent.

[0290] A "response message" is text that contains a response or information to a user's inquiry.

[0291] "Emotion recognition" refers to determining a user's emotional state from the tone and content of their voice.

[0292] A "user terminal" is a digital device used by a user, including devices such as smartphones and tablets.

[0293] "Display and voice transmission" refers to displaying text information on the screen of the user terminal and transmitting it to the user by voice.

[0294] This system is a shopping assistant and guidance system that analyzes user voice input and generates responses. The system uses hardware and software that combines voice recognition, generative AI, secure communication methods, and an emotion recognition engine.

[0295] First, the user installs a dedicated application and uses a user device such as a smartphone or tablet to input voice commands. The user taps the microphone button to begin asking a question. For example, they can input a question by voice such as, "Which shoes are the most popular in this store?"

[0296] Next, the device will record the user's voice using a built-in or external microphone as digital audio data, store it in temporary memory, convert it into a predefined data format, and send it to the server using a secure communication protocol such as SSL / TLS.

[0297] The server first converts the voice data into natural language text using a speech recognition engine (e.g., Google Speech-to-Text API). For example, the voice data "Which shoes are the most popular in this store?" is converted into text "Which shoes are the most popular in this store?"

[0298] The converted text data is passed to a generative AI model (e.g., OpenAI GPT-4) for analysis. This analysis process understands the content of the input text to identify the user's intent. For example, the generative AI model generates an appropriate response in the form of a prompt: "The user is asking about 'the most popular shoes.' Please provide more information."

[0299] The server then searches a product database (e.g., MySQL or PostgreSQL) to retrieve relevant product information to generate a specific response. For example, it might find that "XYZ model shoes" are the most popular. During this process, the generative AI model uses the search results to create a specific response message.

[0300] To understand the user's emotional state, the server uses an emotion recognition engine (e.g., Microsoft Azure's Emotion API) to recognize emotions from voice input and past interaction data. For example, if the user is excited, a response such as "I highly recommend these shoes!" is generated.

[0301] The server then sends the generated response message to the user's device. The device receives the message and uses a character avatar to communicate information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product.

[0302] If the user wants more information, they can speak again, asking, "Do you have any other colors?" The device again records the voice data and sends it to the server. A response is generated and sent through the same process, providing the user with the latest information.

[0303] As described above, the present invention improves the user experience by accurately understanding the user's intentions and emotions and providing an optimal response.

[0304] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0305] Step 1:

[0306] The user launches the dedicated application and taps the microphone button to start voice input, for example, by speaking a question such as, "Which shoes are the most popular in this store?"

[0307] Step 2:

[0308] The device uses the smartphone's built-in microphone to record the user's voice as digital audio data. The recorded audio data is stored in short-term memory. The input is the user's voice, and the output is digital audio data.

[0309] Step 3:

[0310] The terminal transmits digital audio data to the server using a secure communication protocol such as SSL / TLS. The input is digital audio data, and the output is secure data transmission to the server.

[0311] Step 4:

[0312] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts it into natural language text. The input is digital voice data, and the output is natural language text.

[0313] Step 5:

[0314] The server passes the converted text data to a generative AI model (e.g., OpenAI GPT-4) and begins analyzing the text content. A prompt is created and the analysis proceeds. The input is natural language text, and the output is a prompt containing the analysis results.

[0315] Example: Prompt: "User is asking about 'Most Popular Shoes'. Please provide more information."

[0316] Step 6:

[0317] The server sends a query to a product database (e.g., MySQL or PostgreSQL) based on the analysis results of the generative AI model to search for related product information. The input is the analysis results, and the output is the query results.

[0318] Example: Retrieve information from a database that "XYZ model shoes" are the most popular.

[0319] Step 7:

[0320] The server uses an emotion recognition engine (e.g., Microsoft Azure's Emotion API) to recognize the user's emotions from voice input and past interaction data. The input is voice or text data, and the output is the user's emotional state.

[0321] For example, if the user is excited, a response like "I highly recommend these shoes!" will be generated.

[0322] Step 8:

[0323] The server uses a generative AI model to tailor a response message based on the recognized emotion. The generative AI model is used to create a specific response that reflects the emotional information. The input is the emotion recognition result, and the output is the tailored response message.

[0324] Step 9:

[0325] The server sends the generated response message to the user terminal. The input is the adjusted response message, and the output is the data transmission to the user terminal.

[0326] Step 10:

[0327] The terminal displays the received response message, and the character avatar provides information to the user audio and visually. The input is the response message, and the output is audio and visual display.

[0328] Example: An avatar says, "Here are some popular shoes," and displays an image of the product.

[0329] Step 11:

[0330] If the user wants more information, they again use voice input, in this case asking a question such as "Do you have any other colors?" The input is an additional voice question.

[0331] Step 12:

[0332] The terminal again records the additional audio data and sends it to the server. The input is the additional audio data, and the output is transmission to the server.

[0333] Step 13:

[0334] The server analyzes the new voice data and generates a new response. The same process is repeated to provide the information the user seeks. The input is a further voice question, and the output is a new response message.

[0335] For example: "These shoes come in red, blue, and black."

[0336] This enables the system to accurately understand the user's intentions and emotions and provide the most appropriate response.

[0337] (Application example 2)

[0338] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0339] Conventional shopping assistant systems only generate responses based on the user's voice input and do not consider the user's emotions, which prevents them from fully improving user satisfaction. Furthermore, they lack a means to provide information in an easy-to-understand visual and audio format, making it difficult to provide an intuitive and enjoyable user experience. Furthermore, there are limitations to the means to efficiently generate responses each time the user asks additional questions.

[0340] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0341] In this invention, the server includes means for accepting a voice input, means for analyzing the accepted voice input, means for generating a response based on the analysis result, means for transmitting the generated response to a user terminal, means for displaying the response and transmitting it audibly on the user terminal, means for recognizing emotions, means for adjusting the response based on the result of the emotion recognition means, and means for visually displaying the response and transmitting it audibly using a character avatar, thereby enabling response generation that takes the user's emotions into consideration and providing visual and audio information using a character avatar.

[0342] "Means for accepting voice input" refers to a device or function for recognizing voice input from a user and recording it as digital data.

[0343] "Means for analyzing received voice input" refers to technology for processing recorded voice data and converting it into natural language text.

[0344] The "means for generating a response based on the analysis results" refers to a generative AI model or algorithm that converts voice data into text and then creates an appropriate response based on that content.

[0345] The "means for transmitting the generated response to the user terminal" refers to a technique for transferring the generated response message from the server or central system to the user's device.

[0346] "Means for displaying and audibly transmitting a response on a user terminal" refers to equipment or software for visually displaying information and audibly transmitting a response on a user's device.

[0347] "Emotion recognition means" is a technology that analyzes emotions from a user's voice or text and identifies states such as joy, anger, sadness, and happiness.

[0348] The "means for adjusting a response based on the results of the emotion recognition means" is software or an algorithm for utilizing the results of the emotion recognition means to generate a response that is adapted to the emotional state of the user.

[0349] A "means for visually displaying and audibly transmitting responses using character avatars" is a device or software that uses animation or graphic characters to display and communicate information audibly.

[0350] To implement the present invention, the following system configuration and procedure are used.

[0351] System Configuration

[0352] 1. A means of accepting voice input

[0353] The user inputs voice using a smartphone application, and the smartphone's microphone records the voice and stores it as digital data.

[0354] 2. A means of analyzing the received voice input

[0355] This digital voice data is sent via a smartphone application over the internet to a server, which then uses a speech recognition engine such as Google Cloud Speech-to-Text to analyze the voice data and convert it into natural language text.

[0356] 3. A means of generating a response based on the analysis results

[0357] The server uses the parsed text data as input for a generative AI model (e.g., OpenAI API) that generates an appropriate response based on the user's question or request.

[0358] 4. A means of sending the generated response to the user terminal

[0359] The generated response is forwarded from the server to the user's smartphone.

[0360] 5. Means for displaying and audibly transmitting responses on the user terminal

[0361] The user's smartphone displays the received response in visual and audio formats, with a character avatar responding with audio and displaying text information on the screen for easy visual understanding by the user.

[0362] 6. Emotion recognition means

[0363] The server uses emotion recognition engines such as IBM Watson Tone Analyzer to determine the user's emotions based on the user's voice and text data.

[0364] 7. A means of adjusting responses based on the results of emotion recognition measures

[0365] Based on the results of emotion recognition, a generative AI model generates a tailored response that is adapted to the user's emotional state.

[0366] 8. A means of visually displaying and audibly issuing responses using character avatars

[0367] A character avatar within the smartphone application provides visual and audio information to the user.

[0368] Process Overview

[0369] The smartphone application records the user's voice and sends it to a server. The server converts the voice into text and passes it to a generative AI model. The generative AI model generates a response based on the text, and an emotion recognition engine analyzes the user's emotions. This information is combined to generate an appropriate response and send it to the user's device. A character avatar provides visual and audio responses to the user, improving the user experience.

[0370] Specific examples

[0371] For example, if a user asks "Which shoes are the most popular in this store?" in a store, the system works as follows: It records the user's voice input and sends it over the internet to a server. The server uses a speech recognition engine to convert the speech to text and passes that text as input to a generative AI model. The AI ​​model researches the "most popular shoes" and generates text to return the results to the user.

[0372] The generated text is something like, "The most popular shoe right now is the XYZ model. We're especially offering a 25% off campaign." If the emotion recognition engine detects that the user is excited, it adds an emphasis phrase like, "We highly recommend it!"

[0373] The character avatar then vocalizes the response and visually displays a corresponding image and description of the shoe. If the user asks a follow-up question, such as "Do you have any other colors?", the same technical steps are taken to generate the answer.

[0374] Prompt Sentence Examples

[0375] 1. Voice input: "What are the most popular shoes in this store?"

[0376] 2. Prompt the generative AI model: "Which shoes are the most popular in this store? User emotion: Excitement"

[0377] This provides real-time shopping support that takes into account the user's emotions.

[0378] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0379] Step 1:

[0380] The user launches the smartphone application and enters a question or request by voice.

[0381] Input: Speech data (e.g., "Which shoes are the most popular in this store?")

[0382] Output: Digital audio data

[0383] How it works: The smartphone's microphone records the user's voice, and the application converts the voice data into a digital format and stores it.

[0384] Step 2:

[0385] The digital audio data recorded by the device is sent to the server.

[0386] Input: Digital audio data

[0387] Output: Audio data sent to the server

[0388] Specific operation: The smartphone application sends voice data to a server via the Internet.

[0389] Step 3:

[0390] The voice data received by the server is analyzed using a voice recognition engine and converted into natural language text.

[0391] Input: Audio data

[0392] Output: Natural language text (e.g., "Which shoes are the most popular in this store?")

[0393] How it works: The server uses a speech recognition engine such as Google Cloud Speech-to-Text to analyze the voice data and convert it into natural language text.

[0394] Step 4:

[0395] The server passes the converted text to a generative AI model, which generates a corresponding response.

[0396] Input: Natural language text

[0397] Output: Response text (e.g., "The most popular shoe in our store right now is the XYZ model.")

[0398] Specific operation: The server passes text data as input to a generative AI model (e.g., OpenAI API) and generates a response appropriate to the user's question.

[0399] Step 5:

[0400] The server uses an emotion recognition engine to analyze emotions from the user's voice input and text data.

[0401] Input: Audio or text data

[0402] Output: Emotion data (e.g., "excited" or "frustrated")

[0403] Specific operation: The server uses an emotion recognition engine such as IBM Watson Tone Analyzer to recognize the user's emotions.

[0404] Step 6:

[0405] The server adjusts the generated responses based on the emotion data to generate a final response.

[0406] Input: Response text, emotion data

[0407] Output: Tailored response text (e.g., "The most popular shoe in our store right now is the XYZ model! Highly recommended!")

[0408] How it works: The generative AI model takes in emotion data and adds appropriate expressions based on the user's emotional state.

[0409] Step 7:

[0410] The server sends the adjusted response to the user terminal.

[0411] Input: Tailored response text

[0412] Output: Response text sent to the user's terminal

[0413] Specific operation: The server sends the generated response message to the user's smartphone via the Internet.

[0414] Step 8:

[0415] The user's smartphone displays visual and audio responses using a character avatar.

[0416] Input: Response text

[0417] Output: Audio response and visual display

[0418] Specific operation: A character avatar within the smartphone application will vocalize the generated response and display text information and images of related products on the screen.

[0419] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0420] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0421] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0422] [Second embodiment]

[0423] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0424] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0425] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0426] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0427] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0428] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0429] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0430] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0431] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0432] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0433] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0434] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0435] The present invention provides a character avatar that utilizes speech recognition and generation AI to function as a shopping assistant and guidance system. The system includes means for accepting speech input, means for analyzing the speech data, means for generating a response based on the analysis results, means for transmitting the generated response to a user terminal, and means for displaying and audibly transmitting the response on the user terminal.

[0436] System Configuration

[0437] 1. User voice input

[0438] Users launch a dedicated application and voice their question or request, for example, "Which shoes are the most popular in this store?"

[0439] 2. Sending audio data

[0440] The device records the user's voice and sends it as data to the server.

[0441] 3. Analysis of audio data

[0442] The server converts the received voice data into text data using a speech recognition engine. A generative AI model is used to understand the user's intent from this text data. As a result of the analysis, the generative AI generates a query to identify the "most popular shoes."

[0443] 4. Information Search

[0444] The server uses the generated query to search for relevant information from a product database, for example, "the most popular shoes" to identify "XYZ model."

[0445] 5. Generating the Response

[0446] The server generates a response message with the search results, for example, a message containing information such as "Currently, the most popular item is the 'XYZ model' shoe."

[0447] 6. Sending a response to the user

[0448] The server sends the generated response message to the terminal.

[0449] The device conveys the received message to the user audibly and visually through a character avatar, who says, "Here are some popular shoes," while displaying an image of the product.

[0450] 7. Gathering Feedback

[0451] If the user wants more information, they can say something like, "Do these shoes come in other colors?"

[0452] The terminal transmits this additional audio data back to the server.

[0453] The server analyzes the voice data again, generates a new response, and sends it to the terminal.

[0454] The device will then display the received response audibly and visually using a character avatar, for example, "These shoes are available in red, blue, and black."

[0455] Specific examples

[0456] For example, if a user asks "Which shoes are the most popular in this store?" while in a store, the system operates as follows: First, the device accepts the user's voice input and sends it to the server. The server converts the voice data into text and analyzes the content. It then searches the product database based on the analysis results and identifies the "XYZ model." The server generates a response message saying "The XYZ model is the most popular" and sends it to the device. The device then conveys this response message aloud through a character avatar and simultaneously displays an image. If the user then asks, "Are these shoes available in other colors?", the same process is used to provide color information.

[0457] In this way, the present invention utilizes voice recognition and generative AI to provide users with a real-time and efficient shopping assistant service, thereby improving customer satisfaction and reducing the burden on staff.

[0458] The processing flow will be explained below.

[0459] Step 1:

[0460] To begin interacting with the character avatar, users launch a dedicated application and speak their questions or requests into a microphone, for example, "Which shoes are the most popular in this store?"

[0461] Step 2:

[0462] The device records the user's voice input using a microphone and stores it as digital voice data.

[0463] Step 3:

[0464] The device converts the recorded audio data into an appropriate format (e.g., compresses and encodes it) and sends it to a server via the Internet.

[0465] Step 4:

[0466] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[0467] Step 5:

[0468] The server passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[0469] Step 6:

[0470] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[0471] Step 7:

[0472] The server receives the search results and generates a response message to provide to the user, for example, "The most popular shoe at the moment is the 'XYZ model'."

[0473] Step 8:

[0474] The server generates a response message and sends it to the terminal.

[0475] Step 9:

[0476] The device analyzes the received response message, and the character avatar communicates information to the user audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product.

[0477] Step 10:

[0478] If the user wants more information, they can ask a follow-up question, such as "Do these shoes come in other colors?"

[0479] Step 11:

[0480] The device re-records the additional audio data, converts it, and sends it to the server.

[0481] Step 12:

[0482] The server again analyzes the audio data and generates new text to generate appropriate responses to follow-up questions, such as "These shoes are available in red, blue, and black."

[0483] Step 13:

[0484] The server sends a new response message to the terminal.

[0485] Step 14:

[0486] The device then conveys a new response message to the user via a character avatar, who responds aloud, "These shoes are available in red, blue, and black," and displays the color variations on the screen.

[0487] Through the above series of steps, the system of the present invention responds to user requests in real time and provides necessary information, thereby increasing customer satisfaction.

[0488] Example 1

[0489] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0490] Conventional shopping assistant systems have had the problem of being unable to provide a quick and accurate response when users ask questions. In particular, when obtaining information using voice input, the accuracy of voice recognition and analysis is low, making it difficult to accurately understand the user's intent and provide an appropriate response. Furthermore, responses to follow-up questions from users can be slow, resulting in a decline in customer satisfaction.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0492] In this invention, the server includes a means for converting received voice data into text data, a means for analyzing the user's intent using a generative AI model, and a means for generating a query based on the analysis and searching for information from a database. This increases the accuracy of voice recognition, making it possible to accurately analyze the user's intent and quickly generate an appropriate response.

[0493] "Voice input" refers to the act of recording the user's voice through a microphone on a device.

[0494] "Digital audio data" refers to data obtained by converting an analog audio signal into a digital format.

[0495] The "server" is a central system responsible for analyzing voice data, searching the database, generating responses, etc.

[0496] "Text data" is character information converted from speech by a speech recognition engine.

[0497] A "generative AI model" is an algorithm that uses artificial intelligence to understand user intent and generate a response.

[0498] A "query" is a query command for retrieving specific information from a database.

[0499] A "database" is a collection of data that stores specific product information and other related information.

[0500] A "response message" is a server-generated text answer to a user's question.

[0501] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[0502] A "character avatar" is a virtual character that displays a response message audibly and visually.

[0503] This invention is a shopping assistant and guidance system that utilizes speech recognition and generation AI, allowing users to search for product information through voice input and receive responses in real time.

[0504] The system includes the following components:

[0505] A means of accepting voice input

[0506] A means to record received voice input and send it to a server as digital audio data

[0507] A means of converting received voice data into text data

[0508] A means of analyzing user intent using generative AI models

[0509] A means of generating queries based on analysis and retrieving information from a database

[0510] A means for generating a response message based on the searched information and transmitting it to the user terminal.

[0511] Means for displaying a response message audibly and visually at the user terminal

[0512] As a specific example, we use the Google Cloud Speech-to-Text API to convert voice data into text, and then use a generative AI model such as OpenAI's GPT-3 to analyze the captured text data and generate appropriate responses to user questions.

[0513] For example, if a user asks "Which shoes are the most popular in this store?" by voice, the system works as follows: The user launches a dedicated application and speaks the question. The device records this voice and sends it to the server. The server converts the voice data into text data and analyzes it using a generative AI model. The resulting query is run against the database, and the "XYZ model" is identified as the most popular shoe. The server generates a response message and sends it to the device. The device communicates the response to the user audibly and visually through a character avatar.

[0514] Examples of prompts include "What are the most popular shoes in this store?" or "Do you have these shoes in other colors?"

[0515] In this way, by combining speech recognition and generative AI, the present invention provides highly accurate information search and precise responses in a short time, aiming to improve customer satisfaction and reduce the burden on store staff.

[0516] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0517] Step 1:

[0518] The user starts a dedicated application and performs voice input.

[0519] Input: User-uttered speech (e.g., "Which shoes are the most popular in this store?")

[0520] What it does: Records audio through the microphone on your smartphone or tablet.

[0521] Output: Recorded audio data

[0522] Step 2:

[0523] The device digitizes the recorded audio data and sends it to the server.

[0524] Input: Recorded audio data

[0525] What it does: Converts voice data into digital form and sends it as data packets to a server.

[0526] Output: Digital audio data sent to the server

[0527] Step 3:

[0528] The server converts the received voice data into text data.

[0529] Input: Digital audio data sent to the server

[0530] How it works: Converts audio data into text using the Google Cloud Speech-to-Text API or similar.

[0531] Output: Text data (e.g., "Which shoes are the most popular in this store?")

[0532] Step 4:

[0533] The server inputs the text data into a generative AI model and analyzes the user's intent.

[0534] Input: Text data

[0535] How it works: Text data is fed into a generative AI model such as OpenAI's GPT-3, which analyzes the user's intent (the content of the request).

[0536] Output: Analysis results (e.g., a query to identify "most popular shoes")

[0537] Step 5:

[0538] The server retrieves information from a database based on the analysis results.

[0539] Input: The query generated as a result of the analysis

[0540] What it does: Queries the product database to find information about the corresponding product.

[0541] Output: Search results (e.g., "XYZ model" shoes are the most popular)

[0542] Step 6:

[0543] The server generates a response message based on the search results and transmits it to the user terminal.

[0544] Input: Search results

[0545] What it does: Based on the search results, it generates a text response message to the user's question.

[0546] Output: Response message (e.g., "Currently, the most popular shoe is the XYZ model.")

[0547] Step 7:

[0548] The terminal transmits the response message received from the server to the user through the character avatar.

[0549] Input: Response message sent by the server

[0550] What it does: It analyzes the response message and communicates it to the user audibly and visually through a character avatar. For example, the avatar might say, "Here are some popular shoes," while displaying a product image.

[0551] Output: The response displayed and spoken to the user

[0552] Step 8:

[0553] If the user desires more information, they can ask additional questions by voice and start the process again.

[0554] Input: An additional spoken question from the user (e.g., "Do these shoes come in other colors?")

[0555] How it works: The device records audio and sends the digital audio data to the server, which then converts the audio data back into text, analyzes it with a generative AI model, generates a response message, and sends it to the device.

[0556] Output: An additional response that is displayed and spoken to the user (e.g., "These shoes come in red, blue, and black")

[0557] This allows users to efficiently obtain necessary product information in real time.

[0558] (Application example 1)

[0559] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0560] Conventional shopping support systems for brick-and-mortar stores have the problem of not being able to provide efficient support because the product information search and in-store guidance that users face are not performed in real time.In addition, there is a lack of interaction through the user's voice, and there are also issues with familiarity and intuitive operability.

[0561] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0562] In this invention, the server includes means for accepting voice input, means for analyzing the accepted voice input, means for generating a response based on the analysis result, means for transmitting the generated response to a user terminal, means for displaying and audibly transmitting the response on the user terminal, means for generating a response based on a specific question using a generative AI model, means for converting the generated response from text to voice, and means for converting voice data to text data using a voice recognition engine. This allows users to easily obtain product information and store guides by voice, enabling a familiar and intuitive shopping experience.

[0563] The "means for accepting voice input" refers to a microphone or peripheral device for capturing the user's voice, and is a mechanism for acquiring voice signals as digital data.

[0564] "Means for analyzing received voice input" refers to software or algorithms that convert voice data into text data and analyze the content to understand the user's intent or question.

[0565] "Means for generating a response based on the analysis results" refers to the process of using algorithms or software, particularly generative AI models, to generate appropriate responses based on the results of voice analysis.

[0566] "Means for transmitting the generated response to the user terminal" refers to a communication means or protocol for transferring the generated response data from the server to the user's device.

[0567] "Means for displaying and audibly transmitting a response on a user terminal" refers to a mechanism with a display and speaker for visually displaying and audibly playing received response data on a device.

[0568] A "means for generating a response based on a specific question using a generative AI model" is a computational model or algorithm for using generative AI technology to generate a meaningful response based on a user's question.

[0569] "Means for converting the generated response from text to speech" refers to speech synthesis technology or software for converting a text response into speech.

[0570] "Means for converting voice data into text data using a voice recognition engine" refers to a voice recognition technology that analyzes voice input and converts the voice into corresponding text data.

[0571] To implement this invention, a system that combines speech recognition, generative AI, speech synthesis, and communication technologies is required. This system is designed to support shopping in stores and uses smartphones as the main user terminal. The specific operation of the system and its components are described below.

[0572] System configuration

[0573] Accepting voice input

[0574] The user launches a dedicated application on their smartphone and inputs a question or request by voice, such as, "What is the most popular camera in this store?" The smartphone's microphone picks up this voice and captures it as digital data.

[0575] Analysis of voice data and conversion to text

[0576] The smartphone sends the captured voice data to a server installed in the cloud. The server converts the voice data into text data using a speech recognition engine (e.g., Google Speech Recognition API). The converted text data is passed to a generative AI model (e.g., GPT-3) to understand the user's intent.

[0577] Generating a response and retrieving product information

[0578] The server's generative AI model generates an appropriate response based on the analyzed text data. Specifically, it understands the user's question and generates the most appropriate answer. For example, a response such as "The most popular camera at present is model ABC" is generated. In this process, a query is sent to a product information database to obtain the latest sales information and popularity data.

[0579] Text-to-speech conversion of responses

[0580] The generated text response is converted to audio using a speech synthesis engine (e.g., Google Text-to-Speech API), allowing the user to hear the answer to their question aloud.

[0581] Sending and displaying a response to the user terminal

[0582] The server then sends the converted response data to the user's smartphone. The dedicated application then displays the response visually and audibly using a character avatar. For example, the character avatar may say, "Here's a popular camera," while also displaying an image of the product.

[0583] Examples of concrete examples and prompts

[0584] As a concrete example, consider a situation where a user is in a store and asks, "What is the most popular camera in this store?" The system accepts voice input, performs analysis and response generation on the server, and finally responds with audio and visual responses such as, "The most popular camera right now is model ABC." An example of a prompt sentence is as follows:

[0585] "What is the most popular camera in this store?"

[0586] This allows users to easily obtain product information and store guides via voice, providing a friendly and intuitive shopping experience.

[0587] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0588] Step 1:

[0589] A user launches a dedicated application installed on their smartphone and speaks a question or request, for example, "What is the most popular camera in this store?" The smartphone's microphone captures this voice and acquires it as digital data, which is the input and becomes the voice data for subsequent processing.

[0590] Step 2:

[0591] The device sends the captured audio data to the server. This is the audio data transmission process. The input from the device is the user's audio data, and the output is to send this data to the server. Specifically, data is sent to the server using an HTTP request or WebSocket.

[0592] Step 3:

[0593] The server converts the received voice data into text data using a speech recognition engine (e.g., Google Speech Recognition API). The input is the user's voice data, and the output is text data. This conversion is performed by the speech recognition engine. Specifically, it includes extracting features from the voice signal and analyzing phonemes.

[0594] Step 4:

[0595] The server uses a generative AI model (e.g., GPT-3) to generate a response based on the converted text data. The input is text data, and the output is the generated response text. This is where the generative AI model analyzes the meaning of the text and generates the optimal response. For example, a response such as "Currently, the most popular camera is the ABC model" is generated.

[0596] Step 5:

[0597] The server queries a product information database to find additional information to complement the generated response. The input is a text-based query, and the output is the corresponding product information. This step uses a database management system (e.g., MySQL) to look up the product information and parse the information.

[0598] Step 6:

[0599] The server converts the generated text response into audio using a speech synthesis engine (e.g., Google Text-to-Speech API). The input is the generated response text, and the output is the audio data. This conversion involves mapping text to audio and generating an audio file.

[0600] Step 7:

[0601] The server sends the generated voice data to the user's smartphone. The input is voice data, and the output is data transmission to the smartphone. This data transfer is performed using communication protocols such as HTTP requests and WebSocket.

[0602] Step 8:

[0603] The device uses a character avatar to display the received audio data visually and audibly. The input is the audio data sent from the server, and the output is audio playback and information display for the user. Specifically, the audio is played back through the smartphone's speaker, related information is displayed on the display, and the character avatar says things like, "Popular cameras are here."

[0604] The above processing flow allows users to easily obtain product information and store guides by voice, providing a familiar and intuitive shopping experience.

[0605] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0606] The present invention provides a character avatar that utilizes speech recognition, generative AI, and an emotion engine to function as a shopping assistant and guidance system. This system includes means for accepting a user's voice input, analyzing the voice input, generating a response based on the analysis results, transmitting the generated response to a user terminal, and displaying and vocalizing the response on the user terminal. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the user experience is further improved.

[0607] System Configuration

[0608] 1. User voice input

[0609] Users launch a dedicated application and voice their question or request, for example, "Which shoes are the most popular in this store?"

[0610] 2. Sending audio data

[0611] The device records the user's voice input with a microphone and stores it as digital audio data, which is then converted into a suitable format and sent to a server over the Internet.

[0612] 3. Analysis of audio data

[0613] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[0614] 4. Response generation based on analysis results

[0615] The server then passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[0616] 5. Searching for Information

[0617] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[0618] 6. Emotion Recognition with Emotion Engine

[0619] The server uses an emotion engine to recognize the user's emotional state from voice input and the user's past interactions, for example, determining whether the user is frustrated or happy.

[0620] 7. Regulating responses based on emotions

[0621] The server adjusts the response message based on the results of the emotion engine. For example, if the user is frustrated, it generates a more polite response or one that provides additional information.

[0622] 8. Sending the Response

[0623] The server generates a response message and sends it to the terminal. For example, the response message might say, "The most popular shoe at the moment is the 'XYZ model'. It comes in red, blue, and black."

[0624] 9. Display and Call to User

[0625] The device analyzes the received response message, and a character avatar communicates information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product and adding a polite comment based on the user's emotions.

[0626] 10. Receiving User Feedback

[0627] If the user wants more information, they can ask a follow-up question, such as "Do you have other colors?"

[0628] 11. Processing of follow-up questions

[0629] The device again records additional audio data and sends it to the server, which again analyzes the audio data and generates a new response, such as a message containing the information, "These shoes are available in red, blue, and black."

[0630] 12. Sending and Displaying Final Responses

[0631] The server sends a new response message to the terminal, which again provides the information to the user audibly and visually via the character avatar.

[0632] Specific examples

[0633] For example, if a user asks, "Which shoes are the most popular in this store?" while in a store, the system operates as follows: First, the user's voice input is recorded and sent to the server. The server converts the voice data into text and analyzes the content. The system then searches the product database based on the analysis results, identifies an "XYZ model," and recognizes the user's emotions using an emotion engine. For example, if the system recognizes that the user is excited, it generates a response containing a competitive expression such as "These shoes are especially recommended." The server then sends the generated response to the device, which uses a character avatar to convey the information both audibly and visually. If the user subsequently expresses interest and asks the follow-up question, "Do you have any other colors?", the system also provides color information through a similar process.

[0634] In this way, the system of the present invention combines speech recognition, generative AI, and an emotion engine to provide efficient real-time shopping support that adapts to the user's emotional state, thereby improving customer satisfaction and reducing the burden on staff.

[0635] The processing flow will be explained below.

[0636] Step 1:

[0637] The user launches the application and speaks a question or request into the microphone, for example, "Which shoes are the most popular in this store?"

[0638] Step 2:

[0639] The device records the user's voice input using a microphone and stores it as digital voice data.

[0640] Step 3:

[0641] The device converts the recorded audio data into an appropriate format (e.g., compresses and encodes it) and sends it to a server via the Internet.

[0642] Step 4:

[0643] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[0644] Step 5:

[0645] The server passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[0646] Step 6:

[0647] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[0648] Step 7:

[0649] The server receives the search results and passes the user's voice input data to the emotion engine, which analyzes the user's emotional state. For example, the emotion engine recognizes the user's voice data as "excited."

[0650] Step 8:

[0651] The server generates a response message based on the emotion engine results, depending on the user's emotions. For example, if the user is excited, the server creates a response such as "These shoes are especially recommended," including additional reasons for the recommendation.

[0652] Step 9:

[0653] The server sends the generated response message and appropriate product information to the terminal.

[0654] Step 10:

[0655] The device analyzes the response message and the character avatar communicates information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product with additional comments based on the user's emotions.

[0656] Step 11:

[0657] If the user wants more information, they can ask a follow-up question, such as "Do these shoes come in other colors?"

[0658] Step 12:

[0659] The device re-records the additional audio data, converts it, and sends it to the server.

[0660] Step 13:

[0661] The server again analyzes the voice data, generates new text, retrieves product color variation information from the database, and generates a new response message.

[0662] Step 14:

[0663] The server uses the emotion engine again to analyze the user's emotional state when asking the follow-up question and tailor an appropriate response message based on that emotional state. For example, if the user asks a question in a dissatisfied tone, the server generates a polite response such as, "We have a wide variety of colors, so please try them out."

[0664] Step 15:

[0665] The server sends a new response message to the terminal.

[0666] Step 16:

[0667] The device then re-prompts the user with a new response message, both audibly and visually, via a character avatar. The avatar responds audibly, saying, "These shoes are available in red, blue, and black," and displays the color variations on the screen. At the same time, it adds a comment that takes the user's feelings into consideration.

[0668] Through the above series of processing steps, the system of the present invention provides information in real time while adapting to the user's emotional state, realizing an efficient and fulfilling customer experience.

[0669] Example 2

[0670] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0671] In today's shopping experience, users want fast and accurate information, but traditional systems face the challenge of being unable to accurately grasp the user's intentions and emotions and provide appropriate responses. Furthermore, impersonal responses often fail to improve the user experience and leave the user feeling unsatisfied. This can lead to lower customer satisfaction and a loss of purchasing motivation.

[0672] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0673] In this invention, the server includes a means for passing voice data to a voice recognition engine and converting it into natural language text, a means for passing the converted text data to a generative AI model and analyzing the content of the text, and a means for recognizing the user's emotions and adjusting the response message based on the results. This makes it possible to accurately grasp the user's intentions and emotions and quickly generate an appropriate response.

[0674] (Definitions of important terms included in the claims)

[0675] "Voice input" is a means of recording the user's spoken words or voice as digital data.

[0676] "Digital audio data" refers to data obtained by converting an analog audio signal into a digital format.

[0677] "Recording" means the act of recording and saving the user's voice.

[0678] A "server" is a central processing unit that analyzes voice data and transmits the processing results to a user terminal.

[0679] A "speech recognition engine" is software or hardware that converts speech data into natural language text.

[0680] "Natural language text" is text data in a language that humans can understand that has been converted from voice data.

[0681] A "generative AI model" is an artificial intelligence algorithm designed to generate appropriate responses to specific inputs.

[0682] "Text content analysis" is the process of understanding the converted natural language text and identifying the user's intent.

[0683] A "response message" is text that contains a response or information to a user's inquiry.

[0684] "Emotion recognition" refers to determining a user's emotional state from the tone and content of their voice.

[0685] A "user terminal" is a digital device used by a user, including devices such as smartphones and tablets.

[0686] "Display and voice transmission" refers to displaying text information on the screen of the user terminal and transmitting it to the user by voice.

[0687] This system is a shopping assistant and guidance system that analyzes user voice input and generates responses. The system uses hardware and software that combines voice recognition, generative AI, secure communication methods, and an emotion recognition engine.

[0688] First, the user installs a dedicated application and uses a user device such as a smartphone or tablet to input voice commands. The user taps the microphone button to begin asking a question. For example, they can input a question by voice such as, "Which shoes are the most popular in this store?"

[0689] Next, the device will record the user's voice using a built-in or external microphone as digital audio data, store it in temporary memory, convert it into a predefined data format, and send it to the server using a secure communication protocol such as SSL / TLS.

[0690] The server first converts the voice data into natural language text using a speech recognition engine (e.g., Google Speech-to-Text API). For example, the voice data "Which shoes are the most popular in this store?" is converted into text "Which shoes are the most popular in this store?"

[0691] The converted text data is passed to a generative AI model (e.g., OpenAI GPT-4) for analysis. This analysis process understands the content of the input text to identify the user's intent. For example, the generative AI model generates an appropriate response in the form of a prompt: "The user is asking about 'the most popular shoes.' Please provide more information."

[0692] The server then searches a product database (e.g., MySQL or PostgreSQL) to retrieve relevant product information to generate a specific response. For example, it might find that "XYZ model shoes" are the most popular. During this process, the generative AI model uses the search results to create a specific response message.

[0693] To understand the user's emotional state, the server uses an emotion recognition engine (e.g., Microsoft Azure's Emotion API) to recognize emotions from voice input and past interaction data. For example, if the user is excited, a response such as "I highly recommend these shoes!" is generated.

[0694] The server then sends the generated response message to the user's device. The device receives the message and uses a character avatar to communicate information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product.

[0695] If the user wants more information, they can speak again, asking, "Do you have any other colors?" The device again records the voice data and sends it to the server. A response is generated and sent through the same process, providing the user with the latest information.

[0696] As described above, the present invention improves the user experience by accurately understanding the user's intentions and emotions and providing an optimal response.

[0697] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0698] Step 1:

[0699] The user launches the dedicated application and taps the microphone button to start voice input, for example, by speaking a question such as, "Which shoes are the most popular in this store?"

[0700] Step 2:

[0701] The device uses the smartphone's built-in microphone to record the user's voice as digital audio data. The recorded audio data is stored in short-term memory. The input is the user's voice, and the output is digital audio data.

[0702] Step 3:

[0703] The terminal transmits digital audio data to the server using a secure communication protocol such as SSL / TLS. The input is digital audio data, and the output is secure data transmission to the server.

[0704] Step 4:

[0705] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts it into natural language text. The input is digital voice data, and the output is natural language text.

[0706] Step 5:

[0707] The server passes the converted text data to a generative AI model (e.g., OpenAI GPT-4) and begins analyzing the text content. A prompt is created and the analysis proceeds. The input is natural language text, and the output is a prompt containing the analysis results.

[0708] Example: Prompt: "User is asking about 'Most Popular Shoes'. Please provide more information."

[0709] Step 6:

[0710] The server sends a query to a product database (e.g., MySQL or PostgreSQL) based on the analysis results of the generative AI model to search for related product information. The input is the analysis results, and the output is the query results.

[0711] Example: Retrieve information from a database that "XYZ model shoes" are the most popular.

[0712] Step 7:

[0713] The server uses an emotion recognition engine (e.g., Microsoft Azure's Emotion API) to recognize the user's emotions from voice input and past interaction data. The input is voice or text data, and the output is the user's emotional state.

[0714] For example, if the user is excited, a response like "I highly recommend these shoes!" will be generated.

[0715] Step 8:

[0716] The server uses a generative AI model to tailor a response message based on the recognized emotion. The generative AI model is used to create a specific response that reflects the emotional information. The input is the emotion recognition result, and the output is the tailored response message.

[0717] Step 9:

[0718] The server sends the generated response message to the user terminal. The input is the adjusted response message, and the output is the data transmission to the user terminal.

[0719] Step 10:

[0720] The terminal displays the received response message, and the character avatar provides information to the user audio and visually. The input is the response message, and the output is audio and visual display.

[0721] Example: An avatar says, "Here are some popular shoes," and displays an image of the product.

[0722] Step 11:

[0723] If the user wants more information, they again use voice input, in this case asking a question such as "Do you have any other colors?" The input is an additional voice question.

[0724] Step 12:

[0725] The terminal again records the additional audio data and sends it to the server. The input is the additional audio data, and the output is transmission to the server.

[0726] Step 13:

[0727] The server analyzes the new voice data and generates a new response. The same process is repeated to provide the information the user seeks. The input is a further voice question, and the output is a new response message.

[0728] For example: "These shoes come in red, blue, and black."

[0729] This enables the system to accurately understand the user's intentions and emotions and provide the most appropriate response.

[0730] (Application example 2)

[0731] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0732] Conventional shopping assistant systems only generate responses based on the user's voice input and do not consider the user's emotions, which prevents them from fully improving user satisfaction. Furthermore, they lack a means to provide information in an easy-to-understand visual and audio format, making it difficult to provide an intuitive and enjoyable user experience. Furthermore, there are limitations to the means to efficiently generate responses each time the user asks additional questions.

[0733] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0734] In this invention, the server includes means for accepting a voice input, means for analyzing the accepted voice input, means for generating a response based on the analysis result, means for transmitting the generated response to a user terminal, means for displaying the response and transmitting it audibly on the user terminal, means for recognizing emotions, means for adjusting the response based on the result of the emotion recognition means, and means for visually displaying the response and transmitting it audibly using a character avatar, thereby enabling response generation that takes the user's emotions into consideration and providing visual and audio information using a character avatar.

[0735] "Means for accepting voice input" refers to a device or function for recognizing voice input from a user and recording it as digital data.

[0736] "Means for analyzing received voice input" refers to technology for processing recorded voice data and converting it into natural language text.

[0737] The "means for generating a response based on the analysis results" refers to a generative AI model or algorithm that converts voice data into text and then creates an appropriate response based on that content.

[0738] The "means for transmitting the generated response to the user terminal" refers to a technique for transferring the generated response message from the server or central system to the user's device.

[0739] "Means for displaying and audibly transmitting a response on a user terminal" refers to equipment or software for visually displaying information and audibly transmitting a response on a user's device.

[0740] "Emotion recognition means" is a technology that analyzes emotions from a user's voice or text and identifies states such as joy, anger, sadness, and happiness.

[0741] The "means for adjusting a response based on the results of the emotion recognition means" is software or an algorithm for utilizing the results of the emotion recognition means to generate a response that is adapted to the emotional state of the user.

[0742] A "means for visually displaying and audibly transmitting responses using character avatars" is a device or software that uses animation or graphic characters to display and communicate information audibly.

[0743] To implement the present invention, the following system configuration and procedure are used.

[0744] System Configuration

[0745] 1. A means of accepting voice input

[0746] The user inputs voice using a smartphone application, and the smartphone's microphone records the voice and stores it as digital data.

[0747] 2. A means of analyzing the received voice input

[0748] This digital voice data is sent via a smartphone application over the internet to a server, which then uses a speech recognition engine such as Google Cloud Speech-to-Text to analyze the voice data and convert it into natural language text.

[0749] 3. A means of generating a response based on the analysis results

[0750] The server uses the parsed text data as input for a generative AI model (e.g., OpenAI API) that generates an appropriate response based on the user's question or request.

[0751] 4. A means of sending the generated response to the user terminal

[0752] The generated response is forwarded from the server to the user's smartphone.

[0753] 5. Means for displaying and audibly transmitting responses on the user terminal

[0754] The user's smartphone displays the received response in visual and audio formats, with a character avatar responding with audio and displaying text information on the screen for easy visual understanding by the user.

[0755] 6. Emotion recognition means

[0756] The server uses emotion recognition engines such as IBM Watson Tone Analyzer to determine the user's emotions based on the user's voice and text data.

[0757] 7. A means of adjusting responses based on the results of emotion recognition measures

[0758] Based on the results of emotion recognition, a generative AI model generates a tailored response that is adapted to the user's emotional state.

[0759] 8. A means of visually displaying and audibly issuing responses using character avatars

[0760] A character avatar within the smartphone application provides visual and audio information to the user.

[0761] Process Overview

[0762] The smartphone application records the user's voice and sends it to a server. The server converts the voice into text and passes it to a generative AI model. The generative AI model generates a response based on the text, and an emotion recognition engine analyzes the user's emotions. This information is combined to generate an appropriate response and send it to the user's device. A character avatar provides visual and audio responses to the user, improving the user experience.

[0763] Specific examples

[0764] For example, if a user asks "Which shoes are the most popular in this store?" in a store, the system works as follows: It records the user's voice input and sends it over the internet to a server. The server uses a speech recognition engine to convert the speech to text and passes that text as input to a generative AI model. The AI ​​model researches the "most popular shoes" and generates text to return the results to the user.

[0765] The generated text is something like, "The most popular shoe right now is the XYZ model. We're especially offering a 25% off campaign." If the emotion recognition engine detects that the user is excited, it adds an emphasis phrase like, "We highly recommend it!"

[0766] The character avatar then vocalizes the response and visually displays a corresponding image and description of the shoe. If the user asks a follow-up question, such as "Do you have any other colors?", the same technical steps are taken to generate the answer.

[0767] Prompt Sentence Examples

[0768] 1. Voice input: "What are the most popular shoes in this store?"

[0769] 2. Prompt the generative AI model: "Which shoes are the most popular in this store? User emotion: Excitement"

[0770] This provides real-time shopping support that takes into account the user's emotions.

[0771] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0772] Step 1:

[0773] The user launches the smartphone application and enters a question or request by voice.

[0774] Input: Speech data (e.g., "Which shoes are the most popular in this store?")

[0775] Output: Digital audio data

[0776] How it works: The smartphone's microphone records the user's voice, and the application converts the voice data into a digital format and stores it.

[0777] Step 2:

[0778] The digital audio data recorded by the device is sent to the server.

[0779] Input: Digital audio data

[0780] Output: Audio data sent to the server

[0781] Specific operation: The smartphone application sends voice data to a server via the Internet.

[0782] Step 3:

[0783] The voice data received by the server is analyzed using a voice recognition engine and converted into natural language text.

[0784] Input: Audio data

[0785] Output: Natural language text (e.g., "Which shoes are the most popular in this store?")

[0786] How it works: The server uses a speech recognition engine such as Google Cloud Speech-to-Text to analyze the voice data and convert it into natural language text.

[0787] Step 4:

[0788] The server passes the converted text to a generative AI model, which generates a corresponding response.

[0789] Input: Natural language text

[0790] Output: Response text (e.g., "The most popular shoe in our store right now is the XYZ model.")

[0791] Specific operation: The server passes text data as input to a generative AI model (e.g., OpenAI API) and generates a response appropriate to the user's question.

[0792] Step 5:

[0793] The server uses an emotion recognition engine to analyze emotions from the user's voice input and text data.

[0794] Input: Audio or text data

[0795] Output: Emotion data (e.g., "excited" or "frustrated")

[0796] Specific operation: The server uses an emotion recognition engine such as IBM Watson Tone Analyzer to recognize the user's emotions.

[0797] Step 6:

[0798] The server adjusts the generated responses based on the emotion data to generate a final response.

[0799] Input: Response text, emotion data

[0800] Output: Tailored response text (e.g., "The most popular shoe in our store right now is the XYZ model! Highly recommended!")

[0801] How it works: The generative AI model takes in emotion data and adds appropriate expressions based on the user's emotional state.

[0802] Step 7:

[0803] The server sends the adjusted response to the user terminal.

[0804] Input: Tailored response text

[0805] Output: Response text sent to the user's terminal

[0806] Specific operation: The server sends the generated response message to the user's smartphone via the Internet.

[0807] Step 8:

[0808] The user's smartphone displays visual and audio responses using a character avatar.

[0809] Input: Response text

[0810] Output: Audio response and visual display

[0811] Specific operation: A character avatar within the smartphone application will vocalize the generated response and display text information and images of related products on the screen.

[0812] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0813] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0814] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0815] [Third embodiment]

[0816] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0817] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0818] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0819] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0820] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0821] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0822] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0823] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0824] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0825] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0826] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0827] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0828] The present invention provides a character avatar that utilizes speech recognition and generation AI to function as a shopping assistant and guidance system. The system includes means for accepting speech input, means for analyzing the speech data, means for generating a response based on the analysis results, means for transmitting the generated response to a user terminal, and means for displaying and audibly transmitting the response on the user terminal.

[0829] System Configuration

[0830] 1. User voice input

[0831] Users launch a dedicated application and voice their question or request, for example, "Which shoes are the most popular in this store?"

[0832] 2. Sending audio data

[0833] The device records the user's voice and sends it as data to the server.

[0834] 3. Analysis of audio data

[0835] The server converts the received voice data into text data using a speech recognition engine. A generative AI model is used to understand the user's intent from this text data. As a result of the analysis, the generative AI generates a query to identify the "most popular shoes."

[0836] 4. Information Search

[0837] The server uses the generated query to search for relevant information from a product database, for example, "the most popular shoes" to identify "XYZ model."

[0838] 5. Generating the Response

[0839] The server generates a response message with the search results, for example, a message containing information such as "Currently, the most popular item is the 'XYZ model' shoe."

[0840] 6. Sending a response to the user

[0841] The server sends the generated response message to the terminal.

[0842] The device conveys the received message to the user audibly and visually through a character avatar, who says, "Here are some popular shoes," while displaying an image of the product.

[0843] 7. Gathering Feedback

[0844] If the user wants more information, they can say something like, "Do these shoes come in other colors?"

[0845] The terminal transmits this additional audio data back to the server.

[0846] The server analyzes the voice data again, generates a new response, and sends it to the terminal.

[0847] The device will then display the received response audibly and visually using a character avatar, for example, "These shoes are available in red, blue, and black."

[0848] Specific examples

[0849] For example, if a user asks "Which shoes are the most popular in this store?" while in a store, the system operates as follows: First, the device accepts the user's voice input and sends it to the server. The server converts the voice data into text and analyzes the content. It then searches the product database based on the analysis results and identifies the "XYZ model." The server generates a response message saying "The XYZ model is the most popular" and sends it to the device. The device then conveys this response message aloud through a character avatar and simultaneously displays an image. If the user then asks, "Are these shoes available in other colors?", the same process is used to provide color information.

[0850] In this way, the present invention utilizes voice recognition and generative AI to provide users with a real-time and efficient shopping assistant service, thereby improving customer satisfaction and reducing the burden on staff.

[0851] The processing flow will be explained below.

[0852] Step 1:

[0853] To begin interacting with the character avatar, users launch a dedicated application and speak their questions or requests into a microphone, for example, "Which shoes are the most popular in this store?"

[0854] Step 2:

[0855] The device records the user's voice input using a microphone and stores it as digital voice data.

[0856] Step 3:

[0857] The device converts the recorded audio data into an appropriate format (e.g., compresses and encodes it) and sends it to a server via the Internet.

[0858] Step 4:

[0859] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[0860] Step 5:

[0861] The server passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[0862] Step 6:

[0863] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[0864] Step 7:

[0865] The server receives the search results and generates a response message to provide to the user, for example, "The most popular shoe at the moment is the 'XYZ model'."

[0866] Step 8:

[0867] The server generates a response message and sends it to the terminal.

[0868] Step 9:

[0869] The device analyzes the received response message, and the character avatar communicates information to the user audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product.

[0870] Step 10:

[0871] If the user wants more information, they can ask a follow-up question, such as "Do these shoes come in other colors?"

[0872] Step 11:

[0873] The device re-records the additional audio data, converts it, and sends it to the server.

[0874] Step 12:

[0875] The server again analyzes the audio data and generates new text to generate appropriate responses to follow-up questions, such as "These shoes are available in red, blue, and black."

[0876] Step 13:

[0877] The server sends a new response message to the terminal.

[0878] Step 14:

[0879] The device then conveys a new response message to the user via a character avatar, who responds aloud, "These shoes are available in red, blue, and black," and displays the color variations on the screen.

[0880] Through the above series of steps, the system of the present invention responds to user requests in real time and provides necessary information, thereby increasing customer satisfaction.

[0881] Example 1

[0882] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0883] Conventional shopping assistant systems have had the problem of being unable to provide a quick and accurate response when users ask questions. In particular, when obtaining information using voice input, the accuracy of voice recognition and analysis is low, making it difficult to accurately understand the user's intent and provide an appropriate response. Furthermore, responses to follow-up questions from users can be slow, resulting in a decline in customer satisfaction.

[0884] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0885] In this invention, the server includes a means for converting received voice data into text data, a means for analyzing the user's intent using a generative AI model, and a means for generating a query based on the analysis and searching for information from a database. This increases the accuracy of voice recognition, making it possible to accurately analyze the user's intent and quickly generate an appropriate response.

[0886] "Voice input" refers to the act of recording the user's voice through a microphone on a device.

[0887] "Digital audio data" refers to data obtained by converting an analog audio signal into a digital format.

[0888] The "server" is a central system responsible for analyzing voice data, searching the database, generating responses, etc.

[0889] "Text data" is character information converted from speech by a speech recognition engine.

[0890] A "generative AI model" is an algorithm that uses artificial intelligence to understand user intent and generate a response.

[0891] A "query" is a query command for retrieving specific information from a database.

[0892] A "database" is a collection of data that stores specific product information and other related information.

[0893] A "response message" is a server-generated text answer to a user's question.

[0894] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[0895] A "character avatar" is a virtual character that displays a response message audibly and visually.

[0896] This invention is a shopping assistant and guidance system that utilizes speech recognition and generation AI, allowing users to search for product information through voice input and receive responses in real time.

[0897] The system includes the following components:

[0898] A means of accepting voice input

[0899] A means to record received voice input and send it to a server as digital audio data

[0900] A means of converting received voice data into text data

[0901] A means of analyzing user intent using generative AI models

[0902] A means of generating queries based on analysis and retrieving information from a database

[0903] A means for generating a response message based on the searched information and transmitting it to the user terminal.

[0904] Means for displaying a response message audibly and visually at the user terminal

[0905] As a specific example, we use the Google Cloud Speech-to-Text API to convert voice data into text, and then use a generative AI model such as OpenAI's GPT-3 to analyze the captured text data and generate appropriate responses to user questions.

[0906] For example, if a user asks "Which shoes are the most popular in this store?" by voice, the system works as follows: The user launches a dedicated application and speaks the question. The device records this voice and sends it to the server. The server converts the voice data into text data and analyzes it using a generative AI model. The resulting query is run against the database, and the "XYZ model" is identified as the most popular shoe. The server generates a response message and sends it to the device. The device communicates the response to the user audibly and visually through a character avatar.

[0907] Examples of prompts include "What are the most popular shoes in this store?" or "Do you have these shoes in other colors?"

[0908] In this way, by combining speech recognition and generative AI, the present invention provides highly accurate information search and precise responses in a short time, aiming to improve customer satisfaction and reduce the burden on store staff.

[0909] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0910] Step 1:

[0911] The user starts a dedicated application and performs voice input.

[0912] Input: User-uttered speech (e.g., "Which shoes are the most popular in this store?")

[0913] What it does: Records audio through the microphone on your smartphone or tablet.

[0914] Output: Recorded audio data

[0915] Step 2:

[0916] The device digitizes the recorded audio data and sends it to the server.

[0917] Input: Recorded audio data

[0918] What it does: Converts voice data into digital form and sends it as data packets to a server.

[0919] Output: Digital audio data sent to the server

[0920] Step 3:

[0921] The server converts the received voice data into text data.

[0922] Input: Digital audio data sent to the server

[0923] How it works: Converts audio data into text using the Google Cloud Speech-to-Text API or similar.

[0924] Output: Text data (e.g., "Which shoes are the most popular in this store?")

[0925] Step 4:

[0926] The server inputs the text data into a generative AI model and analyzes the user's intent.

[0927] Input: Text data

[0928] How it works: Text data is fed into a generative AI model such as OpenAI's GPT-3, which analyzes the user's intent (the content of the request).

[0929] Output: Analysis results (e.g., a query to identify "most popular shoes")

[0930] Step 5:

[0931] The server retrieves information from a database based on the analysis results.

[0932] Input: The query generated as a result of the analysis

[0933] What it does: Queries the product database to find information about the corresponding product.

[0934] Output: Search results (e.g., "XYZ model" shoes are the most popular)

[0935] Step 6:

[0936] The server generates a response message based on the search results and transmits it to the user terminal.

[0937] Input: Search results

[0938] What it does: Based on the search results, it generates a text response message to the user's question.

[0939] Output: Response message (e.g., "Currently, the most popular shoe is the XYZ model.")

[0940] Step 7:

[0941] The terminal transmits the response message received from the server to the user through the character avatar.

[0942] Input: Response message sent by the server

[0943] What it does: It analyzes the response message and communicates it to the user audibly and visually through a character avatar. For example, the avatar might say, "Here are some popular shoes," while displaying a product image.

[0944] Output: The response displayed and spoken to the user

[0945] Step 8:

[0946] If the user desires more information, they can ask additional questions by voice and start the process again.

[0947] Input: An additional spoken question from the user (e.g., "Do these shoes come in other colors?")

[0948] How it works: The device records audio and sends the digital audio data to the server, which then converts the audio data back into text, analyzes it with a generative AI model, generates a response message, and sends it to the device.

[0949] Output: An additional response that is displayed and spoken to the user (e.g., "These shoes come in red, blue, and black")

[0950] This allows users to efficiently obtain necessary product information in real time.

[0951] (Application example 1)

[0952] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0953] Conventional shopping support systems for brick-and-mortar stores have the problem of not being able to provide efficient support because the product information search and in-store guidance that users face are not performed in real time.In addition, there is a lack of interaction through the user's voice, and there are also issues with familiarity and intuitive operability.

[0954] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0955] In this invention, the server includes means for accepting voice input, means for analyzing the accepted voice input, means for generating a response based on the analysis result, means for transmitting the generated response to a user terminal, means for displaying and audibly transmitting the response on the user terminal, means for generating a response based on a specific question using a generative AI model, means for converting the generated response from text to voice, and means for converting voice data to text data using a voice recognition engine. This allows users to easily obtain product information and store guides by voice, enabling a familiar and intuitive shopping experience.

[0956] The "means for accepting voice input" refers to a microphone or peripheral device for capturing the user's voice, and is a mechanism for acquiring voice signals as digital data.

[0957] "Means for analyzing received voice input" refers to software or algorithms that convert voice data into text data and analyze the content to understand the user's intent or question.

[0958] "Means for generating a response based on the analysis results" refers to the process of using algorithms or software, particularly generative AI models, to generate appropriate responses based on the results of voice analysis.

[0959] "Means for transmitting the generated response to the user terminal" refers to a communication means or protocol for transferring the generated response data from the server to the user's device.

[0960] "Means for displaying and audibly transmitting a response on a user terminal" refers to a mechanism with a display and speaker for visually displaying and audibly playing received response data on a device.

[0961] A "means for generating a response based on a specific question using a generative AI model" is a computational model or algorithm for using generative AI technology to generate a meaningful response based on a user's question.

[0962] "Means for converting the generated response from text to speech" refers to speech synthesis technology or software for converting a text response into speech.

[0963] "Means for converting voice data into text data using a voice recognition engine" refers to a voice recognition technology that analyzes voice input and converts the voice into corresponding text data.

[0964] To implement this invention, a system that combines speech recognition, generative AI, speech synthesis, and communication technologies is required. This system is designed to support shopping in stores and uses smartphones as the main user terminal. The specific operation of the system and its components are described below.

[0965] System configuration

[0966] Accepting voice input

[0967] The user launches a dedicated application on their smartphone and inputs a question or request by voice, such as, "What is the most popular camera in this store?" The smartphone's microphone picks up this voice and captures it as digital data.

[0968] Analysis of voice data and conversion to text

[0969] The smartphone sends the captured voice data to a server installed in the cloud. The server converts the voice data into text data using a speech recognition engine (e.g., Google Speech Recognition API). The converted text data is passed to a generative AI model (e.g., GPT-3) to understand the user's intent.

[0970] Generating a response and retrieving product information

[0971] The server's generative AI model generates an appropriate response based on the analyzed text data. Specifically, it understands the user's question and generates the most appropriate answer. For example, a response such as "The most popular camera at present is model ABC" is generated. In this process, a query is sent to a product information database to obtain the latest sales information and popularity data.

[0972] Text-to-speech conversion of responses

[0973] The generated text response is converted to audio using a speech synthesis engine (e.g., Google Text-to-Speech API), allowing the user to hear the answer to their question aloud.

[0974] Sending and displaying a response to the user terminal

[0975] The server then sends the converted response data to the user's smartphone. The dedicated application then displays the response visually and audibly using a character avatar. For example, the character avatar may say, "Here's a popular camera," while also displaying an image of the product.

[0976] Examples of concrete examples and prompts

[0977] As a concrete example, consider a situation where a user is in a store and asks, "What is the most popular camera in this store?" The system accepts voice input, performs analysis and response generation on the server, and finally responds with audio and visual responses such as, "The most popular camera right now is model ABC." An example of a prompt sentence is as follows:

[0978] "What is the most popular camera in this store?"

[0979] This allows users to easily obtain product information and store guides via voice, providing a friendly and intuitive shopping experience.

[0980] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0981] Step 1:

[0982] A user launches a dedicated application installed on their smartphone and speaks a question or request, for example, "What is the most popular camera in this store?" The smartphone's microphone captures this voice and acquires it as digital data, which is the input and becomes the voice data for subsequent processing.

[0983] Step 2:

[0984] The device sends the captured audio data to the server. This is the audio data transmission process. The input from the device is the user's audio data, and the output is to send this data to the server. Specifically, data is sent to the server using an HTTP request or WebSocket.

[0985] Step 3:

[0986] The server converts the received voice data into text data using a speech recognition engine (e.g., Google Speech Recognition API). The input is the user's voice data, and the output is text data. This conversion is performed by the speech recognition engine. Specifically, it includes extracting features from the voice signal and analyzing phonemes.

[0987] Step 4:

[0988] The server uses a generative AI model (e.g., GPT-3) to generate a response based on the converted text data. The input is text data, and the output is the generated response text. This is where the generative AI model analyzes the meaning of the text and generates the optimal response. For example, a response such as "Currently, the most popular camera is the ABC model" is generated.

[0989] Step 5:

[0990] The server queries a product information database to find additional information to complement the generated response. The input is a text-based query, and the output is the corresponding product information. This step uses a database management system (e.g., MySQL) to look up the product information and parse the information.

[0991] Step 6:

[0992] The server converts the generated text response into audio using a speech synthesis engine (e.g., Google Text-to-Speech API). The input is the generated response text, and the output is the audio data. This conversion involves mapping text to audio and generating an audio file.

[0993] Step 7:

[0994] The server sends the generated voice data to the user's smartphone. The input is voice data, and the output is data transmission to the smartphone. This data transfer is performed using communication protocols such as HTTP requests and WebSocket.

[0995] Step 8:

[0996] The device uses a character avatar to display the received audio data visually and audibly. The input is the audio data sent from the server, and the output is audio playback and information display for the user. Specifically, the audio is played back through the smartphone's speaker, related information is displayed on the display, and the character avatar says things like, "Popular cameras are here."

[0997] The above processing flow allows users to easily obtain product information and store guides by voice, providing a familiar and intuitive shopping experience.

[0998] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0999] The present invention provides a character avatar that utilizes speech recognition, generative AI, and an emotion engine to function as a shopping assistant and guidance system. This system includes means for accepting a user's voice input, analyzing the voice input, generating a response based on the analysis results, transmitting the generated response to a user terminal, and displaying and vocalizing the response on the user terminal. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the user experience is further improved.

[1000] System Configuration

[1001] 1. User voice input

[1002] Users launch a dedicated application and voice their question or request, for example, "Which shoes are the most popular in this store?"

[1003] 2. Sending audio data

[1004] The device records the user's voice input with a microphone and stores it as digital audio data, which is then converted into a suitable format and sent to a server over the Internet.

[1005] 3. Analysis of audio data

[1006] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[1007] 4. Response generation based on analysis results

[1008] The server then passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[1009] 5. Searching for Information

[1010] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[1011] 6. Emotion Recognition with Emotion Engine

[1012] The server uses an emotion engine to recognize the user's emotional state from voice input and the user's past interactions, for example, determining whether the user is frustrated or happy.

[1013] 7. Regulating responses based on emotions

[1014] The server adjusts the response message based on the results of the emotion engine. For example, if the user is frustrated, it generates a more polite response or one that provides additional information.

[1015] 8. Sending the Response

[1016] The server generates a response message and sends it to the terminal. For example, the response message might say, "The most popular shoe at the moment is the 'XYZ model'. It comes in red, blue, and black."

[1017] 9. Display and Call to User

[1018] The device analyzes the received response message, and a character avatar communicates information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product and adding a polite comment based on the user's emotions.

[1019] 10. Receiving User Feedback

[1020] If the user wants more information, they can ask a follow-up question, such as "Do you have other colors?"

[1021] 11. Processing of follow-up questions

[1022] The device again records additional audio data and sends it to the server, which again analyzes the audio data and generates a new response, such as a message containing the information, "These shoes are available in red, blue, and black."

[1023] 12. Sending and Displaying Final Responses

[1024] The server sends a new response message to the terminal, which again provides the information to the user audibly and visually via the character avatar.

[1025] Specific examples

[1026] For example, if a user asks, "Which shoes are the most popular in this store?" while in a store, the system operates as follows: First, the user's voice input is recorded and sent to the server. The server converts the voice data into text and analyzes the content. The system then searches the product database based on the analysis results, identifies an "XYZ model," and recognizes the user's emotions using an emotion engine. For example, if the system recognizes that the user is excited, it generates a response containing a competitive expression such as "These shoes are especially recommended." The server then sends the generated response to the device, which uses a character avatar to convey the information both audibly and visually. If the user subsequently expresses interest and asks the follow-up question, "Do you have any other colors?", the system also provides color information through a similar process.

[1027] In this way, the system of the present invention combines speech recognition, generative AI, and an emotion engine to provide efficient real-time shopping support that adapts to the user's emotional state, thereby improving customer satisfaction and reducing the burden on staff.

[1028] The processing flow will be explained below.

[1029] Step 1:

[1030] The user launches the application and speaks a question or request into the microphone, for example, "Which shoes are the most popular in this store?"

[1031] Step 2:

[1032] The device records the user's voice input using a microphone and stores it as digital voice data.

[1033] Step 3:

[1034] The device converts the recorded audio data into an appropriate format (e.g., compresses and encodes it) and sends it to a server via the Internet.

[1035] Step 4:

[1036] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[1037] Step 5:

[1038] The server passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[1039] Step 6:

[1040] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[1041] Step 7:

[1042] The server receives the search results and passes the user's voice input data to the emotion engine, which analyzes the user's emotional state. For example, the emotion engine recognizes the user's voice data as "excited."

[1043] Step 8:

[1044] The server generates a response message based on the emotion engine results, depending on the user's emotions. For example, if the user is excited, the server creates a response such as "These shoes are especially recommended," including additional reasons for the recommendation.

[1045] Step 9:

[1046] The server sends the generated response message and appropriate product information to the terminal.

[1047] Step 10:

[1048] The device analyzes the response message and the character avatar communicates information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product with additional comments based on the user's emotions.

[1049] Step 11:

[1050] If the user wants more information, they can ask a follow-up question, such as "Do these shoes come in other colors?"

[1051] Step 12:

[1052] The device re-records the additional audio data, converts it, and sends it to the server.

[1053] Step 13:

[1054] The server again analyzes the voice data, generates new text, retrieves product color variation information from the database, and generates a new response message.

[1055] Step 14:

[1056] The server uses the emotion engine again to analyze the user's emotional state when asking the follow-up question and tailor an appropriate response message based on that emotional state. For example, if the user asks a question in a dissatisfied tone, the server generates a polite response such as, "We have a wide variety of colors, so please try them out."

[1057] Step 15:

[1058] The server sends a new response message to the terminal.

[1059] Step 16:

[1060] The device then re-prompts the user with a new response message, both audibly and visually, via a character avatar. The avatar responds audibly, saying, "These shoes are available in red, blue, and black," and displays the color variations on the screen. At the same time, it adds a comment that takes the user's feelings into consideration.

[1061] Through the above series of processing steps, the system of the present invention provides information in real time while adapting to the user's emotional state, realizing an efficient and fulfilling customer experience.

[1062] Example 2

[1063] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1064] In today's shopping experience, users want fast and accurate information, but traditional systems face the challenge of being unable to accurately grasp the user's intentions and emotions and provide appropriate responses. Furthermore, impersonal responses often fail to improve the user experience and leave the user feeling unsatisfied. This can lead to lower customer satisfaction and a loss of purchasing motivation.

[1065] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1066] In this invention, the server includes a means for passing voice data to a voice recognition engine and converting it into natural language text, a means for passing the converted text data to a generative AI model and analyzing the content of the text, and a means for recognizing the user's emotions and adjusting the response message based on the results. This makes it possible to accurately grasp the user's intentions and emotions and quickly generate an appropriate response.

[1067] (Definitions of important terms included in the claims)

[1068] "Voice input" is a means of recording the user's spoken words or voice as digital data.

[1069] "Digital audio data" refers to data obtained by converting an analog audio signal into a digital format.

[1070] "Recording" means the act of recording and saving the user's voice.

[1071] A "server" is a central processing unit that analyzes voice data and transmits the processing results to a user terminal.

[1072] A "speech recognition engine" is software or hardware that converts speech data into natural language text.

[1073] "Natural language text" is text data in a language that humans can understand that has been converted from voice data.

[1074] A "generative AI model" is an artificial intelligence algorithm designed to generate appropriate responses to specific inputs.

[1075] "Text content analysis" is the process of understanding the converted natural language text and identifying the user's intent.

[1076] A "response message" is text that contains a response or information to a user's inquiry.

[1077] "Emotion recognition" refers to determining a user's emotional state from the tone and content of their voice.

[1078] A "user terminal" is a digital device used by a user, including devices such as smartphones and tablets.

[1079] "Display and voice transmission" refers to displaying text information on the screen of the user terminal and transmitting it to the user by voice.

[1080] This system is a shopping assistant and guidance system that analyzes user voice input and generates responses. The system uses hardware and software that combines voice recognition, generative AI, secure communication methods, and an emotion recognition engine.

[1081] First, the user installs a dedicated application and uses a user device such as a smartphone or tablet to input voice commands. The user taps the microphone button to begin asking a question. For example, they can input a question by voice such as, "Which shoes are the most popular in this store?"

[1082] Next, the device will record the user's voice using a built-in or external microphone as digital audio data, store it in temporary memory, convert it into a predefined data format, and send it to the server using a secure communication protocol such as SSL / TLS.

[1083] The server first converts the voice data into natural language text using a speech recognition engine (e.g., Google Speech-to-Text API). For example, the voice data "Which shoes are the most popular in this store?" is converted into text "Which shoes are the most popular in this store?"

[1084] The converted text data is passed to a generative AI model (e.g., OpenAI GPT-4) for analysis. This analysis process understands the content of the input text to identify the user's intent. For example, the generative AI model generates an appropriate response in the form of a prompt: "The user is asking about 'the most popular shoes.' Please provide more information."

[1085] The server then searches a product database (e.g., MySQL or PostgreSQL) to retrieve relevant product information to generate a specific response. For example, it might find that "XYZ model shoes" are the most popular. During this process, the generative AI model uses the search results to create a specific response message.

[1086] To understand the user's emotional state, the server uses an emotion recognition engine (e.g., Microsoft Azure's Emotion API) to recognize emotions from voice input and past interaction data. For example, if the user is excited, a response such as "I highly recommend these shoes!" is generated.

[1087] The server then sends the generated response message to the user's device. The device receives the message and uses a character avatar to communicate information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product.

[1088] If the user wants more information, they can speak again, asking, "Do you have any other colors?" The device again records the voice data and sends it to the server. A response is generated and sent through the same process, providing the user with the latest information.

[1089] As described above, the present invention improves the user experience by accurately understanding the user's intentions and emotions and providing an optimal response.

[1090] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1091] Step 1:

[1092] The user launches the dedicated application and taps the microphone button to start voice input, for example, by speaking a question such as, "Which shoes are the most popular in this store?"

[1093] Step 2:

[1094] The device uses the smartphone's built-in microphone to record the user's voice as digital audio data. The recorded audio data is stored in short-term memory. The input is the user's voice, and the output is digital audio data.

[1095] Step 3:

[1096] The terminal transmits digital audio data to the server using a secure communication protocol such as SSL / TLS. The input is digital audio data, and the output is secure data transmission to the server.

[1097] Step 4:

[1098] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts it into natural language text. The input is digital voice data, and the output is natural language text.

[1099] Step 5:

[1100] The server passes the converted text data to a generative AI model (e.g., OpenAI GPT-4) and begins analyzing the text content. A prompt is created and the analysis proceeds. The input is natural language text, and the output is a prompt containing the analysis results.

[1101] Example: Prompt: "User is asking about 'Most Popular Shoes'. Please provide more information."

[1102] Step 6:

[1103] The server sends a query to a product database (e.g., MySQL or PostgreSQL) based on the analysis results of the generative AI model to search for related product information. The input is the analysis results, and the output is the query results.

[1104] Example: Retrieve information from a database that "XYZ model shoes" are the most popular.

[1105] Step 7:

[1106] The server uses an emotion recognition engine (e.g., Microsoft Azure's Emotion API) to recognize the user's emotions from voice input and past interaction data. The input is voice or text data, and the output is the user's emotional state.

[1107] For example, if the user is excited, a response like "I highly recommend these shoes!" will be generated.

[1108] Step 8:

[1109] The server uses a generative AI model to tailor a response message based on the recognized emotion. The generative AI model is used to create a specific response that reflects the emotional information. The input is the emotion recognition result, and the output is the tailored response message.

[1110] Step 9:

[1111] The server sends the generated response message to the user terminal. The input is the adjusted response message, and the output is the data transmission to the user terminal.

[1112] Step 10:

[1113] The terminal displays the received response message, and the character avatar provides information to the user audio and visually. The input is the response message, and the output is audio and visual display.

[1114] Example: An avatar says, "Here are some popular shoes," and displays an image of the product.

[1115] Step 11:

[1116] If the user wants more information, they again use voice input, in this case asking a question such as "Do you have any other colors?" The input is an additional voice question.

[1117] Step 12:

[1118] The terminal again records the additional audio data and sends it to the server. The input is the additional audio data, and the output is transmission to the server.

[1119] Step 13:

[1120] The server analyzes the new voice data and generates a new response. The same process is repeated to provide the information the user seeks. The input is a further voice question, and the output is a new response message.

[1121] For example: "These shoes come in red, blue, and black."

[1122] This enables the system to accurately understand the user's intentions and emotions and provide the most appropriate response.

[1123] (Application example 2)

[1124] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1125] Conventional shopping assistant systems only generate responses based on the user's voice input and do not consider the user's emotions, which prevents them from fully improving user satisfaction. Furthermore, they lack a means to provide information in an easy-to-understand visual and audio format, making it difficult to provide an intuitive and enjoyable user experience. Furthermore, there are limitations to the means to efficiently generate responses each time the user asks additional questions.

[1126] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1127] In this invention, the server includes means for accepting a voice input, means for analyzing the accepted voice input, means for generating a response based on the analysis result, means for transmitting the generated response to a user terminal, means for displaying the response and transmitting it audibly on the user terminal, means for recognizing emotions, means for adjusting the response based on the result of the emotion recognition means, and means for visually displaying the response and transmitting it audibly using a character avatar, thereby enabling response generation that takes the user's emotions into consideration and providing visual and audio information using a character avatar.

[1128] "Means for accepting voice input" refers to a device or function for recognizing voice input from a user and recording it as digital data.

[1129] "Means for analyzing received voice input" refers to technology for processing recorded voice data and converting it into natural language text.

[1130] The "means for generating a response based on the analysis results" refers to a generative AI model or algorithm that converts voice data into text and then creates an appropriate response based on that content.

[1131] The "means for transmitting the generated response to the user terminal" refers to a technique for transferring the generated response message from the server or central system to the user's device.

[1132] "Means for displaying and audibly transmitting a response on a user terminal" refers to equipment or software for visually displaying information and audibly transmitting a response on a user's device.

[1133] "Emotion recognition means" is a technology that analyzes emotions from a user's voice or text and identifies states such as joy, anger, sadness, and happiness.

[1134] The "means for adjusting a response based on the results of the emotion recognition means" is software or an algorithm for utilizing the results of the emotion recognition means to generate a response that is adapted to the emotional state of the user.

[1135] A "means for visually displaying and audibly transmitting responses using character avatars" is a device or software that uses animation or graphic characters to display and communicate information audibly.

[1136] To implement the present invention, the following system configuration and procedure are used.

[1137] System Configuration

[1138] 1. A means of accepting voice input

[1139] The user inputs voice using a smartphone application, and the smartphone's microphone records the voice and stores it as digital data.

[1140] 2. A means of analyzing the received voice input

[1141] This digital voice data is sent via a smartphone application over the internet to a server, which then uses a speech recognition engine such as Google Cloud Speech-to-Text to analyze the voice data and convert it into natural language text.

[1142] 3. A means of generating a response based on the analysis results

[1143] The server uses the parsed text data as input for a generative AI model (e.g., OpenAI API) that generates an appropriate response based on the user's question or request.

[1144] 4. A means of sending the generated response to the user terminal

[1145] The generated response is forwarded from the server to the user's smartphone.

[1146] 5. Means for displaying and audibly transmitting responses on the user terminal

[1147] The user's smartphone displays the received response in visual and audio formats, with a character avatar responding with audio and displaying text information on the screen for easy visual understanding by the user.

[1148] 6. Emotion recognition means

[1149] The server uses emotion recognition engines such as IBM Watson Tone Analyzer to determine the user's emotions based on the user's voice and text data.

[1150] 7. A means of adjusting responses based on the results of emotion recognition measures

[1151] Based on the results of emotion recognition, a generative AI model generates a tailored response that is adapted to the user's emotional state.

[1152] 8. A means of visually displaying and audibly issuing responses using character avatars

[1153] A character avatar within the smartphone application provides visual and audio information to the user.

[1154] Process Overview

[1155] The smartphone application records the user's voice and sends it to a server. The server converts the voice into text and passes it to a generative AI model. The generative AI model generates a response based on the text, and an emotion recognition engine analyzes the user's emotions. This information is combined to generate an appropriate response and send it to the user's device. A character avatar provides visual and audio responses to the user, improving the user experience.

[1156] Specific examples

[1157] For example, if a user asks "Which shoes are the most popular in this store?" in a store, the system works as follows: It records the user's voice input and sends it over the internet to a server. The server uses a speech recognition engine to convert the speech to text and passes that text as input to a generative AI model. The AI ​​model researches the "most popular shoes" and generates text to return the results to the user.

[1158] The generated text is something like, "The most popular shoe right now is the XYZ model. We're especially offering a 25% off campaign." If the emotion recognition engine detects that the user is excited, it adds an emphasis phrase like, "We highly recommend it!"

[1159] The character avatar then vocalizes the response and visually displays a corresponding image and description of the shoe. If the user asks a follow-up question, such as "Do you have any other colors?", the same technical steps are taken to generate the answer.

[1160] Prompt Sentence Examples

[1161] 1. Voice input: "What are the most popular shoes in this store?"

[1162] 2. Prompt the generative AI model: "Which shoes are the most popular in this store? User emotion: Excitement"

[1163] This provides real-time shopping support that takes into account the user's emotions.

[1164] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1165] Step 1:

[1166] The user launches the smartphone application and enters a question or request by voice.

[1167] Input: Speech data (e.g., "Which shoes are the most popular in this store?")

[1168] Output: Digital audio data

[1169] How it works: The smartphone's microphone records the user's voice, and the application converts the voice data into a digital format and stores it.

[1170] Step 2:

[1171] The digital audio data recorded by the device is sent to the server.

[1172] Input: Digital audio data

[1173] Output: Audio data sent to the server

[1174] Specific operation: The smartphone application sends voice data to a server via the Internet.

[1175] Step 3:

[1176] The voice data received by the server is analyzed using a voice recognition engine and converted into natural language text.

[1177] Input: Audio data

[1178] Output: Natural language text (e.g., "Which shoes are the most popular in this store?")

[1179] How it works: The server uses a speech recognition engine such as Google Cloud Speech-to-Text to analyze the voice data and convert it into natural language text.

[1180] Step 4:

[1181] The server passes the converted text to a generative AI model, which generates a corresponding response.

[1182] Input: Natural language text

[1183] Output: Response text (e.g., "The most popular shoe in our store right now is the XYZ model.")

[1184] Specific operation: The server passes text data as input to a generative AI model (e.g., OpenAI API) and generates a response appropriate to the user's question.

[1185] Step 5:

[1186] The server uses an emotion recognition engine to analyze emotions from the user's voice input and text data.

[1187] Input: Audio or text data

[1188] Output: Emotion data (e.g., "excited" or "frustrated")

[1189] Specific operation: The server uses an emotion recognition engine such as IBM Watson Tone Analyzer to recognize the user's emotions.

[1190] Step 6:

[1191] The server adjusts the generated responses based on the emotion data to generate a final response.

[1192] Input: Response text, emotion data

[1193] Output: Tailored response text (e.g., "The most popular shoe in our store right now is the XYZ model! Highly recommended!")

[1194] How it works: The generative AI model takes in emotion data and adds appropriate expressions based on the user's emotional state.

[1195] Step 7:

[1196] The server sends the adjusted response to the user terminal.

[1197] Input: Tailored response text

[1198] Output: Response text sent to the user's terminal

[1199] Specific operation: The server sends the generated response message to the user's smartphone via the Internet.

[1200] Step 8:

[1201] The user's smartphone displays visual and audio responses using a character avatar.

[1202] Input: Response text

[1203] Output: Audio response and visual display

[1204] Specific operation: A character avatar within the smartphone application will vocalize the generated response and display text information and images of related products on the screen.

[1205] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1206] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1207] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1208] [Fourth embodiment]

[1209] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1210] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1211] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1212] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1213] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1214] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1215] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1216] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1217] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1218] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1219] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1220] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1221] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1222] The present invention provides a character avatar that utilizes speech recognition and generation AI to function as a shopping assistant and guidance system. The system includes means for accepting speech input, means for analyzing the speech data, means for generating a response based on the analysis results, means for transmitting the generated response to a user terminal, and means for displaying and audibly transmitting the response on the user terminal.

[1223] System Configuration

[1224] 1. User voice input

[1225] Users launch a dedicated application and voice their question or request, for example, "Which shoes are the most popular in this store?"

[1226] 2. Sending audio data

[1227] The device records the user's voice and sends it as data to the server.

[1228] 3. Analysis of audio data

[1229] The server converts the received voice data into text data using a speech recognition engine. A generative AI model is used to understand the user's intent from this text data. As a result of the analysis, the generative AI generates a query to identify the "most popular shoes."

[1230] 4. Information Search

[1231] The server uses the generated query to search for relevant information from a product database, for example, "the most popular shoes" to identify "XYZ model."

[1232] 5. Generating the Response

[1233] The server generates a response message with the search results, for example, a message containing information such as "Currently, the most popular item is the 'XYZ model' shoe."

[1234] 6. Sending a response to the user

[1235] The server sends the generated response message to the terminal.

[1236] The device conveys the received message to the user audibly and visually through a character avatar, who says, "Here are some popular shoes," while displaying an image of the product.

[1237] 7. Gathering Feedback

[1238] If the user wants more information, they can say something like, "Do these shoes come in other colors?"

[1239] The terminal transmits this additional audio data back to the server.

[1240] The server analyzes the voice data again, generates a new response, and sends it to the terminal.

[1241] The device will then display the received response audibly and visually using a character avatar, for example, "These shoes are available in red, blue, and black."

[1242] Specific examples

[1243] For example, if a user asks "Which shoes are the most popular in this store?" while in a store, the system operates as follows: First, the device accepts the user's voice input and sends it to the server. The server converts the voice data into text and analyzes the content. It then searches the product database based on the analysis results and identifies the "XYZ model." The server generates a response message saying "The XYZ model is the most popular" and sends it to the device. The device then conveys this response message aloud through a character avatar and simultaneously displays an image. If the user then asks, "Are these shoes available in other colors?", the same process is used to provide color information.

[1244] In this way, the present invention utilizes voice recognition and generative AI to provide users with a real-time and efficient shopping assistant service, thereby improving customer satisfaction and reducing the burden on staff.

[1245] The processing flow will be explained below.

[1246] Step 1:

[1247] To begin interacting with the character avatar, users launch a dedicated application and speak their questions or requests into a microphone, for example, "Which shoes are the most popular in this store?"

[1248] Step 2:

[1249] The device records the user's voice input using a microphone and stores it as digital voice data.

[1250] Step 3:

[1251] The device converts the recorded audio data into an appropriate format (e.g., compresses and encodes it) and sends it to a server via the Internet.

[1252] Step 4:

[1253] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[1254] Step 5:

[1255] The server passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[1256] Step 6:

[1257] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[1258] Step 7:

[1259] The server receives the search results and generates a response message to provide to the user, for example, "The most popular shoe at the moment is the 'XYZ model'."

[1260] Step 8:

[1261] The server generates a response message and sends it to the terminal.

[1262] Step 9:

[1263] The device analyzes the received response message, and the character avatar communicates information to the user audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product.

[1264] Step 10:

[1265] If the user wants more information, they can ask a follow-up question, such as "Do these shoes come in other colors?"

[1266] Step 11:

[1267] The device re-records the additional audio data, converts it, and sends it to the server.

[1268] Step 12:

[1269] The server again analyzes the audio data and generates new text to generate appropriate responses to follow-up questions, such as "These shoes are available in red, blue, and black."

[1270] Step 13:

[1271] The server sends a new response message to the terminal.

[1272] Step 14:

[1273] The device then conveys a new response message to the user via a character avatar, who responds aloud, "These shoes are available in red, blue, and black," and displays the color variations on the screen.

[1274] Through the above series of steps, the system of the present invention responds to user requests in real time and provides necessary information, thereby increasing customer satisfaction.

[1275] Example 1

[1276] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1277] Conventional shopping assistant systems have had the problem of being unable to provide a quick and accurate response when users ask questions. In particular, when obtaining information using voice input, the accuracy of voice recognition and analysis is low, making it difficult to accurately understand the user's intent and provide an appropriate response. Furthermore, responses to follow-up questions from users can be slow, resulting in a decline in customer satisfaction.

[1278] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1279] In this invention, the server includes a means for converting received voice data into text data, a means for analyzing the user's intent using a generative AI model, and a means for generating a query based on the analysis and searching for information from a database. This increases the accuracy of voice recognition, making it possible to accurately analyze the user's intent and quickly generate an appropriate response.

[1280] "Voice input" refers to the act of recording the user's voice through a microphone on a device.

[1281] "Digital audio data" refers to data obtained by converting an analog audio signal into a digital format.

[1282] The "server" is a central system responsible for analyzing voice data, searching the database, generating responses, etc.

[1283] "Text data" is character information converted from speech by a speech recognition engine.

[1284] A "generative AI model" is an algorithm that uses artificial intelligence to understand user intent and generate a response.

[1285] A "query" is a query command for retrieving specific information from a database.

[1286] A "database" is a collection of data that stores specific product information and other related information.

[1287] A "response message" is a server-generated text answer to a user's question.

[1288] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[1289] A "character avatar" is a virtual character that displays a response message audibly and visually.

[1290] This invention is a shopping assistant and guidance system that utilizes speech recognition and generation AI, allowing users to search for product information through voice input and receive responses in real time.

[1291] The system includes the following components:

[1292] A means of accepting voice input

[1293] A means to record received voice input and send it to a server as digital audio data

[1294] A means of converting received voice data into text data

[1295] A means of analyzing user intent using generative AI models

[1296] A means of generating queries based on analysis and retrieving information from a database

[1297] A means for generating a response message based on the searched information and transmitting it to the user terminal.

[1298] Means for displaying a response message audibly and visually at the user terminal

[1299] As a specific example, we use the Google Cloud Speech-to-Text API to convert voice data into text, and then use a generative AI model such as OpenAI's GPT-3 to analyze the captured text data and generate appropriate responses to user questions.

[1300] For example, if a user asks "Which shoes are the most popular in this store?" by voice, the system works as follows: The user launches a dedicated application and speaks the question. The device records this voice and sends it to the server. The server converts the voice data into text data and analyzes it using a generative AI model. The resulting query is run against the database, and the "XYZ model" is identified as the most popular shoe. The server generates a response message and sends it to the device. The device communicates the response to the user audibly and visually through a character avatar.

[1301] Examples of prompts include "What are the most popular shoes in this store?" or "Do you have these shoes in other colors?"

[1302] In this way, by combining speech recognition and generative AI, the present invention provides highly accurate information search and precise responses in a short time, aiming to improve customer satisfaction and reduce the burden on store staff.

[1303] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1304] Step 1:

[1305] The user starts a dedicated application and performs voice input.

[1306] Input: User-uttered speech (e.g., "Which shoes are the most popular in this store?")

[1307] What it does: Records audio through the microphone on your smartphone or tablet.

[1308] Output: Recorded audio data

[1309] Step 2:

[1310] The device digitizes the recorded audio data and sends it to the server.

[1311] Input: Recorded audio data

[1312] What it does: Converts voice data into digital form and sends it as data packets to a server.

[1313] Output: Digital audio data sent to the server

[1314] Step 3:

[1315] The server converts the received voice data into text data.

[1316] Input: Digital audio data sent to the server

[1317] How it works: Converts audio data into text using the Google Cloud Speech-to-Text API or similar.

[1318] Output: Text data (e.g., "Which shoes are the most popular in this store?")

[1319] Step 4:

[1320] The server inputs the text data into a generative AI model and analyzes the user's intent.

[1321] Input: Text data

[1322] How it works: Text data is fed into a generative AI model such as OpenAI's GPT-3, which analyzes the user's intent (the content of the request).

[1323] Output: Analysis results (e.g., a query to identify "most popular shoes")

[1324] Step 5:

[1325] The server retrieves information from a database based on the analysis results.

[1326] Input: The query generated as a result of the analysis

[1327] What it does: Queries the product database to find information about the corresponding product.

[1328] Output: Search results (e.g., "XYZ model" shoes are the most popular)

[1329] Step 6:

[1330] The server generates a response message based on the search results and transmits it to the user terminal.

[1331] Input: Search results

[1332] What it does: Based on the search results, it generates a text response message to the user's question.

[1333] Output: Response message (e.g., "Currently, the most popular shoe is the XYZ model.")

[1334] Step 7:

[1335] The terminal transmits the response message received from the server to the user through the character avatar.

[1336] Input: Response message sent by the server

[1337] What it does: It analyzes the response message and communicates it to the user audibly and visually through a character avatar. For example, the avatar might say, "Here are some popular shoes," while displaying a product image.

[1338] Output: The response displayed and spoken to the user

[1339] Step 8:

[1340] If the user desires more information, they can ask additional questions by voice and start the process again.

[1341] Input: An additional spoken question from the user (e.g., "Do these shoes come in other colors?")

[1342] How it works: The device records audio and sends the digital audio data to the server, which then converts the audio data back into text, analyzes it with a generative AI model, generates a response message, and sends it to the device.

[1343] Output: An additional response that is displayed and spoken to the user (e.g., "These shoes come in red, blue, and black")

[1344] This allows users to efficiently obtain necessary product information in real time.

[1345] (Application example 1)

[1346] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1347] Conventional shopping support systems for brick-and-mortar stores have the problem of not being able to provide efficient support because the product information search and in-store guidance that users face are not performed in real time.In addition, there is a lack of interaction through the user's voice, and there are also issues with familiarity and intuitive operability.

[1348] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1349] In this invention, the server includes means for accepting voice input, means for analyzing the accepted voice input, means for generating a response based on the analysis result, means for transmitting the generated response to a user terminal, means for displaying and audibly transmitting the response on the user terminal, means for generating a response based on a specific question using a generative AI model, means for converting the generated response from text to voice, and means for converting voice data to text data using a voice recognition engine. This allows users to easily obtain product information and store guides by voice, enabling a familiar and intuitive shopping experience.

[1350] The "means for accepting voice input" refers to a microphone or peripheral device for capturing the user's voice, and is a mechanism for acquiring voice signals as digital data.

[1351] "Means for analyzing received voice input" refers to software or algorithms that convert voice data into text data and analyze the content to understand the user's intent or question.

[1352] "Means for generating a response based on the analysis results" refers to the process of using algorithms or software, particularly generative AI models, to generate appropriate responses based on the results of voice analysis.

[1353] "Means for transmitting the generated response to the user terminal" refers to a communication means or protocol for transferring the generated response data from the server to the user's device.

[1354] "Means for displaying and audibly transmitting a response on a user terminal" refers to a mechanism with a display and speaker for visually displaying and audibly playing received response data on a device.

[1355] A "means for generating a response based on a specific question using a generative AI model" is a computational model or algorithm for using generative AI technology to generate a meaningful response based on a user's question.

[1356] "Means for converting the generated response from text to speech" refers to speech synthesis technology or software for converting a text response into speech.

[1357] "Means for converting voice data into text data using a voice recognition engine" refers to a voice recognition technology that analyzes voice input and converts the voice into corresponding text data.

[1358] To implement this invention, a system that combines speech recognition, generative AI, speech synthesis, and communication technologies is required. This system is designed to support shopping in stores and uses smartphones as the main user terminal. The specific operation of the system and its components are described below.

[1359] System configuration

[1360] Accepting voice input

[1361] The user launches a dedicated application on their smartphone and inputs a question or request by voice, such as, "What is the most popular camera in this store?" The smartphone's microphone picks up this voice and captures it as digital data.

[1362] Analysis of voice data and conversion to text

[1363] The smartphone sends the captured voice data to a server installed in the cloud. The server converts the voice data into text data using a speech recognition engine (e.g., Google Speech Recognition API). The converted text data is passed to a generative AI model (e.g., GPT-3) to understand the user's intent.

[1364] Generating a response and retrieving product information

[1365] The server's generative AI model generates an appropriate response based on the analyzed text data. Specifically, it understands the user's question and generates the most appropriate answer. For example, a response such as "The most popular camera at present is model ABC" is generated. In this process, a query is sent to a product information database to obtain the latest sales information and popularity data.

[1366] Text-to-speech conversion of responses

[1367] The generated text response is converted to audio using a speech synthesis engine (e.g., Google Text-to-Speech API), allowing the user to hear the answer to their question aloud.

[1368] Sending and displaying a response to the user terminal

[1369] The server then sends the converted response data to the user's smartphone. The dedicated application then displays the response visually and audibly using a character avatar. For example, the character avatar may say, "Here's a popular camera," while also displaying an image of the product.

[1370] Examples of concrete examples and prompts

[1371] As a concrete example, consider a situation where a user is in a store and asks, "What is the most popular camera in this store?" The system accepts voice input, performs analysis and response generation on the server, and finally responds with audio and visual responses such as, "The most popular camera right now is model ABC." An example of a prompt sentence is as follows:

[1372] "What is the most popular camera in this store?"

[1373] This allows users to easily obtain product information and store guides via voice, providing a friendly and intuitive shopping experience.

[1374] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1375] Step 1:

[1376] A user launches a dedicated application installed on their smartphone and speaks a question or request, for example, "What is the most popular camera in this store?" The smartphone's microphone captures this voice and acquires it as digital data, which is the input and becomes the voice data for subsequent processing.

[1377] Step 2:

[1378] The device sends the captured audio data to the server. This is the audio data transmission process. The input from the device is the user's audio data, and the output is to send this data to the server. Specifically, data is sent to the server using an HTTP request or WebSocket.

[1379] Step 3:

[1380] The server converts the received voice data into text data using a speech recognition engine (e.g., Google Speech Recognition API). The input is the user's voice data, and the output is text data. This conversion is performed by the speech recognition engine. Specifically, it includes extracting features from the voice signal and analyzing phonemes.

[1381] Step 4:

[1382] The server uses a generative AI model (e.g., GPT-3) to generate a response based on the converted text data. The input is text data, and the output is the generated response text. This is where the generative AI model analyzes the meaning of the text and generates the optimal response. For example, a response such as "Currently, the most popular camera is the ABC model" is generated.

[1383] Step 5:

[1384] The server queries a product information database to find additional information to complement the generated response. The input is a text-based query, and the output is the corresponding product information. This step uses a database management system (e.g., MySQL) to look up the product information and parse the information.

[1385] Step 6:

[1386] The server converts the generated text response into audio using a speech synthesis engine (e.g., Google Text-to-Speech API). The input is the generated response text, and the output is the audio data. This conversion involves mapping text to audio and generating an audio file.

[1387] Step 7:

[1388] The server sends the generated voice data to the user's smartphone. The input is voice data, and the output is data transmission to the smartphone. This data transfer is performed using communication protocols such as HTTP requests and WebSocket.

[1389] Step 8:

[1390] The device uses a character avatar to display the received audio data visually and audibly. The input is the audio data sent from the server, and the output is audio playback and information display for the user. Specifically, the audio is played back through the smartphone's speaker, related information is displayed on the display, and the character avatar says things like, "Popular cameras are here."

[1391] The above processing flow allows users to easily obtain product information and store guides by voice, providing a familiar and intuitive shopping experience.

[1392] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1393] The present invention provides a character avatar that utilizes speech recognition, generative AI, and an emotion engine to function as a shopping assistant and guidance system. This system includes means for accepting a user's voice input, analyzing the voice input, generating a response based on the analysis results, transmitting the generated response to a user terminal, and displaying and vocalizing the response on the user terminal. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the user experience is further improved.

[1394] System Configuration

[1395] 1. User voice input

[1396] Users launch a dedicated application and voice their question or request, for example, "Which shoes are the most popular in this store?"

[1397] 2. Sending audio data

[1398] The device records the user's voice input with a microphone and stores it as digital audio data, which is then converted into a suitable format and sent to a server over the Internet.

[1399] 3. Analysis of audio data

[1400] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[1401] 4. Response generation based on analysis results

[1402] The server then passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[1403] 5. Searching for Information

[1404] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[1405] 6. Emotion Recognition with Emotion Engine

[1406] The server uses an emotion engine to recognize the user's emotional state from voice input and the user's past interactions, for example, determining whether the user is frustrated or happy.

[1407] 7. Regulating responses based on emotions

[1408] The server adjusts the response message based on the results of the emotion engine. For example, if the user is frustrated, it generates a more polite response or one that provides additional information.

[1409] 8. Sending the Response

[1410] The server generates a response message and sends it to the terminal. For example, the response message might say, "The most popular shoe at the moment is the 'XYZ model'. It comes in red, blue, and black."

[1411] 9. Display and Call to User

[1412] The device analyzes the received response message, and a character avatar communicates information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product and adding a polite comment based on the user's emotions.

[1413] 10. Receiving User Feedback

[1414] If the user wants more information, they can ask a follow-up question, such as "Do you have other colors?"

[1415] 11. Processing of follow-up questions

[1416] The device again records additional audio data and sends it to the server, which again analyzes the audio data and generates a new response, such as a message containing the information, "These shoes are available in red, blue, and black."

[1417] 12. Sending and Displaying Final Responses

[1418] The server sends a new response message to the terminal, which again provides the information to the user audibly and visually via the character avatar.

[1419] Specific examples

[1420] For example, if a user asks, "Which shoes are the most popular in this store?" while in a store, the system operates as follows: First, the user's voice input is recorded and sent to the server. The server converts the voice data into text and analyzes the content. The system then searches the product database based on the analysis results, identifies an "XYZ model," and recognizes the user's emotions using an emotion engine. For example, if the system recognizes that the user is excited, it generates a response containing a competitive expression such as "These shoes are especially recommended." The server then sends the generated response to the device, which uses a character avatar to convey the information both audibly and visually. If the user subsequently expresses interest and asks the follow-up question, "Do you have any other colors?", the system also provides color information through a similar process.

[1421] In this way, the system of the present invention combines speech recognition, generative AI, and an emotion engine to provide efficient real-time shopping support that adapts to the user's emotional state, thereby improving customer satisfaction and reducing the burden on staff.

[1422] The processing flow will be explained below.

[1423] Step 1:

[1424] The user launches the application and speaks a question or request into the microphone, for example, "Which shoes are the most popular in this store?"

[1425] Step 2:

[1426] The device records the user's voice input using a microphone and stores it as digital voice data.

[1427] Step 3:

[1428] The device converts the recorded audio data into an appropriate format (e.g., compresses and encodes it) and sends it to a server via the Internet.

[1429] Step 4:

[1430] The server passes the received voice data to a speech recognition engine and converts it into natural language text. For example, the voice data "Which shoes are the most popular in this store?" is converted into text.

[1431] Step 5:

[1432] The server passes the converted text data to a generative AI model, which analyzes the text content to determine the user's intent (e.g., "I want to know the most popular shoes").

[1433] Step 6:

[1434] Based on the analysis results, the server queries the product database to find relevant product information, for example, identifying from the database that "XYZ model shoes" are the most popular.

[1435] Step 7:

[1436] The server receives the search results and passes the user's voice input data to the emotion engine, which analyzes the user's emotional state. For example, the emotion engine recognizes the user's voice data as "excited."

[1437] Step 8:

[1438] The server generates a response message based on the emotion engine results, depending on the user's emotions. For example, if the user is excited, the server creates a response such as "These shoes are especially recommended," including additional reasons for the recommendation.

[1439] Step 9:

[1440] The server sends the generated response message and appropriate product information to the terminal.

[1441] Step 10:

[1442] The device analyzes the response message and the character avatar communicates information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product with additional comments based on the user's emotions.

[1443] Step 11:

[1444] If the user wants more information, they can ask a follow-up question, such as "Do these shoes come in other colors?"

[1445] Step 12:

[1446] The device re-records the additional audio data, converts it, and sends it to the server.

[1447] Step 13:

[1448] The server again analyzes the voice data, generates new text, retrieves product color variation information from the database, and generates a new response message.

[1449] Step 14:

[1450] The server uses the emotion engine again to analyze the user's emotional state when asking the follow-up question and tailor an appropriate response message based on that emotional state. For example, if the user asks a question in a dissatisfied tone, the server generates a polite response such as, "We have a wide variety of colors, so please try them out."

[1451] Step 15:

[1452] The server sends a new response message to the terminal.

[1453] Step 16:

[1454] The device then re-prompts the user with a new response message, both audibly and visually, via a character avatar. The avatar responds audibly, saying, "These shoes are available in red, blue, and black," and displays the color variations on the screen. At the same time, it adds a comment that takes the user's feelings into consideration.

[1455] Through the above series of processing steps, the system of the present invention provides information in real time while adapting to the user's emotional state, realizing an efficient and fulfilling customer experience.

[1456] Example 2

[1457] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1458] In today's shopping experience, users want fast and accurate information, but traditional systems face the challenge of being unable to accurately grasp the user's intentions and emotions and provide appropriate responses. Furthermore, impersonal responses often fail to improve the user experience and leave the user feeling unsatisfied. This can lead to lower customer satisfaction and a loss of purchasing motivation.

[1459] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1460] In this invention, the server includes a means for passing voice data to a voice recognition engine and converting it into natural language text, a means for passing the converted text data to a generative AI model and analyzing the content of the text, and a means for recognizing the user's emotions and adjusting the response message based on the results. This makes it possible to accurately grasp the user's intentions and emotions and quickly generate an appropriate response.

[1461] (Definitions of important terms included in the claims)

[1462] "Voice input" is a means of recording the user's spoken words or voice as digital data.

[1463] "Digital audio data" refers to data obtained by converting an analog audio signal into a digital format.

[1464] "Recording" means the act of recording and saving the user's voice.

[1465] A "server" is a central processing unit that analyzes voice data and transmits the processing results to a user terminal.

[1466] A "speech recognition engine" is software or hardware that converts speech data into natural language text.

[1467] "Natural language text" is text data in a language that humans can understand that has been converted from voice data.

[1468] A "generative AI model" is an artificial intelligence algorithm designed to generate appropriate responses to specific inputs.

[1469] "Text content analysis" is the process of understanding the converted natural language text and identifying the user's intent.

[1470] A "response message" is text that contains a response or information to a user's inquiry.

[1471] "Emotion recognition" refers to determining a user's emotional state from the tone and content of their voice.

[1472] A "user terminal" is a digital device used by a user, including devices such as smartphones and tablets.

[1473] "Display and voice transmission" refers to displaying text information on the screen of the user terminal and transmitting it to the user by voice.

[1474] This system is a shopping assistant and guidance system that analyzes user voice input and generates responses. The system uses hardware and software that combines voice recognition, generative AI, secure communication methods, and an emotion recognition engine.

[1475] First, the user installs a dedicated application and uses a user device such as a smartphone or tablet to input voice commands. The user taps the microphone button to begin asking a question. For example, they can input a question by voice such as, "Which shoes are the most popular in this store?"

[1476] Next, the device will record the user's voice using a built-in or external microphone as digital audio data, store it in temporary memory, convert it into a predefined data format, and send it to the server using a secure communication protocol such as SSL / TLS.

[1477] The server first converts the voice data into natural language text using a speech recognition engine (e.g., Google Speech-to-Text API). For example, the voice data "Which shoes are the most popular in this store?" is converted into text "Which shoes are the most popular in this store?"

[1478] The converted text data is passed to a generative AI model (e.g., OpenAI GPT-4) for analysis. This analysis process understands the content of the input text to identify the user's intent. For example, the generative AI model generates an appropriate response in the form of a prompt: "The user is asking about 'the most popular shoes.' Please provide more information."

[1479] The server then searches a product database (e.g., MySQL or PostgreSQL) to retrieve relevant product information to generate a specific response. For example, it might find that "XYZ model shoes" are the most popular. During this process, the generative AI model uses the search results to create a specific response message.

[1480] To understand the user's emotional state, the server uses an emotion recognition engine (e.g., Microsoft Azure's Emotion API) to recognize emotions from voice input and past interaction data. For example, if the user is excited, a response such as "I highly recommend these shoes!" is generated.

[1481] The server then sends the generated response message to the user's device. The device receives the message and uses a character avatar to communicate information to the user both audibly and visually. For example, the avatar may say, "Here are some popular shoes," while displaying an image of the product.

[1482] If the user wants more information, they can speak again, asking, "Do you have any other colors?" The device again records the voice data and sends it to the server. A response is generated and sent through the same process, providing the user with the latest information.

[1483] As described above, the present invention improves the user experience by accurately understanding the user's intentions and emotions and providing an optimal response.

[1484] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1485] Step 1:

[1486] The user launches the dedicated application and taps the microphone button to start voice input, for example, by speaking a question such as, "Which shoes are the most popular in this store?"

[1487] Step 2:

[1488] The device uses the smartphone's built-in microphone to record the user's voice as digital audio data. The recorded audio data is stored in short-term memory. The input is the user's voice, and the output is digital audio data.

[1489] Step 3:

[1490] The terminal transmits digital audio data to the server using a secure communication protocol such as SSL / TLS. The input is digital audio data, and the output is secure data transmission to the server.

[1491] Step 4:

[1492] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts it into natural language text. The input is digital voice data, and the output is natural language text.

[1493] Step 5:

[1494] The server passes the converted text data to a generative AI model (e.g., OpenAI GPT-4) and begins analyzing the text content. A prompt is created and the analysis proceeds. The input is natural language text, and the output is a prompt containing the analysis results.

[1495] Example: Prompt: "User is asking about 'Most Popular Shoes'. Please provide more information."

[1496] Step 6:

[1497] The server sends a query to a product database (e.g., MySQL or PostgreSQL) based on the analysis results of the generative AI model to search for related product information. The input is the analysis results, and the output is the query results.

[1498] Example: Retrieve information from a database that "XYZ model shoes" are the most popular.

[1499] Step 7:

[1500] The server uses an emotion recognition engine (e.g., Microsoft Azure's Emotion API) to recognize the user's emotions from voice input and past interaction data. The input is voice or text data, and the output is the user's emotional state.

[1501] For example, if the user is excited, a response like "I highly recommend these shoes!" will be generated.

[1502] Step 8:

[1503] The server uses a generative AI model to tailor a response message based on the recognized emotion. The generative AI model is used to create a specific response that reflects the emotional information. The input is the emotion recognition result, and the output is the tailored response message.

[1504] Step 9:

[1505] The server sends the generated response message to the user terminal. The input is the adjusted response message, and the output is the data transmission to the user terminal.

[1506] Step 10:

[1507] The terminal displays the received response message, and the character avatar provides information to the user audio and visually. The input is the response message, and the output is audio and visual display.

[1508] Example: An avatar says, "Here are some popular shoes," and displays an image of the product.

[1509] Step 11:

[1510] If the user wants more information, they again use voice input, in this case asking a question such as "Do you have any other colors?" The input is an additional voice question.

[1511] Step 12:

[1512] The terminal again records the additional audio data and sends it to the server. The input is the additional audio data, and the output is transmission to the server.

[1513] Step 13:

[1514] The server analyzes the new voice data and generates a new response. The same process is repeated to provide the information the user seeks. The input is a further voice question, and the output is a new response message.

[1515] For example: "These shoes come in red, blue, and black."

[1516] This enables the system to accurately understand the user's intentions and emotions and provide the most appropriate response.

[1517] (Application example 2)

[1518] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1519] Conventional shopping assistant systems only generate responses based on the user's voice input and do not consider the user's emotions, which prevents them from fully improving user satisfaction. Furthermore, they lack a means to provide information in an easy-to-understand visual and audio format, making it difficult to provide an intuitive and enjoyable user experience. Furthermore, there are limitations to the means to efficiently generate responses each time the user asks additional questions.

[1520] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1521] In this invention, the server includes means for accepting a voice input, means for analyzing the accepted voice input, means for generating a response based on the analysis result, means for transmitting the generated response to a user terminal, means for displaying the response and transmitting it audibly on the user terminal, means for recognizing emotions, means for adjusting the response based on the result of the emotion recognition means, and means for visually displaying the response and transmitting it audibly using a character avatar, thereby enabling response generation that takes the user's emotions into consideration and providing visual and audio information using a character avatar.

[1522] "Means for accepting voice input" refers to a device or function for recognizing voice input from a user and recording it as digital data.

[1523] "Means for analyzing received voice input" refers to technology for processing recorded voice data and converting it into natural language text.

[1524] The "means for generating a response based on the analysis results" refers to a generative AI model or algorithm that converts voice data into text and then creates an appropriate response based on that content.

[1525] The "means for transmitting the generated response to the user terminal" refers to a technique for transferring the generated response message from the server or central system to the user's device.

[1526] "Means for displaying and audibly transmitting a response on a user terminal" refers to equipment or software for visually displaying information and audibly transmitting a response on a user's device.

[1527] "Emotion recognition means" is a technology that analyzes emotions from a user's voice or text and identifies states such as joy, anger, sadness, and happiness.

[1528] The "means for adjusting a response based on the results of the emotion recognition means" is software or an algorithm for utilizing the results of the emotion recognition means to generate a response that is adapted to the emotional state of the user.

[1529] A "means for visually displaying and audibly transmitting responses using character avatars" is a device or software that uses animation or graphic characters to display and communicate information audibly.

[1530] To implement the present invention, the following system configuration and procedure are used.

[1531] System Configuration

[1532] 1. A means of accepting voice input

[1533] The user inputs voice using a smartphone application, and the smartphone's microphone records the voice and stores it as digital data.

[1534] 2. A means of analyzing the received voice input

[1535] This digital voice data is sent via a smartphone application over the internet to a server, which then uses a speech recognition engine such as Google Cloud Speech-to-Text to analyze the voice data and convert it into natural language text.

[1536] 3. A means of generating a response based on the analysis results

[1537] The server uses the parsed text data as input for a generative AI model (e.g., OpenAI API) that generates an appropriate response based on the user's question or request.

[1538] 4. A means of sending the generated response to the user terminal

[1539] The generated response is forwarded from the server to the user's smartphone.

[1540] 5. Means for displaying and audibly transmitting responses on the user terminal

[1541] The user's smartphone displays the received response in visual and audio formats, with a character avatar responding with audio and displaying text information on the screen for easy visual understanding by the user.

[1542] 6. Emotion recognition means

[1543] The server uses emotion recognition engines such as IBM Watson Tone Analyzer to determine the user's emotions based on the user's voice and text data.

[1544] 7. A means of adjusting responses based on the results of emotion recognition measures

[1545] Based on the results of emotion recognition, a generative AI model generates a tailored response that is adapted to the user's emotional state.

[1546] 8. A means of visually displaying and audibly issuing responses using character avatars

[1547] A character avatar within the smartphone application provides visual and audio information to the user.

[1548] Process Overview

[1549] The smartphone application records the user's voice and sends it to a server. The server converts the voice into text and passes it to a generative AI model. The generative AI model generates a response based on the text, and an emotion recognition engine analyzes the user's emotions. This information is combined to generate an appropriate response and send it to the user's device. A character avatar provides visual and audio responses to the user, improving the user experience.

[1550] Specific examples

[1551] For example, if a user asks "Which shoes are the most popular in this store?" in a store, the system works as follows: It records the user's voice input and sends it over the internet to a server. The server uses a speech recognition engine to convert the speech to text and passes that text as input to a generative AI model. The AI ​​model researches the "most popular shoes" and generates text to return the results to the user.

[1552] The generated text is something like, "The most popular shoe right now is the XYZ model. We're especially offering a 25% off campaign." If the emotion recognition engine detects that the user is excited, it adds an emphasis phrase like, "We highly recommend it!"

[1553] The character avatar then vocalizes the response and visually displays a corresponding image and description of the shoe. If the user asks a follow-up question, such as "Do you have any other colors?", the same technical steps are taken to generate the answer.

[1554] Prompt Sentence Examples

[1555] 1. Voice input: "What are the most popular shoes in this store?"

[1556] 2. Prompt the generative AI model: "Which shoes are the most popular in this store? User emotion: Excitement"

[1557] This provides real-time shopping support that takes into account the user's emotions.

[1558] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1559] Step 1:

[1560] The user launches the smartphone application and enters a question or request by voice.

[1561] Input: Speech data (e.g., "Which shoes are the most popular in this store?")

[1562] Output: Digital audio data

[1563] How it works: The smartphone's microphone records the user's voice, and the application converts the voice data into a digital format and stores it.

[1564] Step 2:

[1565] The digital audio data recorded by the device is sent to the server.

[1566] Input: Digital audio data

[1567] Output: Audio data sent to the server

[1568] Specific operation: The smartphone application sends voice data to a server via the Internet.

[1569] Step 3:

[1570] The voice data received by the server is analyzed using a voice recognition engine and converted into natural language text.

[1571] Input: Audio data

[1572] Output: Natural language text (e.g., "Which shoes are the most popular in this store?")

[1573] How it works: The server uses a speech recognition engine such as Google Cloud Speech-to-Text to analyze the voice data and convert it into natural language text.

[1574] Step 4:

[1575] The server passes the converted text to a generative AI model, which generates a corresponding response.

[1576] Input: Natural language text

[1577] Output: Response text (e.g., "The most popular shoe in our store right now is the XYZ model.")

[1578] Specific operation: The server passes text data as input to a generative AI model (e.g., OpenAI API) and generates a response appropriate to the user's question.

[1579] Step 5:

[1580] The server uses an emotion recognition engine to analyze emotions from the user's voice input and text data.

[1581] Input: Audio or text data

[1582] Output: Emotion data (e.g., "excited" or "frustrated")

[1583] Specific operation: The server uses an emotion recognition engine such as IBM Watson Tone Analyzer to recognize the user's emotions.

[1584] Step 6:

[1585] The server adjusts the generated responses based on the emotion data to generate a final response.

[1586] Input: Response text, emotion data

[1587] Output: Tailored response text (e.g., "The most popular shoe in our store right now is the XYZ model! Highly recommended!")

[1588] How it works: The generative AI model takes in emotion data and adds appropriate expressions based on the user's emotional state.

[1589] Step 7:

[1590] The server sends the adjusted response to the user terminal.

[1591] Input: Tailored response text

[1592] Output: Response text sent to the user's terminal

[1593] Specific operation: The server sends the generated response message to the user's smartphone via the Internet.

[1594] Step 8:

[1595] The user's smartphone displays visual and audio responses using a character avatar.

[1596] Input: Response text

[1597] Output: Audio response and visual display

[1598] Specific operation: A character avatar within the smartphone application will vocalize the generated response and display text information and images of related products on the screen.

[1599] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1600] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1601] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1602] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1603] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1604] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1605] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1606] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1607] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1608] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1609] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1610] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1611] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1612] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1613] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1614] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1615] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1616] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1617] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1618] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1619] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1620] The following is further disclosed regarding the above embodiment.

[1621] (Claim 1)

[1622] means for accepting voice input;

[1623] means for analyzing the received voice input;

[1624] means for generating a response based on the parsed results;

[1625] means for transmitting the generated response to the user terminal;

[1626] The system includes means for displaying and vocalizing the response at the user terminal.

[1627] (Claim 2)

[1628] 10. The system of claim 1, further comprising means for retrieving specific product information based on the analysis results and including the information in the generated response.

[1629] (Claim 3)

[1630] 2. The system according to claim 1, further comprising means for accepting an additional question from the user and for performing further analysis and generating a response based on the question.

[1631] "Example 1"

[1632] (Claim 1)

[1633] means for accepting voice input;

[1634] means for recording the received voice input and transmitting the recorded voice input as digital voice data to a server;

[1635] means for converting received voice data into text data;

[1636] A means of analyzing user intent using a generative AI model; and

[1637] a means for generating queries based on the analysis and retrieving information from the database;

[1638] means for generating a response message based on the retrieved information and transmitting the response message to the user terminal;

[1639] The system includes means for audibly and visually displaying a response message at the user terminal.

[1640] (Claim 2)

[1641] 10. The system of claim 1, further comprising means for retrieving specific product information based on the analysis results and including the information in the generated response.

[1642] (Claim 3)

[1643] 2. The system according to claim 1, further comprising means for accepting an additional question from the user and for performing further analysis and generating a response based on the question.

[1644] "Application Example 1"

[1645] (Claim 1)

[1646] means for accepting voice input;

[1647] means for analyzing the received voice input;

[1648] means for generating a response based on the parsed results;

[1649] means for transmitting the generated response to the user terminal;

[1650] means for displaying and vocalizing a response at the user terminal;

[1651] a means for generating a response based on a particular question using a generative AI model;

[1652] means for converting the generated response from text to speech;

[1653] A means of converting voice data into text data using a voice recognition engine

[1654] A system including:

[1655] (Claim 2)

[1656] 10. The system of claim 1, further comprising means for retrieving specific product information based on the analysis results and including the information in the generated response.

[1657] (Claim 3)

[1658] 2. The system according to claim 1, further comprising means for accepting an additional question from the user and for performing further analysis and generating a response based on the question.

[1659] "Example 2: Combining Emotion Engines"

[1660] (Claim 1)

[1661] means for accepting voice input;

[1662] means for recording the received voice input as digital voice data;

[1663] A means for transmitting the recorded audio data to a server;

[1664] A means for passing the voice data to a speech recognition engine and converting it into natural language text;

[1665] A means for passing the converted text data to a generative AI model and analyzing the content of the text;

[1666] means for generating a response based on the analysis results;

[1667] means for recognizing a user's emotion and adjusting a response message based on the recognition result;

[1668] means for transmitting the generated response to the user terminal;

[1669] The system includes means for displaying and vocalizing the response at the user terminal.

[1670] (Claim 2)

[1671] 10. The system of claim 1, further comprising means for retrieving specific product information based on the analysis results and including the information in the generated response.

[1672] (Claim 3)

[1673] 2. The system according to claim 1, further comprising means for accepting an additional question from the user and for performing further analysis and generating a response based on the question.

[1674] "Application example 2 when combining emotion engines"

[1675] (Claim 1)

[1676] means for accepting voice input;

[1677] means for analyzing the received voice input;

[1678] means for generating a response based on the parsed results;

[1679] means for transmitting the generated response to the user terminal;

[1680] means for displaying and vocalizing a response at the user terminal;

[1681] emotion recognition means for recognizing emotions;

[1682] means for adjusting a response based on the results of the emotion recognition means;

[1683] The system includes means for visually displaying and audibly issuing responses using a character avatar.

[1684] (Claim 2)

[1685] 10. The system of claim 1, further comprising means for retrieving specific product information based on the analysis results and including the information in the generated response.

[1686] (Claim 3)

[1687] 2. The system according to claim 1, further comprising means for accepting an additional question from the user and for performing further analysis and generating a response based on the question. [Explanation of symbols]

[1688] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for accepting voice input; means for analyzing the received voice input; means for generating a response based on the parsed results; means for transmitting the generated response to the user terminal; The system includes means for displaying and vocalizing the response at the user terminal.

2. 10. The system of claim 1, further comprising means for retrieving specific product information based on the analysis results and including that information in the generated response.

3. 2. The system according to claim 1, further comprising means for accepting an additional question from the user and for performing further analysis and generating a response based on the question.

Images