System

Abstract

A system is provided.SOLUTION: The interactive guide system includes a means for detecting the approach of a user by a sensor, a means for receiving the input of the user, a means for analyzing the received input of the user, a generation means for generating a response message on the basis of an analysis result, and a display means for displaying the generated response message to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Public facilities, commercial facilities, and event facilities face the challenge of providing visitors with the information they need quickly and accurately. Conventional staffed guidance systems are limited in scope due to limited human resources, making it difficult to adequately respond to visitors' questions and inquiries, especially during busy times and events. This can result in increased visitor dissatisfaction and confusion, potentially lowering the facility's overall reputation. The objective of the present invention is to provide a system that solves these challenges and improves the user experience by providing visitors with information quickly and accurately. [Means for solving the problem]

[0005] The present invention provides an interactive guidance system that includes a means for detecting the approach of a user using a sensor, a means for receiving user input, a means for analyzing the received user input, a means for generating a response message based on the analysis results, and a means for displaying the generated response message to the user. The system also includes a speech recognition means for converting the user's voice input into text data and a search means for searching for information from a database based on the analyzed user input. This system automatically detects visitors using the sensor and can quickly respond to their questions and inquiries through speech recognition and natural language processing. This makes it possible to overcome human resource constraints and increase visitor satisfaction.

[0006] "Proximity sensing means" refers to a physical or electronic device that senses the presence of a user when they approach the system.

[0007] "Means for receiving input" refers to a device or system for obtaining voice or text questions or instructions from a user.

[0008] "Means for parsing" refers to the algorithms or programs used to understand received user input and generate an appropriate response.

[0009] The "generation means" refers to a device or program for creating an appropriate response message for the user based on the analyzed content.

[0010] "Display means" refers to a device or interface for visually or audibly conveying the generated response message to the user.

[0011] "Speech recognition means" refers to a device or program for converting a user's voice input into text data in real time.

[0012] "Search tool" refers to a program or algorithm for extracting relevant information from a database based on analyzed user input. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention provides an interactive guidance system that quickly and accurately provides information to users in public facilities, commercial facilities, and event facilities. This system automatically detects when a user approaches and can provide optimal responses to the user's questions or requests. Specific embodiments are described in detail below.

[0035] Reaction when the user approaches

[0036] 1. The device detects the user using a sensor

[0037] The digital guidance system is equipped with motion sensors that automatically detect when a user approaches.

[0038] When the sensor detects the presence of a user, the device switches from standby mode to active mode.

[0039] 2. The device launches the avatar

[0040] When the device switches to active mode, the avatar displayed on the device screen comes to life and greets the user.

[0041] For example, the avatar might say, "Hello, is there anything I can help you with?"

[0042] The process that receives user input

[0043] 1. The user asks a question

[0044] Users can use voice input or a touch panel to ask questions or make requests. For example, they can ask, "Please tell me today's event schedule."

[0045] 2. The device gets the user's input

[0046] The device is equipped with a microphone and touch sensor, through which the user's voice input and touch operations are acquired.

[0047] In the case of voice input, the voice is converted into text data in real time by a voice recognition engine.

[0048] Server parsing and generation

[0049] 1. Parse the input

[0050] The device transmits the acquired voice and text data to the server.

[0051] The server uses a natural language processing engine to analyze the user's intent. For example, it analyzes text data such as "Please tell me the event schedule" and identifies the information the user wants to know.

[0052] 2. Generate a response message

[0053] Based on the analysis results, the server uses generation AI to create the optimal response message.

[0054] For example, information about the event schedule may be retrieved from a database and a response message such as "Today's event schedule is as follows..." may be generated.

[0055] Response display by terminal

[0056] 1. Display a response message

[0057] The server generates a response message and sends it to the terminal.

[0058] The device's avatar will convey the response message to the user via voice or text.

[0059] For example, "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00."

[0060] Specific examples

[0061] For example, imagine a user standing in front of a digital information system installed at the entrance of a department store. As the user approaches, the device's sensors detect the user and an avatar speaks, saying, "Hello, is there anything I can help you with?" If the user responds, "Can you tell me where the restaurant is?", the device converts the speech into text and sends it to the server. The server analyzes this request, retrieves the restaurant's location information from a database, and generates an optimal guidance message. The generated message is sent to the device, and the avatar responds, "The restaurant is on the third floor, at the back on the right."

[0062] In this way, the interactive guide system of the present invention allows users to quickly and accurately obtain the information they need, which can greatly improve visitor satisfaction.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] The device detects the user with a sensor. When a user approaches the digital signage, the motion sensor detects their presence, causing the device to switch from standby mode to active mode.

[0066] Step 2:

[0067] The device activates the avatar. When the sensor detects the user's approach, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[0068] Step 3:

[0069] The user asks a question. The user uses voice input or a touch panel to ask a question or make a request. For example, the user might ask, "Please tell me today's event schedule."

[0070] Step 4:

[0071] The device receives user input. The device's microphone or touch sensor captures the user's question. In the case of voice input, the speech recognition engine converts the voice into text in real time.

[0072] Step 5:

[0073] The terminal sends the text data to the server. The acquired text data is then sent from the terminal to the server.

[0074] Step 6:

[0075] The server analyzes the input. The server's natural language processing engine analyzes the text data sent and understands the user's intent. For example, it recognizes a request such as "Please tell me the event schedule."

[0076] Step 7:

[0077] The server retrieves the relevant data. Based on the analysis results, the server searches and retrieves relevant information from the database. In this case, it retrieves the event schedule information.

[0078] Step 8:

[0079] The server generates a response message. The AI ​​then creates the optimal response message based on the acquired information. For example, it generates a message such as, "Today's event schedule is a yoga class from 10:00 and a concert from 14:00."

[0080] Step 9:

[0081] The server sends a response message to the terminal. The generated response message is sent from the server to the terminal.

[0082] Step 10:

[0083] The device displays the response to the user. The device's avatar conveys the received response message to the user via voice or text. For example, it may display "Today's event schedule is as follows" and provide details via voice.

[0084] Step 11:

[0085] The user asks a follow-up question. If the user wants to ask more questions, the same process is used to fulfill the new request. For example, "Can you also tell me where the restaurant is?"

[0086] By performing the above steps, the interactive guidance system of the present invention can quickly and accurately respond to user questions.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] Conventional guidance systems have the problem of being unable to detect the user's approach and therefore unable to provide the necessary information quickly. Furthermore, they lack an adequate interface for generating appropriate responses to user input, resulting in low user satisfaction. Furthermore, conventional systems have the problem of being unable to provide the appropriate information the user is looking for due to the low accuracy of voice input and inaccurate analysis results. To solve these problems, the development of an interactive and accurate guidance system is required.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes means for detecting the approach of a user with a sensor, means for switching from standby mode to active mode when the user is detected, means for activating an avatar in active mode and displaying a greeting, means for receiving user input, means for converting the received user input into text data with a voice recognition engine, means for transmitting the text data to the server and analyzing it, means for generating an appropriate response message based on the analysis result using a generative AI model, and means for displaying the generated response message to the user in voice and text. This makes it possible to quickly detect when a user is approaching and provide an appropriate response, thereby improving user satisfaction.

[0092] A "means for detecting user proximity by a sensor" is a device for detecting when a user enters the range of the system.

[0093] "Standby mode" refers to a state in which the system is in a low-power standby state and is monitoring external inputs.

[0094] "Active mode" is a mode in which the system goes into full operation after detecting the approach of a user.

[0095] The "means for activating an avatar and displaying a greeting" refers to a means for displaying an anthropomorphized character on the system screen and greeting the user by voice or text.

[0096] The "means for receiving user input" is a device for receiving questions and instructions from the user through voice input or a touch panel.

[0097] The "means for converting into text data using a voice recognition engine" is software that analyzes the user's voice input in real time and converts it into text data.

[0098] The "means for transmitting text data to a server" is a function for transmitting text data acquired from a user to a server via the Internet or the like.

[0099] The "analysis means" is software that allows the server to analyze text data using a natural language processing engine and understand the user's intent.

[0100] "Means for generating using a generative AI model" refers to the process of using generative AI to automatically create an optimal response message based on the analysis results.

[0101] The "means for displaying the generated response message to the user in voice and text" refers to a means for displaying the response message generated by the server on the terminal and conveying it to the user in voice and text through the avatar.

[0102] The present invention is an interactive guidance system that provides users with information quickly and accurately in public, commercial, and event facilities. The system automatically detects when a user approaches and provides optimal responses to the user's questions and requests.

[0103] Hardware and software used

[0104] The following hardware and software are used to implement the system:

[0105] Motion sensor: Used to detect the user's approach.

[0106] Microphone: Used to receive audio input.

[0107] Touch panel: Used to receive user touch input.

[0108] Display: Used for displaying avatars and text.

[0109] Server: Parses the data and generates a response message.

[0110] Speech recognition engine: Converts the user's speech into text.

[0111] Natural Language Processing (NLP) engine: Parses user input.

[0112] Generative AI model: Generates optimal response messages based on analysis results.

[0113] Program processing explanation

[0114] The operation of the system is as follows.

[0115] 1. User approach detection

[0116] The device uses a motion sensor to detect when the user enters a certain range, which causes the device to switch from standby mode to active mode.

[0117] 2. Launching the avatar

[0118] When in active mode, an avatar appears on the device's display and greets the user, for example, "Hello, how can I help you?"

[0119] 3. Accepting User Input

[0120] The user can ask questions or make requests using voice input or a touch panel. For example, they can input, "Please tell me today's event schedule."

[0121] 4. Acquiring input information

[0122] User input is obtained through the device's microphone or touch sensor. In the case of voice input, the voice recognition engine converts it into text data.

[0123] 5. Data Analysis

[0124] The device sends the acquired text data to the server, which then uses a natural language processing engine to analyze the data and understand the user's intent. For example, the server might analyze a request such as "Please tell me the event schedule."

[0125] 6. Generating a Response Message

[0126] The server uses the generative AI model to generate an appropriate response message based on the analysis results, and the generated message is sent from the server to the device.

[0127] 7. Displaying the Response

[0128] The device displays the received response message to the user in voice and text. For example, an avatar may say, "Today's event schedule is a yoga class from 10:00 and a concert from 14:00."

[0129] Adding specific examples

[0130] For example, when a user stands in front of a digital information system installed at the entrance of a department store, the device's sensors detect the user, and an avatar on the screen speaks to them, saying, "Hello, is there something I can help you with?" If the user says, "Can you tell me where the restaurant is?", the device converts the speech into text and sends it to the server. The server analyzes the request, retrieves restaurant information from a database, and generates a response message, stating, "The restaurant is on the third floor, at the back on the right."

[0131] Prompt Sentence Examples

[0132] An example of an input prompt for a generative AI model might be, "Please provide a clear and easy-to-understand answer to the question asked by the user." Based on this prompt, the generative AI model generates the optimal response message.

[0133] This system therefore provides users with quick and accurate information, improving satisfaction with facility use.

[0134] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0135] Step 1:

[0136] User approach detection

[0137] The device detects the user using a sensor

[0138] Input: The motion sensor detects the user's movement.

[0139] Specific operation: The motion sensor detects the user's position and movement and sends a signal to the device.

[0140] Output: A signal is generated that switches the terminal from standby mode to active mode.

[0141] Step 2:

[0142] Launching the Avatar

[0143] The device launches the avatar

[0144] Input: The terminal receives a signal to switch to active mode.

[0145] Specific behavior: An avatar appears on the device display and greets the user using programmed motions.

[0146] Output: The avatar will say and read "Hello, how can I help you?"

[0147] Step 3:

[0148] Accepting user input

[0149] The user enters a question or request

[0150] Input: The user enters a question or request using voice input or a touch panel.

[0151] Specific operation: The user asks a question such as "Please tell me today's event schedule" using the microphone or touch panel.

[0152] Output: User voice or touch data is captured.

[0153] Step 4:

[0154] Obtaining input information

[0155] The device obtains the user's input information.

[0156] Input: For voice input, the user's voice data. For touch input, the touch operation data.

[0157] How it works: A highly sensitive microphone picks up voices, and a voice recognition engine converts them into text data. Touch input data is also sent from the sensor.

[0158] Output: User input data in text format is generated.

[0159] Step 5:

[0160] Data analysis

[0161] The device sends the acquired data to the server.

[0162] Input: User-entered data in text format.

[0163] Specific operation: The device sends data to the server in real time.

[0164] Output: The server receives the user's text input data.

[0165] The server analyzes the data

[0166] Input: The text data sent to the server.

[0167] Specific operation: The natural language processing engine analyzes text data and understands the user's intent. For example, it analyzes a request such as "Please tell me the event schedule."

[0168] Output: Parsed user intent and required information identified.

[0169] Step 6:

[0170] Generating a response message

[0171] The server generates a response message

[0172] Input: Parsed user intent.

[0173] What it does: Uses a generative AI model to create an optimal response message, retrieving relevant information from the necessary databases.

[0174] Output: The generated response message is formed.

[0175] Step 7:

[0176] Viewing the response

[0177] The server sends a response message to the device.

[0178] Input: The generated response message.

[0179] Specific operation: The server sends the generated response message to the terminal.

[0180] Output: The terminal receives the response message.

[0181] The terminal displays a response message to the user.

[0182] Input: The response message received from the server.

[0183] Specific operation: The device's avatar conveys the received message to the user via voice and text. For example, the avatar may display and speak, "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00."

[0184] Output: The user receives the necessary information visually and audibly.

[0185] (Application example 1)

[0186] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0187] Conventional guidance systems often struggle to respond quickly and accurately to user questions. In particular, when it comes to effectively utilizing voice input, the user experience is often compromised by inaccurate voice recognition technology and natural language processing technology that cannot accurately analyze user intent. Furthermore, responses to user voice input are monotonous, resulting in insufficient user interaction, making it difficult to improve visitor satisfaction.

[0188] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0189] In this invention, the server includes means for detecting the approach of a user with a sensor, means for receiving user input by voice, means for converting the received user voice into text data, means for analyzing the text data, means for generating a response message using a generative AI model based on the analysis results, and means for providing the generated response message to the user in voice and text, thereby enabling the server to provide quick and accurate responses to user questions and requests and realizing an interactive user experience.

[0190] The "means for detecting the approach of a user by a sensor" refers to a device or function for detecting the presence of a user when the user approaches the system.

[0191] The "means for receiving user input by voice" refers to a device or function for acquiring voice data provided by the user.

[0192] The "means for converting received user voice into text data" refers to a process or device for analyzing the user voice data and converting it into text format data.

[0193] "Means for analyzing text data" refers to software or algorithms that, after speech input has been converted into text data, analyze the text data and understand the user's intent.

[0194] A "generative means using a generative AI model" is an artificial intelligence model or algorithm for generating an appropriate response message based on the analyzed user text data.

[0195] A "means for providing a generated response message to a user in both audio and text" is a device or process for providing a generated response message to a user in both audio and text form.

[0196] The present invention provides an interactive guidance system that enables users to quickly and accurately obtain information within a virtual environment. Specific embodiments will be described below.

[0197] System configuration

[0198] The system consists of the following main components:

[0199] 1. How to use a sensor to detect the user's approach:

[0200] A motion sensor is used to detect when a user approaches the system, readying the system for user interaction.

[0201] 2. A way to receive user input by voice:

[0202] A microphone is used to capture the user's voice input, which is then analyzed.

[0203] 3. A means of converting received user speech into text data:

[0204] Use a speech recognition API (for example, Google Speech Recognition API) to convert the acquired speech data into text data in real time.

[0205] 4. Means of analyzing text data:

[0206] The converted text data is then analyzed by a natural language processing engine and processed to understand the user's intent. Specifically, a Transformer-based generative AI model (e.g., GPT-2) is used.

[0207] 5. Generative AI model generation method:

[0208] Based on the analysis results, an appropriate response message is generated, which may involve retrieving information from a database to provide the best possible answer to the user's question or request.

[0209] 6. Means for providing the generated response message to the user in voice and text:

[0210] It uses Google Text-to-Speech (gTTS) to convert the generated text response into audio, providing it to the user in both audio and text form, and conveying information both visually and audibly.

[0211] Data Calculation and Processing

[0212] 1. Acquire and convert voice input:

[0213] The user speaks a question to the system, the microphone captures the voice signal, and the data is converted into text data through a speech recognition API.

[0214] 2. Text data analysis:

[0215] The converted text data is sent to a server, where a natural language processing engine analyzes it to understand the user's intent. This analysis uses the GPT-2 model.

[0216] 3. Generate a response message:

[0217] The generative AI model generates the optimal response message that matches the analyzed data. During this generation process, it searches and retrieves the necessary information from the database and determines the content to be provided to the user.

[0218] 4. Providing generated messages:

[0219] The generated response message is converted from text data to voice data again and provided to the user. At the same time, it is also displayed in text format, so the user can receive information in both voice and text.

[0220] Specific examples

[0221] Consider a scenario in which a user connects to a virtual store through smart glasses and inquires about product availability. When the user verbally asks, "Is this product in stock?", the system captures voice data through a microphone and converts the speech into text using the Google Speech Recognition API. The GPT-2 model then analyzes the text data and retrieves the corresponding inventory information from the inventory database. The generated response message, "This product is in stock," is provided to the user in both voice and text.

[0222] Prompt Sentence Examples

[0223] example:

[0224] "Hello, are you looking for something?"

[0225] "What is the most popular item in this store?"

[0226] "Please let me know about this week's sales."

[0227] This system allows users to obtain fast and accurate information even in virtual environments, improving the overall user experience.

[0228] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0229] Step 1:

[0230] A user approaches the system. The device's motion sensor detects the user and the system switches from standby mode to active mode.

[0231] Input: User approach

[0232] Output: Switching the system to active mode

[0233] What happens: The motion sensor reacts, an internal signal from the device triggers the system's active mode, and an avatar appears on the screen and greets the user.

[0234] Step 2:

[0235] The user asks a question by voice, and the device's microphone picks up the voice and captures the audio signal.

[0236] Input: User's voice question

[0237] Output: Audio data

[0238] What it does: A user speaks into a microphone asking a question, such as "Do you have this product in stock?" The microphone picks up the voice signal and stores it as digital audio data.

[0239] Step 3:

[0240] The voice data acquired by the device is converted into text data via the Google Speech Recognition API.

[0241] Input: Audio data

[0242] Output: Text data

[0243] How it works: Audio data is sent to a cloud service, where it is analyzed by the Google Speech Recognition API and converted into corresponding text data.

[0244] Step 4:

[0245] The server receives the text data and analyzes it using a natural language processing engine (GPT-2), which searches a database to understand the user's intent and determine the appropriate response.

[0246] Input: Text data

[0247] Output: Analysis results and generation of optimal response messages

[0248] How it works: The server inputs text data into the GPT-2 model and begins analysis. During the analysis process, the user's question (e.g., "Is it in stock?") is understood and relevant information (e.g., "inventory data") is searched for in the database.

[0249] Step 5:

[0250] The server uses the generative AI model to generate a response message based on the analysis results.

[0251] Input: Analysis results and information obtained from databases

[0252] Output: Response message (text format)

[0253] Specific behavior: The GPT-2 model generates a text message such as "This product is in stock." The generated message is stored as internal data.

[0254] Step 6:

[0255] The device uses Google Text-to-Speech (gTTS) to convert the generated text message into audio data and provide it to the user.

[0256] Input: Text message

[0257] Output: Audio data and text display

[0258] How it works: A text message is entered into gTTS and converted into voice data, which is then played back through the device's speaker and simultaneously displayed on the screen.

[0259] Step 7:

[0260] The user reviews the response message and, if necessary, asks additional questions or provides instructions, and the cycle repeats.

[0261] Input: User confirmation and follow-up questions

[0262] Output: Additional audio data

[0263] What happens: The user listens to the voice response or reads the text, then speaks again if they have any further questions. This process can be repeated as necessary.

[0264] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0265] The present invention provides an interactive guidance system that not only provides users with information quickly and accurately in public facilities, commercial facilities, and event facilities, but also recognizes the user's emotions to provide more personalized responses. This system automatically detects when a user approaches and can provide optimal responses to the user's questions or requests. Furthermore, it utilizes an emotion engine to recognize the user's emotions and respond based on those emotions. Specific embodiments are described in detail below.

[0266] Basic operation

[0267] Reaction when the user approaches

[0268] 1. The device detects the user using a sensor

[0269] The digital guidance system is equipped with motion sensors that automatically detect when a user approaches.

[0270] When the sensor detects the presence of a user, the device switches from standby mode to active mode.

[0271] 2. The device launches the avatar

[0272] When the device switches to active mode, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[0273] The process that receives user input

[0274] 1. The user asks a question

[0275] The user can use voice input or a touch panel to ask questions or make requests, for example, "Please tell me today's event schedule."

[0276] 2. The device gets the user's input

[0277] The device is equipped with a microphone and touch sensor, through which the user's voice input and touch operations are acquired.

[0278] In the case of voice input, the voice is converted into text data in real time by a voice recognition engine.

[0279] Server parsing and generation

[0280] 1. Parse the input

[0281] The device transmits the acquired voice and text data to the server.

[0282] The server uses a natural language processing engine and an emotion engine to analyze the user's intent and emotions. For example, it analyzes the text data "Please tell me the event schedule" as well as the user's tone of voice and facial expressions to identify their emotional state.

[0283] 2. Generate a response message

[0284] Based on the analysis results, the server uses AI to create the optimal response message. Depending on the user's emotional state, a message that takes into account their emotions, such as "Today's event schedule is as follows. Have a great day!", is generated.

[0285] Response display by terminal

[0286] 1. Display a response message

[0287] The server generates a response message and sends it to the terminal.

[0288] The device's avatar will respond to the user with a voice or text message. For example, it might say, "Today's event schedule is as follows," and provide details in voice.

[0289] Specific examples

[0290] For example, imagine a user standing in front of a digital information system installed at the entrance of a department store. As the user approaches, the device's sensors detect the user's approach and an avatar speaks to them, saying, "Hello, is there anything I can help you with?" If the user responds, "Can you tell me where the restaurant is?", the device converts the speech into text and sends that data to the server. The server analyzes this request and, using an emotion engine, infers, for example, that the user is in a hurry based on the user's tone of voice and facial expression. As a result, it generates a specific, situation-appropriate guidance message: "It seems you're in a hurry. The restaurant is on the third floor, at the back on the right." The generated message is sent to the device, and the avatar relays it to the user.

[0291] In this way, the interactive guidance system of the present invention allows users to quickly and accurately obtain the information they need. Furthermore, the introduction of an emotion engine enables personalized responses that take into account the user's emotions, greatly improving user satisfaction.

[0292] The processing flow will be explained below.

[0293] Step 1:

[0294] The device detects the user with a sensor. When a user approaches the digital signage, the motion sensor detects their presence, causing the device to switch from standby mode to active mode.

[0295] Step 2:

[0296] The device activates the avatar. When the sensor detects the user's approach, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[0297] Step 3:

[0298] The user asks a question. The user uses voice input or a touch panel to ask a question or make a request. For example, the user might ask, "Please tell me today's event schedule."

[0299] Step 4:

[0300] The device receives user input. The device's microphone or touch sensor captures the user's question. In the case of voice input, the speech recognition engine converts the voice into text in real time.

[0301] Step 5:

[0302] The device activates an emotion engine to analyze the user's emotions, analyzing voice tone, speaking style, facial expressions, etc. to determine whether the user is angry, in a hurry, having fun, etc.

[0303] Step 6:

[0304] The device sends text data and emotion data to the server, which then sends the acquired text data and emotion analysis results together.

[0305] Step 7:

[0306] The server analyzes the input. The server's natural language processing engine analyzes the text data sent and understands the user's intent. For example, it recognizes a request such as "Please tell me the event schedule."

[0307] Step 8:

[0308] The server generates a response message based on the emotional data. Based on the analysis results, the server uses generation AI to create an appropriate response message. Depending on the user's emotional state, a message that takes emotions into consideration is generated, such as "Today's event schedule is as follows. Have a great day!"

[0309] Step 9:

[0310] The server sends a response message to the terminal. The server then sends the generated response message to the terminal. If the user is in a hurry, it generates a concise and quick message, but if the user is relaxed, it generates a detailed and polite message.

[0311] Step 10:

[0312] The device displays the response to the user. The device's avatar then relays the received response message to the user via voice or text. For example, it might display "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00," and provide details via voice.

[0313] Step 11:

[0314] The user asks a follow-up question. If the user wants to ask more questions, the same process is used to respond to the new request. For example, answer a question like, "Can you also tell me where the restaurant is?". Even for follow-up questions, sentiment analysis is performed and an appropriate message is generated and displayed.

[0315] By implementing the above steps, the interactive guidance system of the present invention can quickly and accurately respond to user questions. Furthermore, the introduction of an emotion engine enables personalized responses that take into account the user's emotions, greatly improving user satisfaction.

[0316] Example 2

[0317] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0318] Conventional guidance systems have difficulty in providing quick and accurate information to users' questions, and are unable to provide personalized responses that reflect the user's emotions. This has led to problems such as lower user satisfaction and a decrease in frequency of use.

[0319] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0320] In this invention, the server includes means for detecting the approach of a user with a sensor, means for receiving user input, means for analyzing the received user input, means for generating a response message based on the analysis result, means for displaying the generated response message to the user, means for recognizing the user's emotion, and means for personalizing the response message based on the recognized emotion. This enables quick and accurate provision of information, and further allows for personalized responses to be provided in accordance with the user's emotion, thereby improving user satisfaction.

[0321] A "sensor" is a device that detects physical changes or movements and captures them as data.

[0322] A "user" is a person or individual who uses the system to obtain information.

[0323] "Approach" refers to the user moving into the system's recognition range.

[0324] "Detecting" means that the sensor recognizes the user's approach and sends that information to the system.

[0325] "Means" is a general term for methods or devices used to achieve a goal.

[0326] "Receiving" means that the system takes input from the user and processes it.

[0327] "Input" refers to the information or instructions a user provides to a system, and can take the form of voice, touch, or other input.

[0328] "Analyzing" means analyzing the received input data and identifying the user's intentions and the information they are looking for.

[0329] "Generating" means creating a new response message based on the analysis results.

[0330] A "response message" is information or a response that the system provides to the user, and may be in the form of text or audio.

[0331] "Displaying" means conveying the generated response message to the user on the screen or by voice.

[0332] "Emotion" refers to the user's psychological state or mood, and is recognized by the system.

[0333] "Recognizing" means that the system captures the user's emotions and uses them for analysis.

[0334] "Personalization" means customizing a response message according to the user's emotions and individual circumstances.

[0335] The present invention is an interactive guidance system that not only provides users with information quickly and accurately in public facilities, commercial facilities, and event facilities, but also recognizes the user's emotions to provide more personalized responses. This system has the function of automatically detecting when a user approaches and providing the most appropriate response to the user's questions or requests.

[0336] First, the device uses sensors to detect the user's approach. Specifically, a motion sensor detects the user's movement, and when the user comes within a certain distance, the system switches to active mode. Then, an avatar displayed on the device speaks to the user, saying, "Hello, is there anything I can help you with?" The avatar is programmed using game engines such as Unity or Unreal Engine.

[0337] Users can ask questions or make requests using voice input or the touch panel. For example, if they ask, "What is today's event schedule?", the microphone picks up the voice and converts it into text in real time using the Google Cloud Speech-to-Text API. The text data is saved in a temporary file and then sent to the server.

[0338] The server uses the Google Cloud Natural Language API to analyze the received text data and parses the user's intent. It also uses emotion engines such as Amazon Comprehend to analyze the user's emotional state based on their tone of voice and facial expressions. Emotional information such as whether the user is in a hurry can also be collected.

[0339] Based on the analysis results, a generative AI (e.g., OpenAI GPT-4) generates an optimal response message. The prompt is "Please tell me the event schedule. The user seems to be in a hurry." The message generated by the AI ​​model corresponds to the emotion, such as "Today's event schedule is as follows. Have a nice day."

[0340] The server then sends the generated response message to the device, and the device communicates the content to the user in voice and text. For example, if a user asks, "Where is the restaurant?", the server analyzes the question and generates a message saying, "You seem to be in a hurry. The restaurant is on the third floor, at the back on the right," which the device then communicates to the user.

[0341] This system not only allows users to obtain accurate information quickly, but also provides personalized responses that take their emotions into consideration. For example, by inputting a prompt such as "Please generate a response for how the guidance system should greet the user when they approach," into the AI ​​model, an appropriate response can be generated.

[0342] As described above, the interactive guidance system of the present invention can provide users with a high level of satisfaction.

[0343] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0344] Interactive guidance system processing flow

[0345] Step 1:

[0346] The device uses a sensor to detect the user's approach.

[0347] What it does: Motion sensors detect user movement within 20 feet.

[0348] Input: Motion information from the external environment.

[0349] Data processing: Sensors convert physical movements into digital signals.

[0350] Output: A signal indicating the user's approach.

[0351] Next step: Send a signal to switch from standby mode to active mode.

[0352] Step 2:

[0353] The device switches to active mode and activates the avatar.

[0354] What happens: The device's display turns on and an avatar appears, saying, "Hello, how can I help you?"

[0355] Input: User approach detection signal from sensor.

[0356] Data processing: The system switches from standby mode to active mode. The avatar's program is activated.

[0357] Output: A voice message to the user.

[0358] Next: Wait for user input.

[0359] Step 3:

[0360] The user inputs a question by voice or via a touch panel.

[0361] Specific behavior: The user asks "What is today's event schedule?" by voice or by entering text using the touch panel.

[0362] Input: User voice or touch input data.

[0363] Data processing: A microphone and touch sensor capture user input and convert it into digital data. Voice input is converted to text using the Google Cloud Speech-to-Text API.

[0364] Output: User questions in text format.

[0365] Next process: The terminal prepares to send text data to the server.

[0366] Step 4:

[0367] The terminal sends the user's input to the server.

[0368] Specific operation: The acquired text data is sent to the server using the HTTPS protocol.

[0369] Input: The user's question in text form.

[0370] Data processing: The text data is divided into packets and sent to the server over a secure communication channel.

[0371] Output: User question data sent to the server.

[0372] Next: The server parses the user's input data.

[0373] Step 5:

[0374] The server parses the input data.

[0375] What it does: It uses the Google Cloud Natural Language API to analyze the content of the user's question, and Amazon Comprehend to analyze the user's sentiment.

[0376] Input: User question data in text format.

[0377] Data processing: Perform natural language processing and sentiment analysis to identify user intent and emotional state.

[0378] Output: Analysis results including user intent (e.g., event schedule question) and emotional state (e.g., in a hurry).

[0379] Next process: Generate a response message based on the analysis results.

[0380] Step 6:

[0381] The server generates a response message.

[0382] Specific operation: Uses generative AI (e.g., OpenAI GPT-4) to generate a response message based on the analysis results.

[0383] Input: Analysis results, including the user's intent and emotional state.

[0384] Data processing: A prompt is input into the generative AI model to generate the optimal response message. Example prompt: "Generate a response sentence for how the guidance system should greet the user when they approach."

[0385] Output: Personalized response messages depending on the emotion.

[0386] Next: Send a response message to the terminal.

[0387] Step 7:

[0388] The server generates a message and sends it to the terminal.

[0389] Specific operation: Sends a response message to the terminal via the HTTPS protocol.

[0390] Input: The generated response message.

[0391] Data processing: Divide the message into packets and send them securely to the terminal.

[0392] Output: Response message sent to the terminal.

[0393] Next process: The terminal displays and transmits the response message to the user.

[0394] Step 8:

[0395] The terminal displays and transmits the response message.

[0396] Specific operation: The avatar will speak the generated response message to the user, and the text "Today's event schedule is as follows" will be displayed on the screen.

[0397] Input: The response message from the server.

[0398] Data processing: A speech synthesis engine is used to convert text into speech, and a program to move the avatar is executed.

[0399] Output: The response message that is conveyed to the user.

[0400] Next: Wait for further user input if necessary.

[0401] (Application example 2)

[0402] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0403] Conventional interactive guidance systems provide uniform information to users, making it difficult to provide appropriate guidance that reflects each user's individual situation and emotions. In particular, in environments where a variety of information is required, such as brick-and-mortar stores, there is a need to respond quickly and individually to user needs. Furthermore, when a user feels lost or confused, a system that can accurately detect the situation and provide appropriate guidance is needed.

[0404] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0405] In this invention, the server includes means for detecting the approach of a user with a detection device, means for acquiring user input, means for analyzing the acquired user input, means for generating response information based on the analysis result using a generative AI model, display means for providing the generated response information to the user, means for recognizing the user's emotional state, voice recognition means for converting voice input into text data, and means for searching for information from a database based on the analyzed user input. This enables personalized guidance according to the user's emotions, and makes it possible to quickly and accurately provide the information users need even in physical stores.

[0406] A "detection device" is a device that detects the approach of a user, and specifically refers to a motion sensor or camera.

[0407] "User input" refers to information or questions provided by a user, which may be obtained in the form of voice, text, touch, or other input.

[0408] The "analysis means" is a means for understanding and analyzing acquired user input, and utilizes natural language processing and emotion recognition technology.

[0409] A "generative artificial intelligence model" is an artificial intelligence technique for generating appropriate response information based on the results obtained using analytical means.

[0410] "Response information" is information or answers provided in response to user input, and is generated by a generative AI model.

[0411] The "display means" is a means for providing the generated response information to the user, and refers to a display, an audio output device, or the like.

[0412] The "emotional state recognition means" is a means for recognizing the user's emotions from facial expressions, tone of voice, etc., and utilizes an emotion analysis engine or the like.

[0413] The "voice recognition means" is a means for converting voice input into character data, and utilizes a voice recognition engine.

[0414] The "information retrieval means" is a means for retrieving necessary information from a database based on the analyzed user input.

[0415] A "database" is a data storage system that stores necessary information and serves as a source of information for responding to user inquiries.

[0416] The present invention relates to an interactive guidance system that quickly provides appropriate information to users in a physical store. The system includes the following main components:

[0417] Detection device

[0418] The system includes a detection device for detecting the approach of a user. This detection device can be a motion sensor or a camera. For example, the camera detects the user's movement and detects that the user is approaching the system.

[0419] Getting User Input

[0420] User input is acquired in the form of voice, touch operation, and text. Voice input is acquired using a microphone, and touch operation is performed through a touch panel. The acquired voice input is converted into text data by a voice recognition engine.

[0421] Analysis means

[0422] The acquired user input is sent to the server and analyzed by a natural language processing engine and an emotion recognition engine. The natural language processing engine analyzes the input text data and understands the content of the user's request or question. The emotion recognition engine recognizes emotions from the user's tone of voice and facial expressions.

[0423] Response generation using generative artificial intelligence models

[0424] Based on the analysis results, the generative AI model generates optimal response information that is tailored to the user's intent and emotional state and is designed to address their individual needs, such as clearly explaining the solution if the user is confused.

[0425] Display means

[0426] The generated response information is provided to the user via a display and an audio output device. The response is displayed in text format on the display, and if an audio output device is used, audio guidance is also provided.

[0427] Example of a system

[0428] A concrete example of such a system is a physical store guidance system using smart glasses. In this case, facial recognition and emotion analysis are performed while the user wears the glasses and moves around the store. When the user asks, "Where is this item?", a speech recognition engine converts the question into text data, and a generative AI model generates a response such as, "Let's get started right away. Go straight and it's on the next shelf on the right."

[0429] Prompt Sentence Examples

[0430] Examples of prompts to input to a generative AI model include:

[0431] "User's query: 'Where is this item?' Sentiment analysis: 'Confused' Generate the best response."

[0432] In this way, by implementing the interactive guidance system of the present invention, personalized guidance can be provided according to the user's emotions, improving the customer experience in a physical store.

[0433] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0434] Step 1:

[0435] The device detects the user's approach

[0436] Input: User approach status

[0437] Operation: A motion sensor or camera installed on the device detects the user's approach. The camera captures the user's movements and recognizes that the user is approaching the system. Output: The user's presence is detected.

[0438] Step 2:

[0439] The device switches to active mode

[0440] Input: User approach information

[0441] Action: The device switches from standby mode to active mode, and the avatar displayed on the screen comes to life. The avatar speaks to the user, saying, "Hello, how can I help you?" Output: Readiness for user response

[0442] Step 3:

[0443] Getting user input

[0444] Input: User's voice input or touch actions

[0445] Action: The user asks a question or makes a request. For example, "Where is this item?" The device's microphone picks up the voice, or the touch panel receives touch input. Output: The captured user input data.

[0446] Step 4:

[0447] Converting voice input to text data

[0448] Input: User voice input

[0449] Operation: The acquired voice input is converted into text data in real time by the voice recognition engine on the device. Output: Text data converted from voice

[0450] Step 5:

[0451] The server parses the input data

[0452] Input: Acquired text data

[0453] Operation: The server analyzes the text data using a natural language processing engine to understand the user's intention and the content of the question. At the same time, the emotion recognition engine analyzes the user's tone of voice and facial expressions to identify their emotional state. Output: Analyzed user intention and emotional information

[0454] Step 6:

[0455] Generating response information using a generative AI model

[0456] Input: Analyzed user intent and emotion information

[0457] Operation: The server's AI model generates the optimal response information based on the analysis results. For example, if the user is confused, it generates a response such as, "Let's start the guide right away. Go straight and it's on the next shelf on the right." Output: Generated response information

[0458] Step 7:

[0459] Display response information

[0460] Input: Generated response information

[0461] Operation: Provides response information to the user through the terminal's display and audio output device. The response is displayed in text format on the display, and is played back aloud when the audio output device is used. Output: Response information presented to the user

[0462] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0463] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0464] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0465] [Second embodiment]

[0466] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0467] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0468] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0469] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0470] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0471] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0472] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0473] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0474] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0475] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0476] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0477] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0478] The present invention provides an interactive guidance system that quickly and accurately provides information to users in public facilities, commercial facilities, and event facilities. This system automatically detects when a user approaches and can provide optimal responses to the user's questions or requests. Specific embodiments are described in detail below.

[0479] Reaction when the user approaches

[0480] 1. The device detects the user using a sensor

[0481] The digital guidance system is equipped with motion sensors that automatically detect when a user approaches.

[0482] When the sensor detects the presence of a user, the device switches from standby mode to active mode.

[0483] 2. The device launches the avatar

[0484] When the device switches to active mode, the avatar displayed on the device screen comes to life and greets the user.

[0485] For example, the avatar might say, "Hello, is there anything I can help you with?"

[0486] The process that receives user input

[0487] 1. The user asks a question

[0488] Users can use voice input or a touch panel to ask questions or make requests. For example, they can ask, "Please tell me today's event schedule."

[0489] 2. The device gets the user's input

[0490] The device is equipped with a microphone and touch sensor, through which the user's voice input and touch operations are acquired.

[0491] In the case of voice input, the voice is converted into text data in real time by a voice recognition engine.

[0492] Server parsing and generation

[0493] 1. Parse the input

[0494] The device transmits the acquired voice and text data to the server.

[0495] The server uses a natural language processing engine to analyze the user's intent. For example, it analyzes text data such as "Please tell me the event schedule" and identifies the information the user wants to know.

[0496] 2. Generate a response message

[0497] Based on the analysis results, the server uses generation AI to create the optimal response message.

[0498] For example, information about the event schedule may be retrieved from a database and a response message such as "Today's event schedule is as follows..." may be generated.

[0499] Response display by terminal

[0500] 1. Display a response message

[0501] The server generates a response message and sends it to the terminal.

[0502] The device's avatar will convey the response message to the user via voice or text.

[0503] For example, "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00."

[0504] Specific examples

[0505] For example, imagine a user standing in front of a digital information system installed at the entrance of a department store. As the user approaches, the device's sensors detect the user and an avatar speaks, saying, "Hello, is there anything I can help you with?" If the user responds, "Can you tell me where the restaurant is?", the device converts the speech into text and sends it to the server. The server analyzes this request, retrieves the restaurant's location information from a database, and generates an optimal guidance message. The generated message is sent to the device, and the avatar responds, "The restaurant is on the third floor, at the back on the right."

[0506] In this way, the interactive guide system of the present invention allows users to quickly and accurately obtain the information they need, which can greatly improve visitor satisfaction.

[0507] The processing flow will be explained below.

[0508] Step 1:

[0509] The device detects the user with a sensor. When a user approaches the digital signage, the motion sensor detects their presence, causing the device to switch from standby mode to active mode.

[0510] Step 2:

[0511] The device activates the avatar. When the sensor detects the user's approach, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[0512] Step 3:

[0513] The user asks a question. The user uses voice input or a touch panel to ask a question or make a request. For example, the user might ask, "Please tell me today's event schedule."

[0514] Step 4:

[0515] The device receives user input. The device's microphone or touch sensor captures the user's question. In the case of voice input, the speech recognition engine converts the voice into text in real time.

[0516] Step 5:

[0517] The terminal sends the text data to the server. The acquired text data is then sent from the terminal to the server.

[0518] Step 6:

[0519] The server analyzes the input. The server's natural language processing engine analyzes the text data sent and understands the user's intent. For example, it recognizes a request such as "Please tell me the event schedule."

[0520] Step 7:

[0521] The server retrieves the relevant data. Based on the analysis results, the server searches and retrieves relevant information from the database. In this case, it retrieves the event schedule information.

[0522] Step 8:

[0523] The server generates a response message. The AI ​​then creates the optimal response message based on the acquired information. For example, it generates a message such as, "Today's event schedule is a yoga class from 10:00 and a concert from 14:00."

[0524] Step 9:

[0525] The server sends a response message to the terminal. The generated response message is sent from the server to the terminal.

[0526] Step 10:

[0527] The device displays the response to the user. The device's avatar conveys the received response message to the user via voice or text. For example, it may display "Today's event schedule is as follows" and provide details via voice.

[0528] Step 11:

[0529] The user asks a follow-up question. If the user wants to ask more questions, the same process is used to fulfill the new request. For example, "Can you also tell me where the restaurant is?"

[0530] By performing the above steps, the interactive guidance system of the present invention can quickly and accurately respond to user questions.

[0531] Example 1

[0532] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0533] Conventional guidance systems have the problem of being unable to detect the user's approach and therefore unable to provide the necessary information quickly. Furthermore, they lack an adequate interface for generating appropriate responses to user input, resulting in low user satisfaction. Furthermore, conventional systems have the problem of being unable to provide the appropriate information the user is looking for due to the low accuracy of voice input and inaccurate analysis results. To solve these problems, the development of an interactive and accurate guidance system is required.

[0534] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0535] In this invention, the server includes means for detecting the approach of a user with a sensor, means for switching from standby mode to active mode when the user is detected, means for activating an avatar in active mode and displaying a greeting, means for receiving user input, means for converting the received user input into text data with a voice recognition engine, means for transmitting the text data to the server and analyzing it, means for generating an appropriate response message based on the analysis result using a generative AI model, and means for displaying the generated response message to the user in voice and text. This makes it possible to quickly detect when a user is approaching and provide an appropriate response, thereby improving user satisfaction.

[0536] A "means for detecting user proximity by a sensor" is a device for detecting when a user enters the range of the system.

[0537] "Standby mode" refers to a state in which the system is in a low-power standby state and is monitoring external inputs.

[0538] "Active mode" is a mode in which the system goes into full operation after detecting the approach of a user.

[0539] The "means for activating an avatar and displaying a greeting" refers to a means for displaying an anthropomorphized character on the system screen and greeting the user by voice or text.

[0540] The "means for receiving user input" is a device for receiving questions and instructions from the user through voice input or a touch panel.

[0541] The "means for converting into text data using a voice recognition engine" is software that analyzes the user's voice input in real time and converts it into text data.

[0542] The "means for transmitting text data to a server" is a function for transmitting text data acquired from a user to a server via the Internet or the like.

[0543] The "analysis means" is software that allows the server to analyze text data using a natural language processing engine and understand the user's intent.

[0544] "Means for generating using a generative AI model" refers to the process of using generative AI to automatically create an optimal response message based on the analysis results.

[0545] The "means for displaying the generated response message to the user in voice and text" refers to a means for displaying the response message generated by the server on the terminal and conveying it to the user in voice and text through the avatar.

[0546] The present invention is an interactive guidance system that provides users with information quickly and accurately in public, commercial, and event facilities. The system automatically detects when a user approaches and provides optimal responses to the user's questions and requests.

[0547] Hardware and software used

[0548] The following hardware and software are used to implement the system:

[0549] Motion sensor: Used to detect the user's approach.

[0550] Microphone: Used to receive audio input.

[0551] Touch panel: Used to receive user touch input.

[0552] Display: Used for displaying avatars and text.

[0553] Server: Parses the data and generates a response message.

[0554] Speech recognition engine: Converts the user's speech into text.

[0555] Natural Language Processing (NLP) engine: Parses user input.

[0556] Generative AI model: Generates optimal response messages based on analysis results.

[0557] Program processing explanation

[0558] The operation of the system is as follows.

[0559] 1. User approach detection

[0560] The device uses a motion sensor to detect when the user enters a certain range, which causes the device to switch from standby mode to active mode.

[0561] 2. Launching the avatar

[0562] When in active mode, an avatar appears on the device's display and greets the user, for example, "Hello, how can I help you?"

[0563] 3. Accepting User Input

[0564] The user can ask questions or make requests using voice input or a touch panel. For example, they can input, "Please tell me today's event schedule."

[0565] 4. Acquiring input information

[0566] User input is obtained through the device's microphone or touch sensor. In the case of voice input, the voice recognition engine converts it into text data.

[0567] 5. Data Analysis

[0568] The device sends the acquired text data to the server, which then uses a natural language processing engine to analyze the data and understand the user's intent. For example, the server might analyze a request such as "Please tell me the event schedule."

[0569] 6. Generating a Response Message

[0570] The server uses the generative AI model to generate an appropriate response message based on the analysis results, and the generated message is sent from the server to the device.

[0571] 7. Displaying the Response

[0572] The device displays the received response message to the user in voice and text. For example, an avatar may say, "Today's event schedule is a yoga class from 10:00 and a concert from 14:00."

[0573] Adding specific examples

[0574] For example, when a user stands in front of a digital information system installed at the entrance of a department store, the device's sensors detect the user, and an avatar on the screen speaks to them, saying, "Hello, is there something I can help you with?" If the user says, "Can you tell me where the restaurant is?", the device converts the speech into text and sends it to the server. The server analyzes the request, retrieves restaurant information from a database, and generates a response message, stating, "The restaurant is on the third floor, at the back on the right."

[0575] Prompt Sentence Examples

[0576] An example of an input prompt for a generative AI model might be, "Please provide a clear and easy-to-understand answer to the question asked by the user." Based on this prompt, the generative AI model generates the optimal response message.

[0577] This system therefore provides users with quick and accurate information, improving satisfaction with facility use.

[0578] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0579] Step 1:

[0580] User approach detection

[0581] The device detects the user using a sensor

[0582] Input: The motion sensor detects the user's movement.

[0583] Specific operation: The motion sensor detects the user's position and movement and sends a signal to the device.

[0584] Output: A signal is generated that switches the terminal from standby mode to active mode.

[0585] Step 2:

[0586] Launching the Avatar

[0587] The device launches the avatar

[0588] Input: The terminal receives a signal to switch to active mode.

[0589] Specific behavior: An avatar appears on the device display and greets the user using programmed motions.

[0590] Output: The avatar will say and read "Hello, how can I help you?"

[0591] Step 3:

[0592] Accepting user input

[0593] The user enters a question or request

[0594] Input: The user enters a question or request using voice input or a touch panel.

[0595] Specific operation: The user asks a question such as "Please tell me today's event schedule" using the microphone or touch panel.

[0596] Output: User voice or touch data is captured.

[0597] Step 4:

[0598] Obtaining input information

[0599] The device obtains the user's input information.

[0600] Input: For voice input, the user's voice data. For touch input, the touch operation data.

[0601] How it works: A highly sensitive microphone picks up voices, and a voice recognition engine converts them into text data. Touch input data is also sent from the sensor.

[0602] Output: User input data in text format is generated.

[0603] Step 5:

[0604] Data analysis

[0605] The device sends the acquired data to the server.

[0606] Input: User-entered data in text format.

[0607] Specific operation: The device sends data to the server in real time.

[0608] Output: The server receives the user's text input data.

[0609] The server analyzes the data

[0610] Input: The text data sent to the server.

[0611] Specific operation: The natural language processing engine analyzes text data and understands the user's intent. For example, it analyzes a request such as "Please tell me the event schedule."

[0612] Output: Parsed user intent and required information identified.

[0613] Step 6:

[0614] Generating a response message

[0615] The server generates a response message

[0616] Input: Parsed user intent.

[0617] What it does: Uses a generative AI model to create an optimal response message, retrieving relevant information from the necessary databases.

[0618] Output: The generated response message is formed.

[0619] Step 7:

[0620] Viewing the response

[0621] The server sends a response message to the device.

[0622] Input: The generated response message.

[0623] Specific operation: The server sends the generated response message to the terminal.

[0624] Output: The terminal receives the response message.

[0625] The terminal displays a response message to the user.

[0626] Input: The response message received from the server.

[0627] Specific operation: The device's avatar conveys the received message to the user via voice and text. For example, the avatar may display and speak, "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00."

[0628] Output: The user receives the necessary information visually and audibly.

[0629] (Application example 1)

[0630] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0631] Conventional guidance systems often struggle to respond quickly and accurately to user questions. In particular, when it comes to effectively utilizing voice input, the user experience is often compromised by inaccurate voice recognition technology and natural language processing technology that cannot accurately analyze user intent. Furthermore, responses to user voice input are monotonous, resulting in insufficient user interaction, making it difficult to improve visitor satisfaction.

[0632] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0633] In this invention, the server includes means for detecting the approach of a user with a sensor, means for receiving user input by voice, means for converting the received user voice into text data, means for analyzing the text data, means for generating a response message using a generative AI model based on the analysis results, and means for providing the generated response message to the user in voice and text, thereby enabling the server to provide quick and accurate responses to user questions and requests and realizing an interactive user experience.

[0634] The "means for detecting the approach of a user by a sensor" refers to a device or function for detecting the presence of a user when the user approaches the system.

[0635] The "means for receiving user input by voice" refers to a device or function for acquiring voice data provided by the user.

[0636] The "means for converting received user voice into text data" refers to a process or device for analyzing the user voice data and converting it into text format data.

[0637] "Means for analyzing text data" refers to software or algorithms that, after speech input has been converted into text data, analyze the text data and understand the user's intent.

[0638] A "generative means using a generative AI model" is an artificial intelligence model or algorithm for generating an appropriate response message based on the analyzed user text data.

[0639] A "means for providing a generated response message to a user in both audio and text" is a device or process for providing a generated response message to a user in both audio and text form.

[0640] The present invention provides an interactive guidance system that enables users to quickly and accurately obtain information within a virtual environment. Specific embodiments will be described below.

[0641] System configuration

[0642] The system consists of the following main components:

[0643] 1. How to use a sensor to detect the user's approach:

[0644] A motion sensor is used to detect when a user approaches the system, readying the system for user interaction.

[0645] 2. A way to receive user input by voice:

[0646] A microphone is used to capture the user's voice input, which is then analyzed.

[0647] 3. A means of converting received user speech into text data:

[0648] Use a speech recognition API (for example, Google Speech Recognition API) to convert the acquired speech data into text data in real time.

[0649] 4. Means of analyzing text data:

[0650] The converted text data is then analyzed by a natural language processing engine and processed to understand the user's intent. Specifically, a Transformer-based generative AI model (e.g., GPT-2) is used.

[0651] 5. Generative AI model generation method:

[0652] Based on the analysis results, an appropriate response message is generated, which may involve retrieving information from a database to provide the best possible answer to the user's question or request.

[0653] 6. Means for providing the generated response message to the user in voice and text:

[0654] It uses Google Text-to-Speech (gTTS) to convert the generated text response into audio, providing it to the user in both audio and text form, and conveying information both visually and audibly.

[0655] Data Calculation and Processing

[0656] 1. Acquire and convert voice input:

[0657] The user speaks a question to the system, the microphone captures the voice signal, and the data is converted into text data through a speech recognition API.

[0658] 2. Text data analysis:

[0659] The converted text data is sent to a server, where a natural language processing engine analyzes it to understand the user's intent. This analysis uses the GPT-2 model.

[0660] 3. Generate a response message:

[0661] The generative AI model generates the optimal response message that matches the analyzed data. During this generation process, it searches and retrieves the necessary information from the database and determines the content to be provided to the user.

[0662] 4. Providing generated messages:

[0663] The generated response message is converted from text data to voice data again and provided to the user. At the same time, it is also displayed in text format, so the user can receive information in both voice and text.

[0664] Specific examples

[0665] Consider a scenario in which a user connects to a virtual store through smart glasses and inquires about product availability. When the user verbally asks, "Is this product in stock?", the system captures voice data through a microphone and converts the speech into text using the Google Speech Recognition API. The GPT-2 model then analyzes the text data and retrieves the corresponding inventory information from the inventory database. The generated response message, "This product is in stock," is provided to the user in both voice and text.

[0666] Prompt Sentence Examples

[0667] example:

[0668] "Hello, are you looking for something?"

[0669] "What is the most popular item in this store?"

[0670] "Please let me know about this week's sales."

[0671] This system allows users to obtain fast and accurate information even in virtual environments, improving the overall user experience.

[0672] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0673] Step 1:

[0674] A user approaches the system. The device's motion sensor detects the user and the system switches from standby mode to active mode.

[0675] Input: User approach

[0676] Output: Switching the system to active mode

[0677] What happens: The motion sensor reacts, an internal signal from the device triggers the system's active mode, and an avatar appears on the screen and greets the user.

[0678] Step 2:

[0679] The user asks a question by voice, and the device's microphone picks up the voice and captures the audio signal.

[0680] Input: User's voice question

[0681] Output: Audio data

[0682] What it does: A user speaks into a microphone asking a question, such as "Do you have this product in stock?" The microphone picks up the voice signal and stores it as digital audio data.

[0683] Step 3:

[0684] The voice data acquired by the device is converted into text data via the Google Speech Recognition API.

[0685] Input: Audio data

[0686] Output: Text data

[0687] How it works: Audio data is sent to a cloud service, where it is analyzed by the Google Speech Recognition API and converted into corresponding text data.

[0688] Step 4:

[0689] The server receives the text data and analyzes it using a natural language processing engine (GPT-2), which searches a database to understand the user's intent and determine the appropriate response.

[0690] Input: Text data

[0691] Output: Analysis results and generation of optimal response messages

[0692] How it works: The server inputs text data into the GPT-2 model and begins analysis. During the analysis process, the user's question (e.g., "Is it in stock?") is understood and relevant information (e.g., "inventory data") is searched for in the database.

[0693] Step 5:

[0694] The server uses the generative AI model to generate a response message based on the analysis results.

[0695] Input: Analysis results and information obtained from databases

[0696] Output: Response message (text format)

[0697] Specific behavior: The GPT-2 model generates a text message such as "This product is in stock." The generated message is stored as internal data.

[0698] Step 6:

[0699] The device uses Google Text-to-Speech (gTTS) to convert the generated text message into audio data and provide it to the user.

[0700] Input: Text message

[0701] Output: Audio data and text display

[0702] How it works: A text message is entered into gTTS and converted into voice data, which is then played back through the device's speaker and simultaneously displayed on the screen.

[0703] Step 7:

[0704] The user reviews the response message and, if necessary, asks additional questions or provides instructions, and the cycle repeats.

[0705] Input: User confirmation and follow-up questions

[0706] Output: Additional audio data

[0707] What happens: The user listens to the voice response or reads the text, then speaks again if they have any further questions. This process can be repeated as necessary.

[0708] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0709] The present invention provides an interactive guidance system that not only provides users with information quickly and accurately in public facilities, commercial facilities, and event facilities, but also recognizes the user's emotions to provide more personalized responses. This system automatically detects when a user approaches and can provide optimal responses to the user's questions or requests. Furthermore, it utilizes an emotion engine to recognize the user's emotions and respond based on those emotions. Specific embodiments are described in detail below.

[0710] Basic operation

[0711] Reaction when the user approaches

[0712] 1. The device detects the user using a sensor

[0713] The digital guidance system is equipped with motion sensors that automatically detect when a user approaches.

[0714] When the sensor detects the presence of a user, the device switches from standby mode to active mode.

[0715] 2. The device launches the avatar

[0716] When the device switches to active mode, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[0717] The process that receives user input

[0718] 1. The user asks a question

[0719] The user can use voice input or a touch panel to ask questions or make requests, for example, "Please tell me today's event schedule."

[0720] 2. The device gets the user's input

[0721] The device is equipped with a microphone and touch sensor, through which the user's voice input and touch operations are acquired.

[0722] In the case of voice input, the voice is converted into text data in real time by a voice recognition engine.

[0723] Server parsing and generation

[0724] 1. Parse the input

[0725] The device transmits the acquired voice and text data to the server.

[0726] The server uses a natural language processing engine and an emotion engine to analyze the user's intent and emotions. For example, it analyzes the text data "Please tell me the event schedule" as well as the user's tone of voice and facial expressions to identify their emotional state.

[0727] 2. Generate a response message

[0728] Based on the analysis results, the server uses AI to create the optimal response message. Depending on the user's emotional state, a message that takes into account their emotions, such as "Today's event schedule is as follows. Have a great day!", is generated.

[0729] Response display by terminal

[0730] 1. Display a response message

[0731] The server generates a response message and sends it to the terminal.

[0732] The device's avatar will respond to the user with a voice or text message. For example, it might say, "Today's event schedule is as follows," and provide details in voice.

[0733] Specific examples

[0734] For example, imagine a user standing in front of a digital information system installed at the entrance of a department store. As the user approaches, the device's sensors detect the user's approach and an avatar speaks to them, saying, "Hello, is there anything I can help you with?" If the user responds, "Can you tell me where the restaurant is?", the device converts the speech into text and sends that data to the server. The server analyzes this request and, using an emotion engine, infers, for example, that the user is in a hurry based on the user's tone of voice and facial expression. As a result, it generates a specific, situation-appropriate guidance message: "It seems you're in a hurry. The restaurant is on the third floor, at the back on the right." The generated message is sent to the device, and the avatar relays it to the user.

[0735] In this way, the interactive guidance system of the present invention allows users to quickly and accurately obtain the information they need. Furthermore, the introduction of an emotion engine enables personalized responses that take into account the user's emotions, greatly improving user satisfaction.

[0736] The processing flow will be explained below.

[0737] Step 1:

[0738] The device detects the user with a sensor. When a user approaches the digital signage, the motion sensor detects their presence, causing the device to switch from standby mode to active mode.

[0739] Step 2:

[0740] The device activates the avatar. When the sensor detects the user's approach, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[0741] Step 3:

[0742] The user asks a question. The user uses voice input or a touch panel to ask a question or make a request. For example, the user might ask, "Please tell me today's event schedule."

[0743] Step 4:

[0744] The device receives user input. The device's microphone or touch sensor captures the user's question. In the case of voice input, the speech recognition engine converts the voice into text in real time.

[0745] Step 5:

[0746] The device activates an emotion engine to analyze the user's emotions, analyzing voice tone, speaking style, facial expressions, etc. to determine whether the user is angry, in a hurry, having fun, etc.

[0747] Step 6:

[0748] The device sends text data and emotion data to the server, which then sends the acquired text data and emotion analysis results together.

[0749] Step 7:

[0750] The server analyzes the input. The server's natural language processing engine analyzes the text data sent and understands the user's intent. For example, it recognizes a request such as "Please tell me the event schedule."

[0751] Step 8:

[0752] The server generates a response message based on the emotional data. Based on the analysis results, the server uses generation AI to create an appropriate response message. Depending on the user's emotional state, a message that takes emotions into consideration is generated, such as "Today's event schedule is as follows. Have a great day!"

[0753] Step 9:

[0754] The server sends a response message to the terminal. The server then sends the generated response message to the terminal. If the user is in a hurry, it generates a concise and quick message, but if the user is relaxed, it generates a detailed and polite message.

[0755] Step 10:

[0756] The device displays the response to the user. The device's avatar then relays the received response message to the user via voice or text. For example, it might display "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00," and provide details via voice.

[0757] Step 11:

[0758] The user asks a follow-up question. If the user wants to ask more questions, the same process is used to respond to the new request. For example, answer a question like, "Can you also tell me where the restaurant is?". Even for follow-up questions, sentiment analysis is performed and an appropriate message is generated and displayed.

[0759] By implementing the above steps, the interactive guidance system of the present invention can quickly and accurately respond to user questions. Furthermore, the introduction of an emotion engine enables personalized responses that take into account the user's emotions, greatly improving user satisfaction.

[0760] Example 2

[0761] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0762] Conventional guidance systems have difficulty in providing quick and accurate information to users' questions, and are unable to provide personalized responses that reflect the user's emotions. This has led to problems such as lower user satisfaction and a decrease in frequency of use.

[0763] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0764] In this invention, the server includes means for detecting the approach of a user with a sensor, means for receiving user input, means for analyzing the received user input, means for generating a response message based on the analysis result, means for displaying the generated response message to the user, means for recognizing the user's emotion, and means for personalizing the response message based on the recognized emotion. This enables quick and accurate provision of information, and further allows for personalized responses to be provided in accordance with the user's emotion, thereby improving user satisfaction.

[0765] A "sensor" is a device that detects physical changes or movements and captures them as data.

[0766] A "user" is a person or individual who uses the system to obtain information.

[0767] "Approach" refers to the user moving into the system's recognition range.

[0768] "Detecting" means that the sensor recognizes the user's approach and sends that information to the system.

[0769] "Means" is a general term for methods or devices used to achieve a goal.

[0770] "Receiving" means that the system takes input from the user and processes it.

[0771] "Input" refers to the information or instructions a user provides to a system, and can take the form of voice, touch, or other input.

[0772] "Analyzing" means analyzing the received input data and identifying the user's intentions and the information they are looking for.

[0773] "Generating" means creating a new response message based on the analysis results.

[0774] A "response message" is information or a response that the system provides to the user, and may be in the form of text or audio.

[0775] "Displaying" means conveying the generated response message to the user on the screen or by voice.

[0776] "Emotion" refers to the user's psychological state or mood, and is recognized by the system.

[0777] "Recognizing" means that the system captures the user's emotions and uses them for analysis.

[0778] "Personalization" means customizing a response message according to the user's emotions and individual circumstances.

[0779] The present invention is an interactive guidance system that not only provides users with information quickly and accurately in public facilities, commercial facilities, and event facilities, but also recognizes the user's emotions to provide more personalized responses. This system has the function of automatically detecting when a user approaches and providing the most appropriate response to the user's questions or requests.

[0780] First, the device uses sensors to detect the user's approach. Specifically, a motion sensor detects the user's movement, and when the user comes within a certain distance, the system switches to active mode. Then, an avatar displayed on the device speaks to the user, saying, "Hello, is there anything I can help you with?" The avatar is programmed using game engines such as Unity or Unreal Engine.

[0781] Users can ask questions or make requests using voice input or the touch panel. For example, if they ask, "What is today's event schedule?", the microphone picks up the voice and converts it into text in real time using the Google Cloud Speech-to-Text API. The text data is saved in a temporary file and then sent to the server.

[0782] The server uses the Google Cloud Natural Language API to analyze the received text data and parses the user's intent. It also uses emotion engines such as Amazon Comprehend to analyze the user's emotional state based on their tone of voice and facial expressions. Emotional information such as whether the user is in a hurry can also be collected.

[0783] Based on the analysis results, a generative AI (e.g., OpenAI GPT-4) generates an optimal response message. The prompt is "Please tell me the event schedule. The user seems to be in a hurry." The message generated by the AI ​​model corresponds to the emotion, such as "Today's event schedule is as follows. Have a nice day."

[0784] The server then sends the generated response message to the device, and the device communicates the content to the user in voice and text. For example, if a user asks, "Where is the restaurant?", the server analyzes the question and generates a message saying, "You seem to be in a hurry. The restaurant is on the third floor, at the back on the right," which the device then communicates to the user.

[0785] This system not only allows users to obtain accurate information quickly, but also provides personalized responses that take their emotions into consideration. For example, by inputting a prompt such as "Please generate a response for how the guidance system should greet the user when they approach," into the AI ​​model, an appropriate response can be generated.

[0786] As described above, the interactive guidance system of the present invention can provide users with a high level of satisfaction.

[0787] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0788] Interactive guidance system processing flow

[0789] Step 1:

[0790] The device uses a sensor to detect the user's approach.

[0791] What it does: Motion sensors detect user movement within 20 feet.

[0792] Input: Motion information from the external environment.

[0793] Data processing: Sensors convert physical movements into digital signals.

[0794] Output: A signal indicating the user's approach.

[0795] Next step: Send a signal to switch from standby mode to active mode.

[0796] Step 2:

[0797] The device switches to active mode and activates the avatar.

[0798] What happens: The device's display turns on and an avatar appears, saying, "Hello, how can I help you?"

[0799] Input: User approach detection signal from sensor.

[0800] Data processing: The system switches from standby mode to active mode. The avatar's program is activated.

[0801] Output: A voice message to the user.

[0802] Next: Wait for user input.

[0803] Step 3:

[0804] The user inputs a question by voice or via a touch panel.

[0805] Specific behavior: The user asks "What is today's event schedule?" by voice or by entering text using the touch panel.

[0806] Input: User voice or touch input data.

[0807] Data processing: A microphone and touch sensor capture user input and convert it into digital data. Voice input is converted to text using the Google Cloud Speech-to-Text API.

[0808] Output: User questions in text format.

[0809] Next process: The terminal prepares to send text data to the server.

[0810] Step 4:

[0811] The terminal sends the user's input to the server.

[0812] Specific operation: The acquired text data is sent to the server using the HTTPS protocol.

[0813] Input: The user's question in text form.

[0814] Data processing: The text data is divided into packets and sent to the server over a secure communication channel.

[0815] Output: User question data sent to the server.

[0816] Next: The server parses the user's input data.

[0817] Step 5:

[0818] The server parses the input data.

[0819] What it does: It uses the Google Cloud Natural Language API to analyze the content of the user's question, and Amazon Comprehend to analyze the user's sentiment.

[0820] Input: User question data in text format.

[0821] Data processing: Perform natural language processing and sentiment analysis to identify user intent and emotional state.

[0822] Output: Analysis results including user intent (e.g., event schedule question) and emotional state (e.g., in a hurry).

[0823] Next process: Generate a response message based on the analysis results.

[0824] Step 6:

[0825] The server generates a response message.

[0826] Specific operation: Uses generative AI (e.g., OpenAI GPT-4) to generate a response message based on the analysis results.

[0827] Input: Analysis results, including the user's intent and emotional state.

[0828] Data processing: A prompt is input into the generative AI model to generate the optimal response message. Example prompt: "Generate a response sentence for how the guidance system should greet the user when they approach."

[0829] Output: Personalized response messages depending on the emotion.

[0830] Next: Send a response message to the terminal.

[0831] Step 7:

[0832] The server generates a message and sends it to the terminal.

[0833] Specific operation: Sends a response message to the terminal via the HTTPS protocol.

[0834] Input: The generated response message.

[0835] Data processing: Divide the message into packets and send them securely to the terminal.

[0836] Output: Response message sent to the terminal.

[0837] Next process: The terminal displays and transmits the response message to the user.

[0838] Step 8:

[0839] The terminal displays and transmits the response message.

[0840] Specific operation: The avatar will speak the generated response message to the user, and the text "Today's event schedule is as follows" will be displayed on the screen.

[0841] Input: The response message from the server.

[0842] Data processing: A speech synthesis engine is used to convert text into speech, and a program to move the avatar is executed.

[0843] Output: The response message that is conveyed to the user.

[0844] Next: Wait for further user input if necessary.

[0845] (Application example 2)

[0846] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0847] Conventional interactive guidance systems provide uniform information to users, making it difficult to provide appropriate guidance that reflects each user's individual situation and emotions. In particular, in environments where a variety of information is required, such as brick-and-mortar stores, there is a need to respond quickly and individually to user needs. Furthermore, when a user feels lost or confused, a system that can accurately detect the situation and provide appropriate guidance is needed.

[0848] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0849] In this invention, the server includes means for detecting the approach of a user with a detection device, means for acquiring user input, means for analyzing the acquired user input, means for generating response information based on the analysis result using a generative AI model, display means for providing the generated response information to the user, means for recognizing the user's emotional state, voice recognition means for converting voice input into text data, and means for searching for information from a database based on the analyzed user input. This enables personalized guidance according to the user's emotions, and makes it possible to quickly and accurately provide the information users need even in physical stores.

[0850] A "detection device" is a device that detects the approach of a user, and specifically refers to a motion sensor or camera.

[0851] "User input" refers to information or questions provided by a user, which may be obtained in the form of voice, text, touch, or other input.

[0852] The "analysis means" is a means for understanding and analyzing acquired user input, and utilizes natural language processing and emotion recognition technology.

[0853] A "generative artificial intelligence model" is an artificial intelligence technique for generating appropriate response information based on the results obtained using analytical means.

[0854] "Response information" is information or answers provided in response to user input, and is generated by a generative AI model.

[0855] The "display means" is a means for providing the generated response information to the user, and refers to a display, an audio output device, or the like.

[0856] The "emotional state recognition means" is a means for recognizing the user's emotions from facial expressions, tone of voice, etc., and utilizes an emotion analysis engine or the like.

[0857] The "voice recognition means" is a means for converting voice input into character data, and utilizes a voice recognition engine.

[0858] The "information retrieval means" is a means for retrieving necessary information from a database based on the analyzed user input.

[0859] A "database" is a data storage system that stores necessary information and serves as a source of information for responding to user inquiries.

[0860] The present invention relates to an interactive guidance system that quickly provides appropriate information to users in a physical store. The system includes the following main components:

[0861] Detection device

[0862] The system includes a detection device for detecting the approach of a user. This detection device can be a motion sensor or a camera. For example, the camera detects the user's movement and detects that the user is approaching the system.

[0863] Getting User Input

[0864] User input is acquired in the form of voice, touch operation, and text. Voice input is acquired using a microphone, and touch operation is performed through a touch panel. The acquired voice input is converted into text data by a voice recognition engine.

[0865] Analysis means

[0866] The acquired user input is sent to the server and analyzed by a natural language processing engine and an emotion recognition engine. The natural language processing engine analyzes the input text data and understands the content of the user's request or question. The emotion recognition engine recognizes emotions from the user's tone of voice and facial expressions.

[0867] Response generation using generative artificial intelligence models

[0868] Based on the analysis results, the generative AI model generates optimal response information that is tailored to the user's intent and emotional state and is designed to address their individual needs, such as clearly explaining the solution if the user is confused.

[0869] Display means

[0870] The generated response information is provided to the user via a display and an audio output device. The response is displayed in text format on the display, and if an audio output device is used, audio guidance is also provided.

[0871] Example of a system

[0872] A concrete example of such a system is a physical store guidance system using smart glasses. In this case, facial recognition and emotion analysis are performed while the user wears the glasses and moves around the store. When the user asks, "Where is this item?", a speech recognition engine converts the question into text data, and a generative AI model generates a response such as, "Let's get started right away. Go straight and it's on the next shelf on the right."

[0873] Prompt Sentence Examples

[0874] Examples of prompts to input to a generative AI model include:

[0875] "User's query: 'Where is this item?' Sentiment analysis: 'Confused' Generate the best response."

[0876] In this way, by implementing the interactive guidance system of the present invention, personalized guidance can be provided according to the user's emotions, improving the customer experience in a physical store.

[0877] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0878] Step 1:

[0879] The device detects the user's approach

[0880] Input: User approach status

[0881] Operation: A motion sensor or camera installed on the device detects the user's approach. The camera captures the user's movements and recognizes that the user is approaching the system. Output: The user's presence is detected.

[0882] Step 2:

[0883] The device switches to active mode

[0884] Input: User approach information

[0885] Action: The device switches from standby mode to active mode, and the avatar displayed on the screen comes to life. The avatar speaks to the user, saying, "Hello, how can I help you?" Output: Readiness for user response

[0886] Step 3:

[0887] Getting user input

[0888] Input: User's voice input or touch actions

[0889] Action: The user asks a question or makes a request. For example, "Where is this item?" The device's microphone picks up the voice, or the touch panel receives touch input. Output: The captured user input data.

[0890] Step 4:

[0891] Converting voice input to text data

[0892] Input: User voice input

[0893] Operation: The acquired voice input is converted into text data in real time by the voice recognition engine on the device. Output: Text data converted from voice

[0894] Step 5:

[0895] The server parses the input data

[0896] Input: Acquired text data

[0897] Operation: The server analyzes the text data using a natural language processing engine to understand the user's intention and the content of the question. At the same time, the emotion recognition engine analyzes the user's tone of voice and facial expressions to identify their emotional state. Output: Analyzed user intention and emotional information

[0898] Step 6:

[0899] Generating response information using a generative AI model

[0900] Input: Analyzed user intent and emotion information

[0901] Operation: The server's AI model generates the optimal response information based on the analysis results. For example, if the user is confused, it generates a response such as, "Let's start the guide right away. Go straight and it's on the next shelf on the right." Output: Generated response information

[0902] Step 7:

[0903] Display response information

[0904] Input: Generated response information

[0905] Operation: Provides response information to the user through the terminal's display and audio output device. The response is displayed in text format on the display, and is played back aloud when the audio output device is used. Output: Response information presented to the user

[0906] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0907] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0908] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0909] [Third embodiment]

[0910] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0911] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0912] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0913] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0914] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0915] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0916] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0917] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0918] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0919] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0920] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0921] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0922] The present invention provides an interactive guidance system that quickly and accurately provides information to users in public facilities, commercial facilities, and event facilities. This system automatically detects when a user approaches and can provide optimal responses to the user's questions or requests. Specific embodiments are described in detail below.

[0923] Reaction when the user approaches

[0924] 1. The device detects the user using a sensor

[0925] The digital guidance system is equipped with motion sensors that automatically detect when a user approaches.

[0926] When the sensor detects the presence of a user, the device switches from standby mode to active mode.

[0927] 2. The device launches the avatar

[0928] When the device switches to active mode, the avatar displayed on the device screen comes to life and greets the user.

[0929] For example, the avatar might say, "Hello, is there anything I can help you with?"

[0930] The process that receives user input

[0931] 1. The user asks a question

[0932] Users can use voice input or a touch panel to ask questions or make requests. For example, they can ask, "Please tell me today's event schedule."

[0933] 2. The device gets the user's input

[0934] The device is equipped with a microphone and touch sensor, through which the user's voice input and touch operations are acquired.

[0935] In the case of voice input, the voice is converted into text data in real time by a voice recognition engine.

[0936] Server parsing and generation

[0937] 1. Parse the input

[0938] The device transmits the acquired voice and text data to the server.

[0939] The server uses a natural language processing engine to analyze the user's intent. For example, it analyzes text data such as "Please tell me the event schedule" and identifies the information the user wants to know.

[0940] 2. Generate a response message

[0941] Based on the analysis results, the server uses generation AI to create the optimal response message.

[0942] For example, information about the event schedule may be retrieved from a database and a response message such as "Today's event schedule is as follows..." may be generated.

[0943] Response display by terminal

[0944] 1. Display a response message

[0945] The server generates a response message and sends it to the terminal.

[0946] The device's avatar will convey the response message to the user via voice or text.

[0947] For example, "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00."

[0948] Specific examples

[0949] For example, imagine a user standing in front of a digital information system installed at the entrance of a department store. As the user approaches, the device's sensors detect the user and an avatar speaks, saying, "Hello, is there anything I can help you with?" If the user responds, "Can you tell me where the restaurant is?", the device converts the speech into text and sends it to the server. The server analyzes this request, retrieves the restaurant's location information from a database, and generates an optimal guidance message. The generated message is sent to the device, and the avatar responds, "The restaurant is on the third floor, at the back on the right."

[0950] In this way, the interactive guide system of the present invention allows users to quickly and accurately obtain the information they need, which can greatly improve visitor satisfaction.

[0951] The processing flow will be explained below.

[0952] Step 1:

[0953] The device detects the user with a sensor. When a user approaches the digital signage, the motion sensor detects their presence, causing the device to switch from standby mode to active mode.

[0954] Step 2:

[0955] The device activates the avatar. When the sensor detects the user's approach, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[0956] Step 3:

[0957] The user asks a question. The user uses voice input or a touch panel to ask a question or make a request. For example, the user might ask, "Please tell me today's event schedule."

[0958] Step 4:

[0959] The device receives user input. The device's microphone or touch sensor captures the user's question. In the case of voice input, the speech recognition engine converts the voice into text in real time.

[0960] Step 5:

[0961] The terminal sends the text data to the server. The acquired text data is then sent from the terminal to the server.

[0962] Step 6:

[0963] The server analyzes the input. The server's natural language processing engine analyzes the text data sent and understands the user's intent. For example, it recognizes a request such as "Please tell me the event schedule."

[0964] Step 7:

[0965] The server retrieves the relevant data. Based on the analysis results, the server searches and retrieves relevant information from the database. In this case, it retrieves the event schedule information.

[0966] Step 8:

[0967] The server generates a response message. The AI ​​then creates the optimal response message based on the acquired information. For example, it generates a message such as, "Today's event schedule is a yoga class from 10:00 and a concert from 14:00."

[0968] Step 9:

[0969] The server sends a response message to the terminal. The generated response message is sent from the server to the terminal.

[0970] Step 10:

[0971] The device displays the response to the user. The device's avatar conveys the received response message to the user via voice or text. For example, it may display "Today's event schedule is as follows" and provide details via voice.

[0972] Step 11:

[0973] The user asks a follow-up question. If the user wants to ask more questions, the same process is used to fulfill the new request. For example, "Can you also tell me where the restaurant is?"

[0974] By performing the above steps, the interactive guidance system of the present invention can quickly and accurately respond to user questions.

[0975] Example 1

[0976] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0977] Conventional guidance systems have the problem of being unable to detect the user's approach and therefore unable to provide the necessary information quickly. Furthermore, they lack an adequate interface for generating appropriate responses to user input, resulting in low user satisfaction. Furthermore, conventional systems have the problem of being unable to provide the appropriate information the user is looking for due to the low accuracy of voice input and inaccurate analysis results. To solve these problems, the development of an interactive and accurate guidance system is required.

[0978] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0979] In this invention, the server includes means for detecting the approach of a user with a sensor, means for switching from standby mode to active mode when the user is detected, means for activating an avatar in active mode and displaying a greeting, means for receiving user input, means for converting the received user input into text data with a voice recognition engine, means for transmitting the text data to the server and analyzing it, means for generating an appropriate response message based on the analysis result using a generative AI model, and means for displaying the generated response message to the user in voice and text. This makes it possible to quickly detect when a user is approaching and provide an appropriate response, thereby improving user satisfaction.

[0980] A "means for detecting user proximity by a sensor" is a device for detecting when a user enters the range of the system.

[0981] "Standby mode" refers to a state in which the system is in a low-power standby state and is monitoring external inputs.

[0982] "Active mode" is a mode in which the system goes into full operation after detecting the approach of a user.

[0983] The "means for activating an avatar and displaying a greeting" refers to a means for displaying an anthropomorphized character on the system screen and greeting the user by voice or text.

[0984] The "means for receiving user input" is a device for receiving questions and instructions from the user through voice input or a touch panel.

[0985] The "means for converting into text data using a voice recognition engine" is software that analyzes the user's voice input in real time and converts it into text data.

[0986] The "means for transmitting text data to a server" is a function for transmitting text data acquired from a user to a server via the Internet or the like.

[0987] The "analysis means" is software that allows the server to analyze text data using a natural language processing engine and understand the user's intent.

[0988] "Means for generating using a generative AI model" refers to the process of using generative AI to automatically create an optimal response message based on the analysis results.

[0989] The "means for displaying the generated response message to the user in voice and text" refers to a means for displaying the response message generated by the server on the terminal and conveying it to the user in voice and text through the avatar.

[0990] The present invention is an interactive guidance system that provides users with information quickly and accurately in public, commercial, and event facilities. The system automatically detects when a user approaches and provides optimal responses to the user's questions and requests.

[0991] Hardware and software used

[0992] The following hardware and software are used to implement the system:

[0993] Motion sensor: Used to detect the user's approach.

[0994] Microphone: Used to receive audio input.

[0995] Touch panel: Used to receive user touch input.

[0996] Display: Used for displaying avatars and text.

[0997] Server: Parses the data and generates a response message.

[0998] Speech recognition engine: Converts the user's speech into text.

[0999] Natural Language Processing (NLP) engine: Parses user input.

[1000] Generative AI model: Generates optimal response messages based on analysis results.

[1001] Program processing explanation

[1002] The operation of the system is as follows.

[1003] 1. User approach detection

[1004] The device uses a motion sensor to detect when the user enters a certain range, which causes the device to switch from standby mode to active mode.

[1005] 2. Launching the avatar

[1006] When in active mode, an avatar appears on the device's display and greets the user, for example, "Hello, how can I help you?"

[1007] 3. Accepting User Input

[1008] The user can ask questions or make requests using voice input or a touch panel. For example, they can input, "Please tell me today's event schedule."

[1009] 4. Acquiring input information

[1010] User input is obtained through the device's microphone or touch sensor. In the case of voice input, the voice recognition engine converts it into text data.

[1011] 5. Data Analysis

[1012] The device sends the acquired text data to the server, which then uses a natural language processing engine to analyze the data and understand the user's intent. For example, the server might analyze a request such as "Please tell me the event schedule."

[1013] 6. Generating a Response Message

[1014] The server uses the generative AI model to generate an appropriate response message based on the analysis results, and the generated message is sent from the server to the device.

[1015] 7. Displaying the Response

[1016] The device displays the received response message to the user in voice and text. For example, an avatar may say, "Today's event schedule is a yoga class from 10:00 and a concert from 14:00."

[1017] Adding specific examples

[1018] For example, when a user stands in front of a digital information system installed at the entrance of a department store, the device's sensors detect the user, and an avatar on the screen speaks to them, saying, "Hello, is there something I can help you with?" If the user says, "Can you tell me where the restaurant is?", the device converts the speech into text and sends it to the server. The server analyzes the request, retrieves restaurant information from a database, and generates a response message, stating, "The restaurant is on the third floor, at the back on the right."

[1019] Prompt Sentence Examples

[1020] An example of an input prompt for a generative AI model might be, "Please provide a clear and easy-to-understand answer to the question asked by the user." Based on this prompt, the generative AI model generates the optimal response message.

[1021] This system therefore provides users with quick and accurate information, improving satisfaction with facility use.

[1022] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1023] Step 1:

[1024] User approach detection

[1025] The device detects the user using a sensor

[1026] Input: The motion sensor detects the user's movement.

[1027] Specific operation: The motion sensor detects the user's position and movement and sends a signal to the device.

[1028] Output: A signal is generated that switches the terminal from standby mode to active mode.

[1029] Step 2:

[1030] Launching the Avatar

[1031] The device launches the avatar

[1032] Input: The terminal receives a signal to switch to active mode.

[1033] Specific behavior: An avatar appears on the device display and greets the user using programmed motions.

[1034] Output: The avatar will say and read "Hello, how can I help you?"

[1035] Step 3:

[1036] Accepting user input

[1037] The user enters a question or request

[1038] Input: The user enters a question or request using voice input or a touch panel.

[1039] Specific operation: The user asks a question such as "Please tell me today's event schedule" using the microphone or touch panel.

[1040] Output: User voice or touch data is captured.

[1041] Step 4:

[1042] Obtaining input information

[1043] The device obtains the user's input information.

[1044] Input: For voice input, the user's voice data. For touch input, the touch operation data.

[1045] How it works: A highly sensitive microphone picks up voices, and a voice recognition engine converts them into text data. Touch input data is also sent from the sensor.

[1046] Output: User input data in text format is generated.

[1047] Step 5:

[1048] Data analysis

[1049] The device sends the acquired data to the server.

[1050] Input: User-entered data in text format.

[1051] Specific operation: The device sends data to the server in real time.

[1052] Output: The server receives the user's text input data.

[1053] The server analyzes the data

[1054] Input: The text data sent to the server.

[1055] Specific operation: The natural language processing engine analyzes text data and understands the user's intent. For example, it analyzes a request such as "Please tell me the event schedule."

[1056] Output: Parsed user intent and required information identified.

[1057] Step 6:

[1058] Generating a response message

[1059] The server generates a response message

[1060] Input: Parsed user intent.

[1061] What it does: Uses a generative AI model to create an optimal response message, retrieving relevant information from the necessary databases.

[1062] Output: The generated response message is formed.

[1063] Step 7:

[1064] Viewing the response

[1065] The server sends a response message to the device.

[1066] Input: The generated response message.

[1067] Specific operation: The server sends the generated response message to the terminal.

[1068] Output: The terminal receives the response message.

[1069] The terminal displays a response message to the user.

[1070] Input: The response message received from the server.

[1071] Specific operation: The device's avatar conveys the received message to the user via voice and text. For example, the avatar may display and speak, "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00."

[1072] Output: The user receives the necessary information visually and audibly.

[1073] (Application example 1)

[1074] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1075] Conventional guidance systems often struggle to respond quickly and accurately to user questions. In particular, when it comes to effectively utilizing voice input, the user experience is often compromised by inaccurate voice recognition technology and natural language processing technology that cannot accurately analyze user intent. Furthermore, responses to user voice input are monotonous, resulting in insufficient user interaction, making it difficult to improve visitor satisfaction.

[1076] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1077] In this invention, the server includes means for detecting the approach of a user with a sensor, means for receiving user input by voice, means for converting the received user voice into text data, means for analyzing the text data, means for generating a response message using a generative AI model based on the analysis results, and means for providing the generated response message to the user in voice and text, thereby enabling the server to provide quick and accurate responses to user questions and requests and realizing an interactive user experience.

[1078] The "means for detecting the approach of a user by a sensor" refers to a device or function for detecting the presence of a user when the user approaches the system.

[1079] The "means for receiving user input by voice" refers to a device or function for acquiring voice data provided by the user.

[1080] The "means for converting received user voice into text data" refers to a process or device for analyzing the user voice data and converting it into text format data.

[1081] "Means for analyzing text data" refers to software or algorithms that, after speech input has been converted into text data, analyze the text data and understand the user's intent.

[1082] A "generative means using a generative AI model" is an artificial intelligence model or algorithm for generating an appropriate response message based on the analyzed user text data.

[1083] A "means for providing a generated response message to a user in both audio and text" is a device or process for providing a generated response message to a user in both audio and text form.

[1084] The present invention provides an interactive guidance system that enables users to quickly and accurately obtain information within a virtual environment. Specific embodiments will be described below.

[1085] System configuration

[1086] The system consists of the following main components:

[1087] 1. How to use a sensor to detect the user's approach:

[1088] A motion sensor is used to detect when a user approaches the system, readying the system for user interaction.

[1089] 2. A way to receive user input by voice:

[1090] A microphone is used to capture the user's voice input, which is then analyzed.

[1091] 3. A means of converting received user speech into text data:

[1092] Use a speech recognition API (for example, Google Speech Recognition API) to convert the acquired speech data into text data in real time.

[1093] 4. Means of analyzing text data:

[1094] The converted text data is then analyzed by a natural language processing engine and processed to understand the user's intent. Specifically, a Transformer-based generative AI model (e.g., GPT-2) is used.

[1095] 5. Generative AI model generation method:

[1096] Based on the analysis results, an appropriate response message is generated, which may involve retrieving information from a database to provide the best possible answer to the user's question or request.

[1097] 6. Means for providing the generated response message to the user in voice and text:

[1098] It uses Google Text-to-Speech (gTTS) to convert the generated text response into audio, providing it to the user in both audio and text form, and conveying information both visually and audibly.

[1099] Data Calculation and Processing

[1100] 1. Acquire and convert voice input:

[1101] The user speaks a question to the system, the microphone captures the voice signal, and the data is converted into text data through a speech recognition API.

[1102] 2. Text data analysis:

[1103] The converted text data is sent to a server, where a natural language processing engine analyzes it to understand the user's intent. This analysis uses the GPT-2 model.

[1104] 3. Generate a response message:

[1105] The generative AI model generates the optimal response message that matches the analyzed data. During this generation process, it searches and retrieves the necessary information from the database and determines the content to be provided to the user.

[1106] 4. Providing generated messages:

[1107] The generated response message is converted from text data to voice data again and provided to the user. At the same time, it is also displayed in text format, so the user can receive information in both voice and text.

[1108] Specific examples

[1109] Consider a scenario in which a user connects to a virtual store through smart glasses and inquires about product availability. When the user verbally asks, "Is this product in stock?", the system captures voice data through a microphone and converts the speech into text using the Google Speech Recognition API. The GPT-2 model then analyzes the text data and retrieves the corresponding inventory information from the inventory database. The generated response message, "This product is in stock," is provided to the user in both voice and text.

[1110] Prompt Sentence Examples

[1111] example:

[1112] "Hello, are you looking for something?"

[1113] "What is the most popular item in this store?"

[1114] "Please let me know about this week's sales."

[1115] This system allows users to obtain fast and accurate information even in virtual environments, improving the overall user experience.

[1116] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1117] Step 1:

[1118] A user approaches the system. The device's motion sensor detects the user and the system switches from standby mode to active mode.

[1119] Input: User approach

[1120] Output: Switching the system to active mode

[1121] What happens: The motion sensor reacts, an internal signal from the device triggers the system's active mode, and an avatar appears on the screen and greets the user.

[1122] Step 2:

[1123] The user asks a question by voice, and the device's microphone picks up the voice and captures the audio signal.

[1124] Input: User's voice question

[1125] Output: Audio data

[1126] What it does: A user speaks into a microphone asking a question, such as "Do you have this product in stock?" The microphone picks up the voice signal and stores it as digital audio data.

[1127] Step 3:

[1128] The voice data acquired by the device is converted into text data via the Google Speech Recognition API.

[1129] Input: Audio data

[1130] Output: Text data

[1131] How it works: Audio data is sent to a cloud service, where it is analyzed by the Google Speech Recognition API and converted into corresponding text data.

[1132] Step 4:

[1133] The server receives the text data and analyzes it using a natural language processing engine (GPT-2), which searches a database to understand the user's intent and determine the appropriate response.

[1134] Input: Text data

[1135] Output: Analysis results and generation of optimal response messages

[1136] How it works: The server inputs text data into the GPT-2 model and begins analysis. During the analysis process, the user's question (e.g., "Is it in stock?") is understood and relevant information (e.g., "inventory data") is searched for in the database.

[1137] Step 5:

[1138] The server uses the generative AI model to generate a response message based on the analysis results.

[1139] Input: Analysis results and information obtained from databases

[1140] Output: Response message (text format)

[1141] Specific behavior: The GPT-2 model generates a text message such as "This product is in stock." The generated message is stored as internal data.

[1142] Step 6:

[1143] The device uses Google Text-to-Speech (gTTS) to convert the generated text message into audio data and provide it to the user.

[1144] Input: Text message

[1145] Output: Audio data and text display

[1146] How it works: A text message is entered into gTTS and converted into voice data, which is then played back through the device's speaker and simultaneously displayed on the screen.

[1147] Step 7:

[1148] The user reviews the response message and, if necessary, asks additional questions or provides instructions, and the cycle repeats.

[1149] Input: User confirmation and follow-up questions

[1150] Output: Additional audio data

[1151] What happens: The user listens to the voice response or reads the text, then speaks again if they have any further questions. This process can be repeated as necessary.

[1152] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1153] The present invention provides an interactive guidance system that not only provides users with information quickly and accurately in public facilities, commercial facilities, and event facilities, but also recognizes the user's emotions to provide more personalized responses. This system automatically detects when a user approaches and can provide optimal responses to the user's questions or requests. Furthermore, it utilizes an emotion engine to recognize the user's emotions and respond based on those emotions. Specific embodiments are described in detail below.

[1154] Basic operation

[1155] Reaction when the user approaches

[1156] 1. The device detects the user using a sensor

[1157] The digital guidance system is equipped with motion sensors that automatically detect when a user approaches.

[1158] When the sensor detects the presence of a user, the device switches from standby mode to active mode.

[1159] 2. The device launches the avatar

[1160] When the device switches to active mode, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[1161] The process that receives user input

[1162] 1. The user asks a question

[1163] The user can use voice input or a touch panel to ask questions or make requests, for example, "Please tell me today's event schedule."

[1164] 2. The device gets the user's input

[1165] The device is equipped with a microphone and touch sensor, through which the user's voice input and touch operations are acquired.

[1166] In the case of voice input, the voice is converted into text data in real time by a voice recognition engine.

[1167] Server parsing and generation

[1168] 1. Parse the input

[1169] The device transmits the acquired voice and text data to the server.

[1170] The server uses a natural language processing engine and an emotion engine to analyze the user's intent and emotions. For example, it analyzes the text data "Please tell me the event schedule" as well as the user's tone of voice and facial expressions to identify their emotional state.

[1171] 2. Generate a response message

[1172] Based on the analysis results, the server uses AI to create the optimal response message. Depending on the user's emotional state, a message that takes into account their emotions, such as "Today's event schedule is as follows. Have a great day!", is generated.

[1173] Response display by terminal

[1174] 1. Display a response message

[1175] The server generates a response message and sends it to the terminal.

[1176] The device's avatar will respond to the user with a voice or text message. For example, it might say, "Today's event schedule is as follows," and provide details in voice.

[1177] Specific examples

[1178] For example, imagine a user standing in front of a digital information system installed at the entrance of a department store. As the user approaches, the device's sensors detect the user's approach and an avatar speaks to them, saying, "Hello, is there anything I can help you with?" If the user responds, "Can you tell me where the restaurant is?", the device converts the speech into text and sends that data to the server. The server analyzes this request and, using an emotion engine, infers, for example, that the user is in a hurry based on the user's tone of voice and facial expression. As a result, it generates a specific, situation-appropriate guidance message: "It seems you're in a hurry. The restaurant is on the third floor, at the back on the right." The generated message is sent to the device, and the avatar relays it to the user.

[1179] In this way, the interactive guidance system of the present invention allows users to quickly and accurately obtain the information they need. Furthermore, the introduction of an emotion engine enables personalized responses that take into account the user's emotions, greatly improving user satisfaction.

[1180] The processing flow will be explained below.

[1181] Step 1:

[1182] The device detects the user with a sensor. When a user approaches the digital signage, the motion sensor detects their presence, causing the device to switch from standby mode to active mode.

[1183] Step 2:

[1184] The device activates the avatar. When the sensor detects the user's approach, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[1185] Step 3:

[1186] The user asks a question. The user uses voice input or a touch panel to ask a question or make a request. For example, the user might ask, "Please tell me today's event schedule."

[1187] Step 4:

[1188] The device receives user input. The device's microphone or touch sensor captures the user's question. In the case of voice input, the speech recognition engine converts the voice into text in real time.

[1189] Step 5:

[1190] The device activates an emotion engine to analyze the user's emotions, analyzing voice tone, speaking style, facial expressions, etc. to determine whether the user is angry, in a hurry, having fun, etc.

[1191] Step 6:

[1192] The device sends text data and emotion data to the server, which then sends the acquired text data and emotion analysis results together.

[1193] Step 7:

[1194] The server analyzes the input. The server's natural language processing engine analyzes the text data sent and understands the user's intent. For example, it recognizes a request such as "Please tell me the event schedule."

[1195] Step 8:

[1196] The server generates a response message based on the emotional data. Based on the analysis results, the server uses generation AI to create an appropriate response message. Depending on the user's emotional state, a message that takes emotions into consideration is generated, such as "Today's event schedule is as follows. Have a great day!"

[1197] Step 9:

[1198] The server sends a response message to the terminal. The server then sends the generated response message to the terminal. If the user is in a hurry, it generates a concise and quick message, but if the user is relaxed, it generates a detailed and polite message.

[1199] Step 10:

[1200] The device displays the response to the user. The device's avatar then relays the received response message to the user via voice or text. For example, it might display "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00," and provide details via voice.

[1201] Step 11:

[1202] The user asks a follow-up question. If the user wants to ask more questions, the same process is used to respond to the new request. For example, answer a question like, "Can you also tell me where the restaurant is?". Even for follow-up questions, sentiment analysis is performed and an appropriate message is generated and displayed.

[1203] By implementing the above steps, the interactive guidance system of the present invention can quickly and accurately respond to user questions. Furthermore, the introduction of an emotion engine enables personalized responses that take into account the user's emotions, greatly improving user satisfaction.

[1204] Example 2

[1205] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1206] Conventional guidance systems have difficulty in providing quick and accurate information to users' questions, and are unable to provide personalized responses that reflect the user's emotions. This has led to problems such as lower user satisfaction and a decrease in frequency of use.

[1207] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1208] In this invention, the server includes means for detecting the approach of a user with a sensor, means for receiving user input, means for analyzing the received user input, means for generating a response message based on the analysis result, means for displaying the generated response message to the user, means for recognizing the user's emotion, and means for personalizing the response message based on the recognized emotion. This enables quick and accurate provision of information, and further allows for personalized responses to be provided in accordance with the user's emotion, thereby improving user satisfaction.

[1209] A "sensor" is a device that detects physical changes or movements and captures them as data.

[1210] A "user" is a person or individual who uses the system to obtain information.

[1211] "Approach" refers to the user moving into the system's recognition range.

[1212] "Detecting" means that the sensor recognizes the user's approach and sends that information to the system.

[1213] "Means" is a general term for methods or devices used to achieve a goal.

[1214] "Receiving" means that the system takes input from the user and processes it.

[1215] "Input" refers to the information or instructions a user provides to a system, and can take the form of voice, touch, or other input.

[1216] "Analyzing" means analyzing the received input data and identifying the user's intentions and the information they are looking for.

[1217] "Generating" means creating a new response message based on the analysis results.

[1218] A "response message" is information or a response that the system provides to the user, and may be in the form of text or audio.

[1219] "Displaying" means conveying the generated response message to the user on the screen or by voice.

[1220] "Emotion" refers to the user's psychological state or mood, and is recognized by the system.

[1221] "Recognizing" means that the system captures the user's emotions and uses them for analysis.

[1222] "Personalization" means customizing a response message according to the user's emotions and individual circumstances.

[1223] The present invention is an interactive guidance system that not only provides users with information quickly and accurately in public facilities, commercial facilities, and event facilities, but also recognizes the user's emotions to provide more personalized responses. This system has the function of automatically detecting when a user approaches and providing the most appropriate response to the user's questions or requests.

[1224] First, the device uses sensors to detect the user's approach. Specifically, a motion sensor detects the user's movement, and when the user comes within a certain distance, the system switches to active mode. Then, an avatar displayed on the device speaks to the user, saying, "Hello, is there anything I can help you with?" The avatar is programmed using game engines such as Unity or Unreal Engine.

[1225] Users can ask questions or make requests using voice input or the touch panel. For example, if they ask, "What is today's event schedule?", the microphone picks up the voice and converts it into text in real time using the Google Cloud Speech-to-Text API. The text data is saved in a temporary file and then sent to the server.

[1226] The server uses the Google Cloud Natural Language API to analyze the received text data and parses the user's intent. It also uses emotion engines such as Amazon Comprehend to analyze the user's emotional state based on their tone of voice and facial expressions. Emotional information such as whether the user is in a hurry can also be collected.

[1227] Based on the analysis results, a generative AI (e.g., OpenAI GPT-4) generates an optimal response message. The prompt is "Please tell me the event schedule. The user seems to be in a hurry." The message generated by the AI ​​model corresponds to the emotion, such as "Today's event schedule is as follows. Have a nice day."

[1228] The server then sends the generated response message to the device, and the device communicates the content to the user in voice and text. For example, if a user asks, "Where is the restaurant?", the server analyzes the question and generates a message saying, "You seem to be in a hurry. The restaurant is on the third floor, at the back on the right," which the device then communicates to the user.

[1229] This system not only allows users to obtain accurate information quickly, but also provides personalized responses that take their emotions into consideration. For example, by inputting a prompt such as "Please generate a response for how the guidance system should greet the user when they approach," into the AI ​​model, an appropriate response can be generated.

[1230] As described above, the interactive guidance system of the present invention can provide users with a high level of satisfaction.

[1231] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1232] Interactive guidance system processing flow

[1233] Step 1:

[1234] The device uses a sensor to detect the user's approach.

[1235] What it does: Motion sensors detect user movement within 20 feet.

[1236] Input: Motion information from the external environment.

[1237] Data processing: Sensors convert physical movements into digital signals.

[1238] Output: A signal indicating the user's approach.

[1239] Next step: Send a signal to switch from standby mode to active mode.

[1240] Step 2:

[1241] The device switches to active mode and activates the avatar.

[1242] What happens: The device's display turns on and an avatar appears, saying, "Hello, how can I help you?"

[1243] Input: User approach detection signal from sensor.

[1244] Data processing: The system switches from standby mode to active mode. The avatar's program is activated.

[1245] Output: A voice message to the user.

[1246] Next: Wait for user input.

[1247] Step 3:

[1248] The user inputs a question by voice or via a touch panel.

[1249] Specific behavior: The user asks "What is today's event schedule?" by voice or by entering text using the touch panel.

[1250] Input: User voice or touch input data.

[1251] Data processing: A microphone and touch sensor capture user input and convert it into digital data. Voice input is converted to text using the Google Cloud Speech-to-Text API.

[1252] Output: User questions in text format.

[1253] Next process: The terminal prepares to send text data to the server.

[1254] Step 4:

[1255] The terminal sends the user's input to the server.

[1256] Specific operation: The acquired text data is sent to the server using the HTTPS protocol.

[1257] Input: The user's question in text form.

[1258] Data processing: The text data is divided into packets and sent to the server over a secure communication channel.

[1259] Output: User question data sent to the server.

[1260] Next: The server parses the user's input data.

[1261] Step 5:

[1262] The server parses the input data.

[1263] What it does: It uses the Google Cloud Natural Language API to analyze the content of the user's question, and Amazon Comprehend to analyze the user's sentiment.

[1264] Input: User question data in text format.

[1265] Data processing: Perform natural language processing and sentiment analysis to identify user intent and emotional state.

[1266] Output: Analysis results including user intent (e.g., event schedule question) and emotional state (e.g., in a hurry).

[1267] Next process: Generate a response message based on the analysis results.

[1268] Step 6:

[1269] The server generates a response message.

[1270] Specific operation: Uses generative AI (e.g., OpenAI GPT-4) to generate a response message based on the analysis results.

[1271] Input: Analysis results, including the user's intent and emotional state.

[1272] Data processing: A prompt is input into the generative AI model to generate the optimal response message. Example prompt: "Generate a response sentence for how the guidance system should greet the user when they approach."

[1273] Output: Personalized response messages depending on the emotion.

[1274] Next: Send a response message to the terminal.

[1275] Step 7:

[1276] The server generates a message and sends it to the terminal.

[1277] Specific operation: Sends a response message to the terminal via the HTTPS protocol.

[1278] Input: The generated response message.

[1279] Data processing: Divide the message into packets and send them securely to the terminal.

[1280] Output: Response message sent to the terminal.

[1281] Next process: The terminal displays and transmits the response message to the user.

[1282] Step 8:

[1283] The terminal displays and transmits the response message.

[1284] Specific operation: The avatar will speak the generated response message to the user, and the text "Today's event schedule is as follows" will be displayed on the screen.

[1285] Input: The response message from the server.

[1286] Data processing: A speech synthesis engine is used to convert text into speech, and a program to move the avatar is executed.

[1287] Output: The response message that is conveyed to the user.

[1288] Next: Wait for further user input if necessary.

[1289] (Application example 2)

[1290] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1291] Conventional interactive guidance systems provide uniform information to users, making it difficult to provide appropriate guidance that reflects each user's individual situation and emotions. In particular, in environments where a variety of information is required, such as brick-and-mortar stores, there is a need to respond quickly and individually to user needs. Furthermore, when a user feels lost or confused, a system that can accurately detect the situation and provide appropriate guidance is needed.

[1292] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1293] In this invention, the server includes means for detecting the approach of a user with a detection device, means for acquiring user input, means for analyzing the acquired user input, means for generating response information based on the analysis result using a generative AI model, display means for providing the generated response information to the user, means for recognizing the user's emotional state, voice recognition means for converting voice input into text data, and means for searching for information from a database based on the analyzed user input. This enables personalized guidance according to the user's emotions, and makes it possible to quickly and accurately provide the information users need even in physical stores.

[1294] A "detection device" is a device that detects the approach of a user, and specifically refers to a motion sensor or camera.

[1295] "User input" refers to information or questions provided by a user, which may be obtained in the form of voice, text, touch, or other input.

[1296] The "analysis means" is a means for understanding and analyzing acquired user input, and utilizes natural language processing and emotion recognition technology.

[1297] A "generative artificial intelligence model" is an artificial intelligence technique for generating appropriate response information based on the results obtained using analytical means.

[1298] "Response information" is information or answers provided in response to user input, and is generated by a generative AI model.

[1299] The "display means" is a means for providing the generated response information to the user, and refers to a display, an audio output device, or the like.

[1300] The "emotional state recognition means" is a means for recognizing the user's emotions from facial expressions, tone of voice, etc., and utilizes an emotion analysis engine or the like.

[1301] The "voice recognition means" is a means for converting voice input into character data, and utilizes a voice recognition engine.

[1302] The "information retrieval means" is a means for retrieving necessary information from a database based on the analyzed user input.

[1303] A "database" is a data storage system that stores necessary information and serves as a source of information for responding to user inquiries.

[1304] The present invention relates to an interactive guidance system that quickly provides appropriate information to users in a physical store. The system includes the following main components:

[1305] Detection device

[1306] The system includes a detection device for detecting the approach of a user. This detection device can be a motion sensor or a camera. For example, the camera detects the user's movement and detects that the user is approaching the system.

[1307] Getting User Input

[1308] User input is acquired in the form of voice, touch operation, and text. Voice input is acquired using a microphone, and touch operation is performed through a touch panel. The acquired voice input is converted into text data by a voice recognition engine.

[1309] Analysis means

[1310] The acquired user input is sent to the server and analyzed by a natural language processing engine and an emotion recognition engine. The natural language processing engine analyzes the input text data and understands the content of the user's request or question. The emotion recognition engine recognizes emotions from the user's tone of voice and facial expressions.

[1311] Response generation using generative artificial intelligence models

[1312] Based on the analysis results, the generative AI model generates optimal response information that is tailored to the user's intent and emotional state and is designed to address their individual needs, such as clearly explaining the solution if the user is confused.

[1313] Display means

[1314] The generated response information is provided to the user via a display and an audio output device. The response is displayed in text format on the display, and if an audio output device is used, audio guidance is also provided.

[1315] Example of a system

[1316] A concrete example of such a system is a physical store guidance system using smart glasses. In this case, facial recognition and emotion analysis are performed while the user wears the glasses and moves around the store. When the user asks, "Where is this item?", a speech recognition engine converts the question into text data, and a generative AI model generates a response such as, "Let's get started right away. Go straight and it's on the next shelf on the right."

[1317] Prompt Sentence Examples

[1318] Examples of prompts to input to a generative AI model include:

[1319] "User's query: 'Where is this item?' Sentiment analysis: 'Confused' Generate the best response."

[1320] In this way, by implementing the interactive guidance system of the present invention, personalized guidance can be provided according to the user's emotions, improving the customer experience in a physical store.

[1321] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1322] Step 1:

[1323] The device detects the user's approach

[1324] Input: User approach status

[1325] Operation: A motion sensor or camera installed on the device detects the user's approach. The camera captures the user's movements and recognizes that the user is approaching the system. Output: The user's presence is detected.

[1326] Step 2:

[1327] The device switches to active mode

[1328] Input: User approach information

[1329] Action: The device switches from standby mode to active mode, and the avatar displayed on the screen comes to life. The avatar speaks to the user, saying, "Hello, how can I help you?" Output: Readiness for user response

[1330] Step 3:

[1331] Getting user input

[1332] Input: User's voice input or touch actions

[1333] Action: The user asks a question or makes a request. For example, "Where is this item?" The device's microphone picks up the voice, or the touch panel receives touch input. Output: The captured user input data.

[1334] Step 4:

[1335] Converting voice input to text data

[1336] Input: User voice input

[1337] Operation: The acquired voice input is converted into text data in real time by the voice recognition engine on the device. Output: Text data converted from voice

[1338] Step 5:

[1339] The server parses the input data

[1340] Input: Acquired text data

[1341] Operation: The server analyzes the text data using a natural language processing engine to understand the user's intention and the content of the question. At the same time, the emotion recognition engine analyzes the user's tone of voice and facial expressions to identify their emotional state. Output: Analyzed user intention and emotional information

[1342] Step 6:

[1343] Generating response information using a generative AI model

[1344] Input: Analyzed user intent and emotion information

[1345] Operation: The server's AI model generates the optimal response information based on the analysis results. For example, if the user is confused, it generates a response such as, "Let's start the guide right away. Go straight and it's on the next shelf on the right." Output: Generated response information

[1346] Step 7:

[1347] Display response information

[1348] Input: Generated response information

[1349] Operation: Provides response information to the user through the terminal's display and audio output device. The response is displayed in text format on the display, and is played back aloud when the audio output device is used. Output: Response information presented to the user

[1350] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1351] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1352] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1353] [Fourth embodiment]

[1354] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1355] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1356] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1357] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1358] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1359] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1360] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1361] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1362] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1363] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1364] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1365] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1366] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1367] The present invention provides an interactive guidance system that quickly and accurately provides information to users in public facilities, commercial facilities, and event facilities. This system automatically detects when a user approaches and can provide optimal responses to the user's questions or requests. Specific embodiments are described in detail below.

[1368] Reaction when the user approaches

[1369] 1. The device detects the user using a sensor

[1370] The digital guidance system is equipped with motion sensors that automatically detect when a user approaches.

[1371] When the sensor detects the presence of a user, the device switches from standby mode to active mode.

[1372] 2. The device launches the avatar

[1373] When the device switches to active mode, the avatar displayed on the device screen comes to life and greets the user.

[1374] For example, the avatar might say, "Hello, is there anything I can help you with?"

[1375] The process that receives user input

[1376] 1. The user asks a question

[1377] Users can use voice input or a touch panel to ask questions or make requests. For example, they can ask, "Please tell me today's event schedule."

[1378] 2. The device gets the user's input

[1379] The device is equipped with a microphone and touch sensor, through which the user's voice input and touch operations are acquired.

[1380] In the case of voice input, the voice is converted into text data in real time by a voice recognition engine.

[1381] Server parsing and generation

[1382] 1. Parse the input

[1383] The device transmits the acquired voice and text data to the server.

[1384] The server uses a natural language processing engine to analyze the user's intent. For example, it analyzes text data such as "Please tell me the event schedule" and identifies the information the user wants to know.

[1385] 2. Generate a response message

[1386] Based on the analysis results, the server uses generation AI to create the optimal response message.

[1387] For example, information about the event schedule may be retrieved from a database and a response message such as "Today's event schedule is as follows..." may be generated.

[1388] Response display by terminal

[1389] 1. Display a response message

[1390] The server generates a response message and sends it to the terminal.

[1391] The device's avatar will convey the response message to the user via voice or text.

[1392] For example, "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00."

[1393] Specific examples

[1394] For example, imagine a user standing in front of a digital information system installed at the entrance of a department store. As the user approaches, the device's sensors detect the user and an avatar speaks, saying, "Hello, is there anything I can help you with?" If the user responds, "Can you tell me where the restaurant is?", the device converts the speech into text and sends it to the server. The server analyzes this request, retrieves the restaurant's location information from a database, and generates an optimal guidance message. The generated message is sent to the device, and the avatar responds, "The restaurant is on the third floor, at the back on the right."

[1395] In this way, the interactive guide system of the present invention allows users to quickly and accurately obtain the information they need, which can greatly improve visitor satisfaction.

[1396] The processing flow will be explained below.

[1397] Step 1:

[1398] The device detects the user with a sensor. When a user approaches the digital signage, the motion sensor detects their presence, causing the device to switch from standby mode to active mode.

[1399] Step 2:

[1400] The device activates the avatar. When the sensor detects the user's approach, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[1401] Step 3:

[1402] The user asks a question. The user uses voice input or a touch panel to ask a question or make a request. For example, the user might ask, "Please tell me today's event schedule."

[1403] Step 4:

[1404] The device receives user input. The device's microphone or touch sensor captures the user's question. In the case of voice input, the speech recognition engine converts the voice into text in real time.

[1405] Step 5:

[1406] The terminal sends the text data to the server. The acquired text data is then sent from the terminal to the server.

[1407] Step 6:

[1408] The server analyzes the input. The server's natural language processing engine analyzes the text data sent and understands the user's intent. For example, it recognizes a request such as "Please tell me the event schedule."

[1409] Step 7:

[1410] The server retrieves the relevant data. Based on the analysis results, the server searches and retrieves relevant information from the database. In this case, it retrieves the event schedule information.

[1411] Step 8:

[1412] The server generates a response message. The AI ​​then creates the optimal response message based on the acquired information. For example, it generates a message such as, "Today's event schedule is a yoga class from 10:00 and a concert from 14:00."

[1413] Step 9:

[1414] The server sends a response message to the terminal. The generated response message is sent from the server to the terminal.

[1415] Step 10:

[1416] The device displays the response to the user. The device's avatar conveys the received response message to the user via voice or text. For example, it may display "Today's event schedule is as follows" and provide details via voice.

[1417] Step 11:

[1418] The user asks a follow-up question. If the user wants to ask more questions, the same process is used to fulfill the new request. For example, "Can you also tell me where the restaurant is?"

[1419] By performing the above steps, the interactive guidance system of the present invention can quickly and accurately respond to user questions.

[1420] Example 1

[1421] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1422] Conventional guidance systems have the problem of being unable to detect the user's approach and therefore unable to provide the necessary information quickly. Furthermore, they lack an adequate interface for generating appropriate responses to user input, resulting in low user satisfaction. Furthermore, conventional systems have the problem of being unable to provide the appropriate information the user is looking for due to the low accuracy of voice input and inaccurate analysis results. To solve these problems, the development of an interactive and accurate guidance system is required.

[1423] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1424] In this invention, the server includes means for detecting the approach of a user with a sensor, means for switching from standby mode to active mode when the user is detected, means for activating an avatar in active mode and displaying a greeting, means for receiving user input, means for converting the received user input into text data with a voice recognition engine, means for transmitting the text data to the server and analyzing it, means for generating an appropriate response message based on the analysis result using a generative AI model, and means for displaying the generated response message to the user in voice and text. This makes it possible to quickly detect when a user is approaching and provide an appropriate response, thereby improving user satisfaction.

[1425] A "means for detecting user proximity by a sensor" is a device for detecting when a user enters the range of the system.

[1426] "Standby mode" refers to a state in which the system is in a low-power standby state and is monitoring external inputs.

[1427] "Active mode" is a mode in which the system goes into full operation after detecting the approach of a user.

[1428] The "means for activating an avatar and displaying a greeting" refers to a means for displaying an anthropomorphized character on the system screen and greeting the user by voice or text.

[1429] The "means for receiving user input" is a device for receiving questions and instructions from the user through voice input or a touch panel.

[1430] The "means for converting into text data using a voice recognition engine" is software that analyzes the user's voice input in real time and converts it into text data.

[1431] The "means for transmitting text data to a server" is a function for transmitting text data acquired from a user to a server via the Internet or the like.

[1432] The "analysis means" is software that allows the server to analyze text data using a natural language processing engine and understand the user's intent.

[1433] "Means for generating using a generative AI model" refers to the process of using generative AI to automatically create an optimal response message based on the analysis results.

[1434] The "means for displaying the generated response message to the user in voice and text" refers to a means for displaying the response message generated by the server on the terminal and conveying it to the user in voice and text through the avatar.

[1435] The present invention is an interactive guidance system that provides users with information quickly and accurately in public, commercial, and event facilities. The system automatically detects when a user approaches and provides optimal responses to the user's questions and requests.

[1436] Hardware and software used

[1437] The following hardware and software are used to implement the system:

[1438] Motion sensor: Used to detect the user's approach.

[1439] Microphone: Used to receive audio input.

[1440] Touch panel: Used to receive user touch input.

[1441] Display: Used for displaying avatars and text.

[1442] Server: Parses the data and generates a response message.

[1443] Speech recognition engine: Converts the user's speech into text.

[1444] Natural Language Processing (NLP) engine: Parses user input.

[1445] Generative AI model: Generates optimal response messages based on analysis results.

[1446] Program processing explanation

[1447] The operation of the system is as follows.

[1448] 1. User approach detection

[1449] The device uses a motion sensor to detect when the user enters a certain range, which causes the device to switch from standby mode to active mode.

[1450] 2. Launching the avatar

[1451] When in active mode, an avatar appears on the device's display and greets the user, for example, "Hello, how can I help you?"

[1452] 3. Accepting User Input

[1453] The user can ask questions or make requests using voice input or a touch panel. For example, they can input, "Please tell me today's event schedule."

[1454] 4. Acquiring input information

[1455] User input is obtained through the device's microphone or touch sensor. In the case of voice input, the voice recognition engine converts it into text data.

[1456] 5. Data Analysis

[1457] The device sends the acquired text data to the server, which then uses a natural language processing engine to analyze the data and understand the user's intent. For example, the server might analyze a request such as "Please tell me the event schedule."

[1458] 6. Generating a Response Message

[1459] The server uses the generative AI model to generate an appropriate response message based on the analysis results, and the generated message is sent from the server to the device.

[1460] 7. Displaying the Response

[1461] The device displays the received response message to the user in voice and text. For example, an avatar may say, "Today's event schedule is a yoga class from 10:00 and a concert from 14:00."

[1462] Adding specific examples

[1463] For example, when a user stands in front of a digital information system installed at the entrance of a department store, the device's sensors detect the user, and an avatar on the screen speaks to them, saying, "Hello, is there something I can help you with?" If the user says, "Can you tell me where the restaurant is?", the device converts the speech into text and sends it to the server. The server analyzes the request, retrieves restaurant information from a database, and generates a response message, stating, "The restaurant is on the third floor, at the back on the right."

[1464] Prompt Sentence Examples

[1465] An example of an input prompt for a generative AI model might be, "Please provide a clear and easy-to-understand answer to the question asked by the user." Based on this prompt, the generative AI model generates the optimal response message.

[1466] This system therefore provides users with quick and accurate information, improving satisfaction with facility use.

[1467] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1468] Step 1:

[1469] User approach detection

[1470] The device detects the user using a sensor

[1471] Input: The motion sensor detects the user's movement.

[1472] Specific operation: The motion sensor detects the user's position and movement and sends a signal to the device.

[1473] Output: A signal is generated that switches the terminal from standby mode to active mode.

[1474] Step 2:

[1475] Launching the Avatar

[1476] The device launches the avatar

[1477] Input: The terminal receives a signal to switch to active mode.

[1478] Specific behavior: An avatar appears on the device display and greets the user using programmed motions.

[1479] Output: The avatar will say and read "Hello, how can I help you?"

[1480] Step 3:

[1481] Accepting user input

[1482] The user enters a question or request

[1483] Input: The user enters a question or request using voice input or a touch panel.

[1484] Specific operation: The user asks a question such as "Please tell me today's event schedule" using the microphone or touch panel.

[1485] Output: User voice or touch data is captured.

[1486] Step 4:

[1487] Obtaining input information

[1488] The device obtains the user's input information.

[1489] Input: For voice input, the user's voice data. For touch input, the touch operation data.

[1490] How it works: A highly sensitive microphone picks up voices, and a voice recognition engine converts them into text data. Touch input data is also sent from the sensor.

[1491] Output: User input data in text format is generated.

[1492] Step 5:

[1493] Data analysis

[1494] The device sends the acquired data to the server.

[1495] Input: User-entered data in text format.

[1496] Specific operation: The device sends data to the server in real time.

[1497] Output: The server receives the user's text input data.

[1498] The server analyzes the data

[1499] Input: The text data sent to the server.

[1500] Specific operation: The natural language processing engine analyzes text data and understands the user's intent. For example, it analyzes a request such as "Please tell me the event schedule."

[1501] Output: Parsed user intent and required information identified.

[1502] Step 6:

[1503] Generating a response message

[1504] The server generates a response message

[1505] Input: Parsed user intent.

[1506] What it does: Uses a generative AI model to create an optimal response message, retrieving relevant information from the necessary databases.

[1507] Output: The generated response message is formed.

[1508] Step 7:

[1509] Viewing the response

[1510] The server sends a response message to the device.

[1511] Input: The generated response message.

[1512] Specific operation: The server sends the generated response message to the terminal.

[1513] Output: The terminal receives the response message.

[1514] The terminal displays a response message to the user.

[1515] Input: The response message received from the server.

[1516] Specific operation: The device's avatar conveys the received message to the user via voice and text. For example, the avatar may display and speak, "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00."

[1517] Output: The user receives the necessary information visually and audibly.

[1518] (Application example 1)

[1519] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1520] Conventional guidance systems often struggle to respond quickly and accurately to user questions. In particular, when it comes to effectively utilizing voice input, the user experience is often compromised by inaccurate voice recognition technology and natural language processing technology that cannot accurately analyze user intent. Furthermore, responses to user voice input are monotonous, resulting in insufficient user interaction, making it difficult to improve visitor satisfaction.

[1521] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1522] In this invention, the server includes means for detecting the approach of a user with a sensor, means for receiving user input by voice, means for converting the received user voice into text data, means for analyzing the text data, means for generating a response message using a generative AI model based on the analysis results, and means for providing the generated response message to the user in voice and text, thereby enabling the server to provide quick and accurate responses to user questions and requests and realizing an interactive user experience.

[1523] The "means for detecting the approach of a user by a sensor" refers to a device or function for detecting the presence of a user when the user approaches the system.

[1524] The "means for receiving user input by voice" refers to a device or function for acquiring voice data provided by the user.

[1525] The "means for converting received user voice into text data" refers to a process or device for analyzing the user voice data and converting it into text format data.

[1526] "Means for analyzing text data" refers to software or algorithms that, after speech input has been converted into text data, analyze the text data and understand the user's intent.

[1527] A "generative means using a generative AI model" is an artificial intelligence model or algorithm for generating an appropriate response message based on the analyzed user text data.

[1528] A "means for providing a generated response message to a user in both audio and text" is a device or process for providing a generated response message to a user in both audio and text form.

[1529] The present invention provides an interactive guidance system that enables users to quickly and accurately obtain information within a virtual environment. Specific embodiments will be described below.

[1530] System configuration

[1531] The system consists of the following main components:

[1532] 1. How to use a sensor to detect the user's approach:

[1533] A motion sensor is used to detect when a user approaches the system, readying the system for user interaction.

[1534] 2. A way to receive user input by voice:

[1535] A microphone is used to capture the user's voice input, which is then analyzed.

[1536] 3. A means of converting received user speech into text data:

[1537] Use a speech recognition API (for example, Google Speech Recognition API) to convert the acquired speech data into text data in real time.

[1538] 4. Means of analyzing text data:

[1539] The converted text data is then analyzed by a natural language processing engine and processed to understand the user's intent. Specifically, a Transformer-based generative AI model (e.g., GPT-2) is used.

[1540] 5. Generative AI model generation method:

[1541] Based on the analysis results, an appropriate response message is generated, which may involve retrieving information from a database to provide the best possible answer to the user's question or request.

[1542] 6. Means for providing the generated response message to the user in voice and text:

[1543] It uses Google Text-to-Speech (gTTS) to convert the generated text response into audio, providing it to the user in both audio and text form, and conveying information both visually and audibly.

[1544] Data Calculation and Processing

[1545] 1. Acquire and convert voice input:

[1546] The user speaks a question to the system, the microphone captures the voice signal, and the data is converted into text data through a speech recognition API.

[1547] 2. Text data analysis:

[1548] The converted text data is sent to a server, where a natural language processing engine analyzes it to understand the user's intent. This analysis uses the GPT-2 model.

[1549] 3. Generate a response message:

[1550] The generative AI model generates the optimal response message that matches the analyzed data. During this generation process, it searches and retrieves the necessary information from the database and determines the content to be provided to the user.

[1551] 4. Providing generated messages:

[1552] The generated response message is converted from text data to voice data again and provided to the user. At the same time, it is also displayed in text format, so the user can receive information in both voice and text.

[1553] Specific examples

[1554] Consider a scenario in which a user connects to a virtual store through smart glasses and inquires about product availability. When the user verbally asks, "Is this product in stock?", the system captures voice data through a microphone and converts the speech into text using the Google Speech Recognition API. The GPT-2 model then analyzes the text data and retrieves the corresponding inventory information from the inventory database. The generated response message, "This product is in stock," is provided to the user in both voice and text.

[1555] Prompt Sentence Examples

[1556] example:

[1557] "Hello, are you looking for something?"

[1558] "What is the most popular item in this store?"

[1559] "Please let me know about this week's sales."

[1560] This system allows users to obtain fast and accurate information even in virtual environments, improving the overall user experience.

[1561] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1562] Step 1:

[1563] A user approaches the system. The device's motion sensor detects the user and the system switches from standby mode to active mode.

[1564] Input: User approach

[1565] Output: Switching the system to active mode

[1566] What happens: The motion sensor reacts, an internal signal from the device triggers the system's active mode, and an avatar appears on the screen and greets the user.

[1567] Step 2:

[1568] The user asks a question by voice, and the device's microphone picks up the voice and captures the audio signal.

[1569] Input: User's voice question

[1570] Output: Audio data

[1571] What it does: A user speaks into a microphone asking a question, such as "Do you have this product in stock?" The microphone picks up the voice signal and stores it as digital audio data.

[1572] Step 3:

[1573] The voice data acquired by the device is converted into text data via the Google Speech Recognition API.

[1574] Input: Audio data

[1575] Output: Text data

[1576] How it works: Audio data is sent to a cloud service, where it is analyzed by the Google Speech Recognition API and converted into corresponding text data.

[1577] Step 4:

[1578] The server receives the text data and analyzes it using a natural language processing engine (GPT-2), which searches a database to understand the user's intent and determine the appropriate response.

[1579] Input: Text data

[1580] Output: Analysis results and generation of optimal response messages

[1581] How it works: The server inputs text data into the GPT-2 model and begins analysis. During the analysis process, the user's question (e.g., "Is it in stock?") is understood and relevant information (e.g., "inventory data") is searched for in the database.

[1582] Step 5:

[1583] The server uses the generative AI model to generate a response message based on the analysis results.

[1584] Input: Analysis results and information obtained from databases

[1585] Output: Response message (text format)

[1586] Specific behavior: The GPT-2 model generates a text message such as "This product is in stock." The generated message is stored as internal data.

[1587] Step 6:

[1588] The device uses Google Text-to-Speech (gTTS) to convert the generated text message into audio data and provide it to the user.

[1589] Input: Text message

[1590] Output: Audio data and text display

[1591] How it works: A text message is entered into gTTS and converted into voice data, which is then played back through the device's speaker and simultaneously displayed on the screen.

[1592] Step 7:

[1593] The user reviews the response message and, if necessary, asks additional questions or provides instructions, and the cycle repeats.

[1594] Input: User confirmation and follow-up questions

[1595] Output: Additional audio data

[1596] What happens: The user listens to the voice response or reads the text, then speaks again if they have any further questions. This process can be repeated as necessary.

[1597] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1598] The present invention provides an interactive guidance system that not only provides users with information quickly and accurately in public facilities, commercial facilities, and event facilities, but also recognizes the user's emotions to provide more personalized responses. This system automatically detects when a user approaches and can provide optimal responses to the user's questions or requests. Furthermore, it utilizes an emotion engine to recognize the user's emotions and respond based on those emotions. Specific embodiments are described in detail below.

[1599] Basic operation

[1600] Reaction when the user approaches

[1601] 1. The device detects the user using a sensor

[1602] The digital guidance system is equipped with motion sensors that automatically detect when a user approaches.

[1603] When the sensor detects the presence of a user, the device switches from standby mode to active mode.

[1604] 2. The device launches the avatar

[1605] When the device switches to active mode, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[1606] The process that receives user input

[1607] 1. The user asks a question

[1608] The user can use voice input or a touch panel to ask questions or make requests, for example, "Please tell me today's event schedule."

[1609] 2. The device gets the user's input

[1610] The device is equipped with a microphone and touch sensor, through which the user's voice input and touch operations are acquired.

[1611] In the case of voice input, the voice is converted into text data in real time by a voice recognition engine.

[1612] Server parsing and generation

[1613] 1. Parse the input

[1614] The device transmits the acquired voice and text data to the server.

[1615] The server uses a natural language processing engine and an emotion engine to analyze the user's intent and emotions. For example, it analyzes the text data "Please tell me the event schedule" as well as the user's tone of voice and facial expressions to identify their emotional state.

[1616] 2. Generate a response message

[1617] Based on the analysis results, the server uses AI to create the optimal response message. Depending on the user's emotional state, a message that takes into account their emotions, such as "Today's event schedule is as follows. Have a great day!", is generated.

[1618] Response display by terminal

[1619] 1. Display a response message

[1620] The server generates a response message and sends it to the terminal.

[1621] The device's avatar will respond to the user with a voice or text message. For example, it might say, "Today's event schedule is as follows," and provide details in voice.

[1622] Specific examples

[1623] For example, imagine a user standing in front of a digital information system installed at the entrance of a department store. As the user approaches, the device's sensors detect the user's approach and an avatar speaks to them, saying, "Hello, is there anything I can help you with?" If the user responds, "Can you tell me where the restaurant is?", the device converts the speech into text and sends that data to the server. The server analyzes this request and, using an emotion engine, infers, for example, that the user is in a hurry based on the user's tone of voice and facial expression. As a result, it generates a specific, situation-appropriate guidance message: "It seems you're in a hurry. The restaurant is on the third floor, at the back on the right." The generated message is sent to the device, and the avatar relays it to the user.

[1624] In this way, the interactive guidance system of the present invention allows users to quickly and accurately obtain the information they need. Furthermore, the introduction of an emotion engine enables personalized responses that take into account the user's emotions, greatly improving user satisfaction.

[1625] The processing flow will be explained below.

[1626] Step 1:

[1627] The device detects the user with a sensor. When a user approaches the digital signage, the motion sensor detects their presence, causing the device to switch from standby mode to active mode.

[1628] Step 2:

[1629] The device activates the avatar. When the sensor detects the user's approach, the avatar displayed on the device screen comes to life and speaks to the user, saying, "Hello, is there anything I can help you with?"

[1630] Step 3:

[1631] The user asks a question. The user uses voice input or a touch panel to ask a question or make a request. For example, the user might ask, "Please tell me today's event schedule."

[1632] Step 4:

[1633] The device receives user input. The device's microphone or touch sensor captures the user's question. In the case of voice input, the speech recognition engine converts the voice into text in real time.

[1634] Step 5:

[1635] The device activates an emotion engine to analyze the user's emotions, analyzing voice tone, speaking style, facial expressions, etc. to determine whether the user is angry, in a hurry, having fun, etc.

[1636] Step 6:

[1637] The device sends text data and emotion data to the server, which then sends the acquired text data and emotion analysis results together.

[1638] Step 7:

[1639] The server analyzes the input. The server's natural language processing engine analyzes the text data sent and understands the user's intent. For example, it recognizes a request such as "Please tell me the event schedule."

[1640] Step 8:

[1641] The server generates a response message based on the emotional data. Based on the analysis results, the server uses generation AI to create an appropriate response message. Depending on the user's emotional state, a message that takes emotions into consideration is generated, such as "Today's event schedule is as follows. Have a great day!"

[1642] Step 9:

[1643] The server sends a response message to the terminal. The server then sends the generated response message to the terminal. If the user is in a hurry, it generates a concise and quick message, but if the user is relaxed, it generates a detailed and polite message.

[1644] Step 10:

[1645] The device displays the response to the user. The device's avatar then relays the received response message to the user via voice or text. For example, it might display "Today's event schedule includes a yoga class from 10:00 and a concert from 14:00," and provide details via voice.

[1646] Step 11:

[1647] The user asks a follow-up question. If the user wants to ask more questions, the same process is used to respond to the new request. For example, answer a question like, "Can you also tell me where the restaurant is?". Even for follow-up questions, sentiment analysis is performed and an appropriate message is generated and displayed.

[1648] By implementing the above steps, the interactive guidance system of the present invention can quickly and accurately respond to user questions. Furthermore, the introduction of an emotion engine enables personalized responses that take into account the user's emotions, greatly improving user satisfaction.

[1649] Example 2

[1650] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1651] Conventional guidance systems have difficulty in providing quick and accurate information to users' questions, and are unable to provide personalized responses that reflect the user's emotions. This has led to problems such as lower user satisfaction and a decrease in frequency of use.

[1652] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1653] In this invention, the server includes means for detecting the approach of a user with a sensor, means for receiving user input, means for analyzing the received user input, means for generating a response message based on the analysis result, means for displaying the generated response message to the user, means for recognizing the user's emotion, and means for personalizing the response message based on the recognized emotion. This enables quick and accurate provision of information, and further allows for personalized responses to be provided in accordance with the user's emotion, thereby improving user satisfaction.

[1654] A "sensor" is a device that detects physical changes or movements and captures them as data.

[1655] A "user" is a person or individual who uses the system to obtain information.

[1656] "Approach" refers to the user moving into the system's recognition range.

[1657] "Detecting" means that the sensor recognizes the user's approach and sends that information to the system.

[1658] "Means" is a general term for methods or devices used to achieve a goal.

[1659] "Receiving" means that the system takes input from the user and processes it.

[1660] "Input" refers to the information or instructions a user provides to a system, and can take the form of voice, touch, or other input.

[1661] "Analyzing" means analyzing the received input data and identifying the user's intentions and the information they are looking for.

[1662] "Generating" means creating a new response message based on the analysis results.

[1663] A "response message" is information or a response that the system provides to the user, and may be in the form of text or audio.

[1664] "Displaying" means conveying the generated response message to the user on the screen or by voice.

[1665] "Emotion" refers to the user's psychological state or mood, and is recognized by the system.

[1666] "Recognizing" means that the system captures the user's emotions and uses them for analysis.

[1667] "Personalization" means customizing a response message according to the user's emotions and individual circumstances.

[1668] The present invention is an interactive guidance system that not only provides users with information quickly and accurately in public facilities, commercial facilities, and event facilities, but also recognizes the user's emotions to provide more personalized responses. This system has the function of automatically detecting when a user approaches and providing the most appropriate response to the user's questions or requests.

[1669] First, the device uses sensors to detect the user's approach. Specifically, a motion sensor detects the user's movement, and when the user comes within a certain distance, the system switches to active mode. Then, an avatar displayed on the device speaks to the user, saying, "Hello, is there anything I can help you with?" The avatar is programmed using game engines such as Unity or Unreal Engine.

[1670] Users can ask questions or make requests using voice input or the touch panel. For example, if they ask, "What is today's event schedule?", the microphone picks up the voice and converts it into text in real time using the Google Cloud Speech-to-Text API. The text data is saved in a temporary file and then sent to the server.

[1671] The server uses the Google Cloud Natural Language API to analyze the received text data and parses the user's intent. It also uses emotion engines such as Amazon Comprehend to analyze the user's emotional state based on their tone of voice and facial expressions. Emotional information such as whether the user is in a hurry can also be collected.

[1672] Based on the analysis results, a generative AI (e.g., OpenAI GPT-4) generates an optimal response message. The prompt is "Please tell me the event schedule. The user seems to be in a hurry." The message generated by the AI ​​model corresponds to the emotion, such as "Today's event schedule is as follows. Have a nice day."

[1673] The server then sends the generated response message to the device, and the device communicates the content to the user in voice and text. For example, if a user asks, "Where is the restaurant?", the server analyzes the question and generates a message saying, "You seem to be in a hurry. The restaurant is on the third floor, at the back on the right," which the device then communicates to the user.

[1674] This system not only allows users to obtain accurate information quickly, but also provides personalized responses that take their emotions into consideration. For example, by inputting a prompt such as "Please generate a response for how the guidance system should greet the user when they approach," into the AI ​​model, an appropriate response can be generated.

[1675] As described above, the interactive guidance system of the present invention can provide users with a high level of satisfaction.

[1676] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1677] Interactive guidance system processing flow

[1678] Step 1:

[1679] The device uses a sensor to detect the user's approach.

[1680] What it does: Motion sensors detect user movement within 20 feet.

[1681] Input: Motion information from the external environment.

[1682] Data processing: Sensors convert physical movements into digital signals.

[1683] Output: A signal indicating the user's approach.

[1684] Next step: Send a signal to switch from standby mode to active mode.

[1685] Step 2:

[1686] The device switches to active mode and activates the avatar.

[1687] What happens: The device's display turns on and an avatar appears, saying, "Hello, how can I help you?"

[1688] Input: User approach detection signal from sensor.

[1689] Data processing: The system switches from standby mode to active mode. The avatar's program is activated.

[1690] Output: A voice message to the user.

[1691] Next: Wait for user input.

[1692] Step 3:

[1693] The user inputs a question by voice or via a touch panel.

[1694] Specific behavior: The user asks "What is today's event schedule?" by voice or by entering text using the touch panel.

[1695] Input: User voice or touch input data.

[1696] Data processing: A microphone and touch sensor capture user input and convert it into digital data. Voice input is converted to text using the Google Cloud Speech-to-Text API.

[1697] Output: User questions in text format.

[1698] Next process: The terminal prepares to send text data to the server.

[1699] Step 4:

[1700] The terminal sends the user's input to the server.

[1701] Specific operation: The acquired text data is sent to the server using the HTTPS protocol.

[1702] Input: The user's question in text form.

[1703] Data processing: The text data is divided into packets and sent to the server over a secure communication channel.

[1704] Output: User question data sent to the server.

[1705] Next: The server parses the user's input data.

[1706] Step 5:

[1707] The server parses the input data.

[1708] What it does: It uses the Google Cloud Natural Language API to analyze the content of the user's question, and Amazon Comprehend to analyze the user's sentiment.

[1709] Input: User question data in text format.

[1710] Data processing: Perform natural language processing and sentiment analysis to identify user intent and emotional state.

[1711] Output: Analysis results including user intent (e.g., event schedule question) and emotional state (e.g., in a hurry).

[1712] Next process: Generate a response message based on the analysis results.

[1713] Step 6:

[1714] The server generates a response message.

[1715] Specific operation: Uses generative AI (e.g., OpenAI GPT-4) to generate a response message based on the analysis results.

[1716] Input: Analysis results, including the user's intent and emotional state.

[1717] Data processing: A prompt is input into the generative AI model to generate the optimal response message. Example prompt: "Generate a response sentence for how the guidance system should greet the user when they approach."

[1718] Output: Personalized response messages depending on the emotion.

[1719] Next: Send a response message to the terminal.

[1720] Step 7:

[1721] The server generates a message and sends it to the terminal.

[1722] Specific operation: Sends a response message to the terminal via the HTTPS protocol.

[1723] Input: The generated response message.

[1724] Data processing: Divide the message into packets and send them securely to the terminal.

[1725] Output: Response message sent to the terminal.

[1726] Next process: The terminal displays and transmits the response message to the user.

[1727] Step 8:

[1728] The terminal displays and transmits the response message.

[1729] Specific operation: The avatar will speak the generated response message to the user, and the text "Today's event schedule is as follows" will be displayed on the screen.

[1730] Input: The response message from the server.

[1731] Data processing: A speech synthesis engine is used to convert text into speech, and a program to move the avatar is executed.

[1732] Output: The response message that is conveyed to the user.

[1733] Next: Wait for further user input if necessary.

[1734] (Application example 2)

[1735] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1736] Conventional interactive guidance systems provide uniform information to users, making it difficult to provide appropriate guidance that reflects each user's individual situation and emotions. In particular, in environments where a variety of information is required, such as brick-and-mortar stores, there is a need to respond quickly and individually to user needs. Furthermore, when a user feels lost or confused, a system that can accurately detect the situation and provide appropriate guidance is needed.

[1737] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1738] In this invention, the server includes means for detecting the approach of a user with a detection device, means for acquiring user input, means for analyzing the acquired user input, means for generating response information based on the analysis result using a generative AI model, display means for providing the generated response information to the user, means for recognizing the user's emotional state, voice recognition means for converting voice input into text data, and means for searching for information from a database based on the analyzed user input. This enables personalized guidance according to the user's emotions, and makes it possible to quickly and accurately provide the information users need even in physical stores.

[1739] A "detection device" is a device that detects the approach of a user, and specifically refers to a motion sensor or camera.

[1740] "User input" refers to information or questions provided by a user, which may be obtained in the form of voice, text, touch, or other input.

[1741] The "analysis means" is a means for understanding and analyzing acquired user input, and utilizes natural language processing and emotion recognition technology.

[1742] A "generative artificial intelligence model" is an artificial intelligence technique for generating appropriate response information based on the results obtained using analytical means.

[1743] "Response information" is information or answers provided in response to user input, and is generated by a generative AI model.

[1744] The "display means" is a means for providing the generated response information to the user, and refers to a display, an audio output device, or the like.

[1745] The "emotional state recognition means" is a means for recognizing the user's emotions from facial expressions, tone of voice, etc., and utilizes an emotion analysis engine or the like.

[1746] The "voice recognition means" is a means for converting voice input into character data, and utilizes a voice recognition engine.

[1747] The "information retrieval means" is a means for retrieving necessary information from a database based on the analyzed user input.

[1748] A "database" is a data storage system that stores necessary information and serves as a source of information for responding to user inquiries.

[1749] The present invention relates to an interactive guidance system that quickly provides appropriate information to users in a physical store. The system includes the following main components:

[1750] Detection device

[1751] The system includes a detection device for detecting the approach of a user. This detection device can be a motion sensor or a camera. For example, the camera detects the user's movement and detects that the user is approaching the system.

[1752] Getting User Input

[1753] User input is acquired in the form of voice, touch operation, and text. Voice input is acquired using a microphone, and touch operation is performed through a touch panel. The acquired voice input is converted into text data by a voice recognition engine.

[1754] Analysis means

[1755] The acquired user input is sent to the server and analyzed by a natural language processing engine and an emotion recognition engine. The natural language processing engine analyzes the input text data and understands the content of the user's request or question. The emotion recognition engine recognizes emotions from the user's tone of voice and facial expressions.

[1756] Response generation using generative artificial intelligence models

[1757] Based on the analysis results, the generative AI model generates optimal response information that is tailored to the user's intent and emotional state and is designed to address their individual needs, such as clearly explaining the solution if the user is confused.

[1758] Display means

[1759] The generated response information is provided to the user via a display and an audio output device. The response is displayed in text format on the display, and if an audio output device is used, audio guidance is also provided.

[1760] Example of a system

[1761] A concrete example of such a system is a physical store guidance system using smart glasses. In this case, facial recognition and emotion analysis are performed while the user wears the glasses and moves around the store. When the user asks, "Where is this item?", a speech recognition engine converts the question into text data, and a generative AI model generates a response such as, "Let's get started right away. Go straight and it's on the next shelf on the right."

[1762] Prompt Sentence Examples

[1763] Examples of prompts to input to a generative AI model include:

[1764] "User's query: 'Where is this item?' Sentiment analysis: 'Confused' Generate the best response."

[1765] In this way, by implementing the interactive guidance system of the present invention, personalized guidance can be provided according to the user's emotions, improving the customer experience in a physical store.

[1766] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1767] Step 1:

[1768] The device detects the user's approach

[1769] Input: User approach status

[1770] Operation: A motion sensor or camera installed on the device detects the user's approach. The camera captures the user's movements and recognizes that the user is approaching the system. Output: The user's presence is detected.

[1771] Step 2:

[1772] The device switches to active mode

[1773] Input: User approach information

[1774] Action: The device switches from standby mode to active mode, and the avatar displayed on the screen comes to life. The avatar speaks to the user, saying, "Hello, how can I help you?" Output: Readiness for user response

[1775] Step 3:

[1776] Getting user input

[1777] Input: User's voice input or touch actions

[1778] Action: The user asks a question or makes a request. For example, "Where is this item?" The device's microphone picks up the voice, or the touch panel receives touch input. Output: The captured user input data.

[1779] Step 4:

[1780] Converting voice input to text data

[1781] Input: User voice input

[1782] Operation: The acquired voice input is converted into text data in real time by the voice recognition engine on the device. Output: Text data converted from voice

[1783] Step 5:

[1784] The server parses the input data

[1785] Input: Acquired text data

[1786] Operation: The server analyzes the text data using a natural language processing engine to understand the user's intention and the content of the question. At the same time, the emotion recognition engine analyzes the user's tone of voice and facial expressions to identify their emotional state. Output: Analyzed user intention and emotional information

[1787] Step 6:

[1788] Generating response information using a generative AI model

[1789] Input: Analyzed user intent and emotion information

[1790] Operation: The server's AI model generates the optimal response information based on the analysis results. For example, if the user is confused, it generates a response such as, "Let's start the guide right away. Go straight and it's on the next shelf on the right." Output: Generated response information

[1791] Step 7:

[1792] Display response information

[1793] Input: Generated response information

[1794] Operation: Provides response information to the user through the terminal's display and audio output device. The response is displayed in text format on the display, and is played back aloud when the audio output device is used. Output: Response information presented to the user

[1795] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1796] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1797] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1798] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1799] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1800] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1801] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1802] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1803] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1804] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1805] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1806] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1807] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1808] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1809] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1810] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1811] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1812] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1813] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1814] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1815] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1816] The following is further disclosed regarding the above embodiment.

[1817] (Claim 1)

[1818] a means for detecting the approach of a user by a sensor;

[1819] means for receiving user input;

[1820] means for parsing received user input;

[1821] generating means for generating a response message based on the analysis result;

[1822] a display means for displaying the generated response message to a user;

[1823] An interactive guidance system including:

[1824] (Claim 2)

[1825] 10. The interactive guidance system of claim 1, further comprising a voice recognition means for converting a user's voice input into text data.

[1826] (Claim 3)

[1827] 10. The interactive guidance system of claim 1, further comprising a search means for searching information from a database based on the analyzed user input.

[1828]

[1829] "Example 1"

[1830] (Claim 1)

[1831] a means for detecting the approach of a user by a sensor;

[1832] means for switching from a standby mode to an active mode upon detecting a user;

[1833] a means for launching the avatar in active mode and displaying a greeting;

[1834] means for receiving user input;

[1835] A means for converting received user input into text data using a speech recognition engine;

[1836] means for transmitting text data to a server and analyzing the data;

[1837] A means for generating an appropriate response message based on the analysis results using a generation AI model;

[1838] means for displaying the generated response message to the user in audio and text;

[1839] A system including:

[1840] (Claim 2)

[1841] 10. The system of claim 1, further comprising a speech recognition means for converting a user's speech input into text data.

[1842] (Claim 3)

[1843] 10. The system of claim 1, further comprising means for retrieving information from a database based on the parsed user input.

[1844] "Application Example 1"

[1845] (Claim 1)

[1846] a means for detecting the approach of a user by a sensor;

[1847] means for receiving user input by voice;

[1848] A means for converting the received user voice into text data;

[1849] a means for analyzing text data;

[1850] A generation means using a generative AI model that generates a response message based on the analysis result;

[1851] means for providing the generated response message to the user in voice and text;

[1852] A system including:

[1853] (Claim 2)

[1854] 10. The system of claim 1, wherein the system implements a process of converting user input to text in real time using speech recognition means.

[1855] (Claim 3)

[1856] 10. The system of claim 1, further comprising a search means for searching information from a database based on the analyzed user input to generate an optimal response message.

[1857] "Example 2: Combining Emotion Engines"

[1858] (Claim 1)

[1859] a means for detecting the approach of a user with a sensor;

[1860] means for receiving user input;

[1861] means for parsing received user input;

[1862] generating means for generating a response message based on the analysis result;

[1863] means for displaying the generated response message to a user;

[1864] means for recognizing a user's emotion;

[1865] means for personalizing a response message based on the recognized emotion;

[1866] A system including:

[1867] (Claim 2)

[1868] 10. The system of claim 1, further comprising a speech recognition means for converting a user's voice input into text.

[1869] (Claim 3)

[1870] 10. The system of claim 1, further comprising means for retrieving information from a database based on the parsed user input.

[1871] "Application example 2 when combining emotion engines"

[1872] (Claim 1)

[1873] means for detecting the approach of a user by a detection device;

[1874] a means for obtaining user input;

[1875] a means for parsing the captured user input;

[1876] means for generating response information based on the analysis results using a generative artificial intelligence model;

[1877] a display means for displaying the generated response information to a user;

[1878] A system including:

[1879] (Claim 2)

[1880] 10. The system of claim 1, further comprising means for recognizing an emotional state of a user.

[1881] (Claim 3)

[1882] 10. The system of claim 1, further comprising a speech recognition means for converting speech input into text data.

[1883] (Claim 4)

[1884] 10. The system of claim 1, further comprising means for retrieving information from a database based on the parsed user input. [Explanation of symbols]

[1885] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for detecting the approach of a user by a sensor; means for receiving user input; means for parsing received user input; generating means for generating a response message based on the analysis result; a display means for displaying the generated response message to a user; An interactive guidance system including:

2. 2. The interactive guidance system according to claim 1, further comprising a voice recognition means for converting a user's voice input into text data.

3. 2. The interactive guidance system of claim 1, further comprising a search means for searching information from a database based on the analyzed user input.

Images