system

Abstract

Provide a system. [Solution] a device including an ultradirectional microphone that receives a user's voice input; a server means including a voice recognition engine for converting the voice input into character data; a server means including a natural language processing engine that analyzes the character data and understands the user's intent; a response generating means for generating an appropriate response based on the intention; a server means including a voice synthesis engine for converting the answer into voice data; a device including an ultra-directional speaker that outputs the audio data to a user; A system including a device including a SIM that communicates with said server means over a mobile communications network.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Providing guidance and information to visitors is an important service in modern commercial facilities and public areas, but the resulting labor shortage is a serious problem. In particular, the declining working-age population and the high costs of the hospitality industry place a heavy burden on business operations. Visitors also need a direct and private way to obtain information, instead of searching on their smartphones. However, existing voice guidance systems lack sufficient privacy protection and background noise suppression, preventing an improved user experience. Therefore, a system that protects privacy, suppresses background noise, and provides guidance in natural conversation is needed. [Means for solving the problem]

[0005] The present invention provides a system including a device including a superdirectional microphone that receives a user's voice input, server means including a speech recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intent, response generation means that generates an appropriate response based on the intent, server means including a speech synthesis engine that converts the response into voice data, a device including a superdirectional speaker that outputs the voice data to the user, and a device including a SIM that communicates with the server means via a mobile communications network. This system enables a highly accurate, real-time, natural conversational guidance service to be provided in commercial facilities and public areas, which is expected to improve visitor satisfaction and operational efficiency.

[0006] A super-directional microphone is a microphone that can focus on capturing sounds coming from a specified direction, effectively suppressing ambient and background noise.

[0007] "Device" refers to any physical device, such as electronic equipment or communication equipment.

[0008] A "speech recognition engine" is a software or hardware system that converts voice data into text. It analyzes voice in real time and generates text data.

[0009] "Server means" refers to a device or software configuration that has the function of a server that processes data and performs communication.

[0010] "Character data" refers to information about a character string generated by speech recognition.

[0011] A "natural language processing engine" is a software or hardware system that analyzes text data and understands the user's intent.

[0012] A "response generator" is a software or hardware system for generating appropriate responses based on the user's intent as understood through natural language processing.

[0013] A "speech synthesis engine" is a software or hardware system for converting text data into speech data.

[0014] A "super-directional speaker" is a speaker that can focus sound in a specified direction, providing clear sound while maintaining privacy.

[0015] A "mobile communication network" is a network system that uses mobile communications and refers to a network that is connected using a SIM card.

[0016] A "SIM" is a module used in mobile phones and communication devices for user authentication and data communication.

[0017] "System" refers to an integrated entity consisting of multiple devices, software, and hardware working together. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] DETAILED DESCRIPTION OF THE INVENTION The present invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in commercial facilities and public areas. An embodiment of the present invention will be described in detail below.

[0040] The system includes a device including an ultradirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, and server means including a voice synthesis engine that converts the response into voice data, a device including an ultradirectional speaker that outputs the voice data to the user, and a device including a SIM that communicates with the server means via a mobile communication network.

[0041] Program processing details (natural language explanation)

[0042] User Interface Design

[0043] The devices in this system are designed to be easy for users to operate, with a simple and intuitive UI that accepts touch panels and voice input.

[0044] Building a voice recognition system

[0045] The device's built-in superdirectional microphone captures the user's voice input and transmits the voice data to a server. The server then converts the voice data into text using a cloud-based speech recognition engine. For example, the server may use Google® Cloud Speech-to-Text to convert the voice data into text.

[0046] Building a natural language processing system

[0047] The server then sends the converted text data to a natural language processing engine to analyze the user's intent. For example, it uses the Google Language API to properly interpret the user's question or instruction. The server then understands the user's request based on the analysis results and generates an appropriate response.

[0048] Building a speech synthesis system

[0049] The server sends the generated response message to a speech synthesis engine to generate voice data. For example, Google Text-to-Speech is used to convert the text data into voice. The generated voice data is then sent from the server to the device.

[0050] Audio output

[0051] The device's built-in ultra-directional speaker plays back the audio data received from the server, and by emitting audio in a specific direction, it provides audio guidance to users while protecting their privacy.

[0052] Specific examples

[0053] Example 1: Library guide

[0054] User: "Excuse me, where is the restaurant?"

[0055] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0056] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[0057] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[0058] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[0059] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0060] The device's ultra-directional speaker provides a voice prompt: "The restaurant is on the third floor, on your right."

[0061] Example 2: Providing event information

[0062] User: "What's happening today?"

[0063] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0064] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[0065] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[0066] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[0067] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0068] The device's ultra-directional speaker provides a voice prompt saying, "There will be a concert starting at 3 o'clock today."

[0069] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in the operation of commercial facilities and public areas.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] A user approaches the device and asks, "Where is the bathroom?"

[0073] Step 2:

[0074] The device's ultra-directional microphone captures the user's voice and stores the voice data in a buffer.

[0075] Step 3:

[0076] The terminal generates an HTTP POST request to send the audio data to the server, and attaches the audio data.

[0077] Step 4:

[0078] The server receives the voice data and sends it to the voice recognition engine, specifically by calling the voice recognition engine's API.

[0079] Step 5:

[0080] The server receives text data (e.g., "Where is the toilet?") from the speech recognition engine.

[0081] Step 6:

[0082] The server sends the text data to the natural language processing engine. Specifically, it calls the API of the natural language processing engine and instructs it to analyze the text data.

[0083] Step 7:

[0084] The server receives the analysis results (e.g., the keyword "toilet" and a request to obtain location information) from the natural language processing engine.

[0085] Step 8:

[0086] The server accesses the database and searches for location information of facilities related to the keyword "toilet." For example, it retrieves information that the toilet is on the second floor, on the left.

[0087] Step 9:

[0088] Based on the location information obtained by the server, a response message is generated: "The toilet is on the second floor, on the left."

[0089] Step 10:

[0090] The server sends the response message to the speech synthesis engine to generate voice data. Specifically, it calls the speech synthesis engine's API to convert the text data into voice data.

[0091] Step 11:

[0092] The server returns the generated audio data to the terminal as an HTTP response.

[0093] Step 12:

[0094] The device receives the voice data and provides audible guidance to the user through a super-directional speaker, saying, "The toilet is on the second floor, on your left."

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] Conventional guidance systems in commercial facilities and public areas have faced challenges in protecting privacy when providing information to visitors using voice input, and there has also been a lack of technology to create an interface that is intuitive and easy for visitors to operate.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes a terminal including a highly directional microphone that receives a user's voice input, an information processing device including a voice recognition engine that converts the voice input into text data, an information processing device including a natural language processing engine that analyzes the text data and understands the user's intention, an information processing device including a response generation means that generates an appropriate answer based on the intention, an information processing device including a voice synthesis engine that converts the answer into voice data, a terminal including a highly directional speaker that outputs the voice data to the user, and a communication module that communicates with the information processing device via a wireless communication network. This makes it possible to provide information to visitors through an intuitive and easy-to-use interface while protecting their privacy.

[0100] "User" refers to a person who uses the system to input voice information and obtain guidance information.

[0101] A "highly directional microphone" is a microphone that can efficiently capture sound from a specific direction.

[0102] "Terminal" refers to a device that a user can operate directly or use for voice input.

[0103] A "voice recognition engine" is software or a system for converting voice data into text data.

[0104] "Information processing device" refers to a computing device that includes a speech recognition engine, a natural language processing engine, a response generation means, and a speech synthesis engine.

[0105] A "natural language processing engine" is software or a system that analyzes text data and understands the user's intent.

[0106] A "response generation means" is software or a system that has the function of generating an appropriate response based on the user's intentions.

[0107] A "speech synthesis engine" is software or a system for converting text data into speech data.

[0108] A "highly directional speaker" is a speaker that can effectively transmit sound in a specific direction.

[0109] "Wireless communication network" refers to the communications infrastructure for transmitting and receiving data between a server and a terminal.

[0110] The term "communication module" refers to a device for communicating with an information processing device through a wireless communication network.

[0111] This invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in a commercial facility or public area. The system includes a terminal including a highly directional microphone that receives voice input from a user, an information processing device including a voice recognition engine that converts the voice input into text data, an information processing device including a natural language processing engine that analyzes the text data and understands the user's intent, an information processing device including a response generation means that generates an appropriate response based on the intent, an information processing device including a voice synthesis engine that converts the response into voice data, a terminal including a highly directional speaker that outputs the voice data to the user, and a communication module that communicates with the information processing device via a wireless communication network.

[0112] Device and Technology Configuration

[0113] Terminal

[0114] Users input questions and instructions into the device by voice. The device is equipped with a highly directional microphone, which can capture voice input from a specific direction with high accuracy. The device then transmits the received voice data to an information processing device via a wireless communication network.

[0115] Information processing device

[0116] The information processing device includes a speech recognition engine, a natural language processing engine, a response generation means, and a speech synthesis engine. Received speech data is first converted into text data using the speech recognition engine. For example, Google Cloud Speech-to-Text is used to convert the speech data into text data. The text data is then sent to the natural language processing engine, which analyzes the user's intent. The Google Language API is used to analyze the user's question or instruction and generate an appropriate response. The response generation means creates an appropriate answer based on the analysis results, and the speech synthesis engine converts the answer into speech data. Text data is converted into speech data using Google Text-to-Speech.

[0117] Audio Output

[0118] The device outputs the received audio data to the user using a highly directional speaker, which transmits audio in a specific direction, preventing it from being heard by those around it and providing the user with audio guidance that protects their privacy.

[0119] Specific examples

[0120] Example 1: Library guide

[0121] User: "Excuse me, where is the restaurant?"

[0122] The terminal's highly directional microphone captures the voice and transmits the voice data to an information processing device.

[0123] The information processing device receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[0124] The information processing device analyzes the keyword "restaurant" using a natural language processing engine.

[0125] The information processing device retrieves optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on your right."

[0126] The information processing device converts the response message into voice data using a voice synthesis engine and transmits the voice data to the terminal.

[0127] The device's highly directional speaker provides voice guidance: "The restaurant is on the third floor, on your right."

[0128] Example 2: Providing event information

[0129] User: "What's happening today?"

[0130] The terminal's highly directional microphone captures the voice and transmits the voice data to an information processing device.

[0131] The information processing device receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[0132] The information processing device analyzes the keyword "today's events" using a natural language processing engine.

[0133] The information processing device acquires the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[0134] The information processing device converts the response message into voice data using a voice synthesis engine and transmits the voice data to the terminal.

[0135] The device's highly directional speaker provides a voice prompt saying, "There will be a concert starting at 3 o'clock today."

[0136] In this way, the invention allows visitors to easily obtain the information they need, while also alleviating the labor shortage in the operation of commercial facilities and public areas.

[0137] Prompt Sentence Examples

[0138] "Generate a program to design a system that provides natural conversational guidance to visitors in a commercial establishment."

[0139] "Please provide detailed instructions in natural language for building a system that recognizes input speech and returns an appropriate response."

[0140] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0141] Step 1: Receiving Audio Input

[0142] The user inputs questions and instructions into the terminal by voice.

[0143] The device's highly directional microphone captures the user's voice input.

[0144] The terminal processes the captured audio data to digitize it and transmits it to a server over a wireless communication network.

[0145] Input: User's spoken command (e.g., "Where is the restaurant?")

[0146] Output: Audio data sent to the server

[0147] Step 2: Convert audio data to text

[0148] The server receives the voice data sent from the terminal.

[0149] The server runs the voice data through a voice recognition engine, specifically a cloud-based voice recognition engine (e.g., a general-purpose voice recognition engine).

[0150] The server converts the voice data into character data (text information) using a voice recognition engine.

[0151] Input: Audio data sent from the device

[0152] Output: Text data (e.g., "Where is the restaurant?")

[0153] Step 3: Parsing character data

[0154] The server sends the generated character data to a natural language processing engine.

[0155] The server uses a natural language processing engine to analyze the text data and interpret the user's intent. Specifically, it uses a "text analysis engine."

[0156] Based on the analysis results, the server identifies the user's requests and questions.

[0157] Input: Text data (e.g., "Where is the restaurant?")

[0158] Output: Interpreted intent (e.g., "asking for restaurant locations")

[0159] Step 4: Generate an appropriate response

[0160] The server generates an appropriate response message based on the user's intent.

[0161] The server retrieves the necessary information from a database or external API. For example, it retrieves the "location of the restaurant" from the facility's floor map information.

[0162] The server uses a "response generation engine" to create a response message based on the information it has obtained.

[0163] Input: Interpreted intent (e.g., "asking for restaurant locations")

[0164] Output: Response message (e.g., "The restaurant is on the third floor, on the right.")

[0165] Step 5: Convert the response message to audio data

[0166] The server sends the generated response message to the speech synthesis engine.

[0167] The server uses a speech synthesis engine to generate voice data from text data. Specifically, it uses a "voice conversion engine."

[0168] The server transmits the generated voice data to the terminal.

[0169] Input: Response message (character data)

[0170] Output: Speech data (e.g., "The restaurant is on the third floor, on the right.")

[0171] Step 6: Outputting audio

[0172] The terminal receives the voice data sent from the server.

[0173] The terminal uses a highly directional speaker to play back the received audio data, allowing users to receive guidance information in a manner that protects their privacy without being seen by other visitors.

[0174] Input: Audio data sent from the server

[0175] Output: Spoken directions provided to the user (e.g., "The restaurant is on the third floor, on your right.")

[0176] Through these steps, the system can provide visitors with appropriate information while functioning efficiently and protecting their privacy.

[0177] (Application example 1)

[0178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] In recent years, there has been a demand for methods to quickly and accurately provide visitors with the information they need in commercial facilities and public areas. However, previous guidance systems have had problems such as complex user interfaces and insufficient privacy protection. Furthermore, providing highly accurate information and guidance in real time has been difficult. In brick-and-mortar stores in particular, the inability to smoothly provide product information and location information has led to issues that can lead to lower customer satisfaction and reduced business efficiency.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0181] In this invention, the server includes a device including a superdirectional microphone that receives a user's voice input, means including a voice recognition engine that converts the voice input into text data, means including a natural language processing engine that analyzes the text data and understands the user's intent, response generation means that generates a response message, means including a voice synthesis engine that converts the response into voice data, and means that provides the voice data to the user via a mobile information terminal with a built-in speaker or a mobile machine with a superdirectional speaker. This enables real-time, highly accurate, and privacy-protected guidance via voice input. Furthermore, use in commercial facilities and brick-and-mortar stores can improve customer convenience and streamline store operations.

[0182] 1. A "superdirectional microphone" is a microphone that has the ability to capture only sounds from a specific direction, and is a device that can accurately capture specific sounds by eliminating surrounding noise.

[0183] 2. "Speech Recognition Engine" means a software or hardware component that analyzes and converts voice input into text data.

[0184] 3. "Text data" means text data converted from speech by a speech recognition engine.

[0185] 4. A "natural language processing engine" is an algorithm or system that analyzes input text data and understands the user's intent and meaning.

[0186] 5. "Response generator" means a system or algorithm that creates an appropriate response message based on the user's intent as analyzed by the natural language processing engine.

[0187] 6. "Speech synthesis engine" means a software or hardware component that has the function of converting text data into speech data.

[0188] 7. An "ultra-directional speaker" is a speaker that has the ability to emit sound in a specific direction and is a device designed so that only specific users can hear the sound.

[0189] 8. "Personal digital assistant" means a portable device (e.g., smartphone, tablet) with a built-in speaker for providing audio data to the user.

[0190] 9. "Mobile Machine" means a robot or guide device that contains an ultradirectional speaker and is capable of moving to provide audio data to a user.

[0191] 10. "Mobile communication network" means a network that provides mobile communications for data communication between the entire system and the server.

[0192] 11. "SIM" means a subscriber identity module for use in a mobile communications network.

[0193] DETAILED DESCRIPTION OF THE INVENTION The present invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in commercial facilities and public areas. An embodiment of the present invention will be described in detail below.

[0194] The system includes a device including a superdirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intent, response generation means that generates an appropriate response based on the intent, server means including a voice synthesis engine that converts the response into voice data, and a device including a superdirectional speaker that outputs the voice data to the user, or a mobile information terminal with a built-in speaker. These components also comprise a device including a SIM that communicates with the server means via a mobile communication network.

[0195] Program processing details (natural language explanation)

[0196] User Interface Design

[0197] The smartphones and robots used as terminals are equipped with a user interface that allows voice input and simple touch operations, making it easy for users to use the guidance service.

[0198] Building a voice recognition system

[0199] The device's built-in or super-directional microphone captures the user's voice input, which is then sent to a server, where it converts the speech into text using a cloud-based speech recognition engine such as Google Cloud Speech-to-Text.

[0200] Building a natural language processing system

[0201] The server then sends the converted text data to a natural language processing engine such as Google Language API to analyze the user's intent, allowing it to properly understand the user's question or instruction and generate a response.

[0202] Building a speech synthesis system

[0203] The server sends the generated response message to a speech synthesis engine such as Google Text-to-Speech to generate voice data, which is then sent to the device.

[0204] Audio output

[0205] Voice data synthesized on the server is played from the device's ultra-directional speaker or built-in speaker, providing guidance to the user while protecting their privacy.

[0206] Specific examples

[0207] Example 1: Checking product location

[0208] User: "Excuse me, where is the tea?"

[0209] The device's ultra-directional microphone or built-in microphone captures the sound and transmits the audio data to the server.

[0210] The server converts the voice data into text data and parses it as "Where is the tea?"

[0211] The server identifies the keyword "tea" and retrieves the most appropriate information from a product location database.

[0212] The server generates a response message saying, "Tea is in aisle 3 of the beverage section."

[0213] The server converts this response message into voice data and transmits it to the terminal.

[0214] The device's ultra-directional speaker or built-in speaker announces, "Tea is located in aisle 3 of the beverage section."

[0215] Examples of prompt statements

[0216] Example prompt sentence:

[0217] Please provide a sample audio of a user asking, "Where is the tea?"

[0218] In this way, the present invention allows visitors to easily obtain the information they need, and also improves the efficiency of operations in commercial facilities and public areas, thereby alleviating labor shortages.

[0219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0220] Step 1:

[0221] This is a situation where a user performs voice input. The user asks a question to a device (e.g., a smartphone or a robot) such as "Where is the tea?" The input is the user's voice, and the output is an analog voice signal.

[0222] Step 2:

[0223] The device's superdirectional microphone or built-in microphone receives the user's voice. The received voice is converted into digital voice data and sent to the server. The input is an analog voice signal, and the output is digital voice data.

[0224] Step 3:

[0225] The server sends the received voice data to a speech recognition engine such as Google Cloud Speech-to-Text, where it is converted into text data. At this stage, the voice data is analyzed and output as text data. The input is digital voice data, and the output is text data.

[0226] Step 4:

[0227] The server sends the text data to a natural language processing engine such as Google Language API to analyze the user's intent. The server extracts the keyword "tea" from the text data "Where is the tea?" and uses this keyword to search the appropriate database. The input is text data, and the output is the search query and search results.

[0228] Step 5:

[0229] The server searches the database based on the analyzed keywords and generates a response message such as "Tea is in aisle 3 of the beverage section." The response generator creates an appropriate answer in text format. The input is the search results, and the output is the text response message.

[0230] Step 6:

[0231] The server sends the generated response message to a speech synthesis engine such as Google Text-to-Speech to generate audio data. The server converts the text data into audio data and sends it to the device. The input is the text response message, and the output is digital audio data.

[0232] Step 7:

[0233] The device's ultra-directional speaker or built-in speaker plays the audio data sent from the server. The user receives a voice prompt saying, "Tea is in aisle 3 in the beverage section." The input is digital audio data, and the output is the audio being played.

[0234] This will enable users to obtain necessary information in real time through voice guidance in commercial facilities and public areas.

[0235] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0236] The present invention combines a system for providing visitor guidance in commercial facilities and public areas through natural conversation while protecting privacy with an emotion engine that recognizes user emotions. Hereinafter, an embodiment of the present invention will be described in detail.

[0237] This system comprises a device including an ultradirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, server means including a voice synthesis engine that converts the response into voice data, a device including an ultradirectional speaker that outputs the voice data to the user, and a device including a SIM that communicates with the server means via a mobile communication network.Furthermore, this invention incorporates an emotion engine that recognizes emotions from the user's voice input.

[0238] Program processing details (natural language explanation)

[0239] User Interface Design

[0240] The devices in this system are designed to be easy for users to operate, with a simple and intuitive UI that accepts touch panels and voice input.

[0241] Building a voice recognition system

[0242] The device's onboard super-directional microphone captures the user's voice input and transmits the voice data to a server, which then converts the voice data into text using a cloud-based speech recognition engine. For example, the server may use a cloud speech recognition service to convert the voice data into text.

[0243] Building a natural language processing system

[0244] The server then sends the converted text data to a natural language processing engine to analyze the user's intent. For example, it uses a natural language processing service to properly interpret the user's questions and instructions. The server then understands the user's request based on the analysis results and generates an appropriate response.

[0245] Building an Emotion Engine

[0246] The server sends the text data generated by speech recognition to an emotion engine, which analyzes the user's emotions. For example, in cooperation with a natural language processing engine, the emotion engine can recognize emotions from the user's tone of voice and vocabulary. For example, if the user is angry or in distress, it can respond accordingly.

[0247] Enhanced response generation

[0248] The server uses the emotion engine to generate a response that matches the user's emotion. For example, if the user is in trouble, the response generator will use a gentle tone. This allows for more personalized responses.

[0249] Building a speech synthesis system

[0250] The server sends the generated response message to a speech synthesis engine to generate voice data. For example, a speech synthesis service is used to convert text data into voice. The generated voice data is then sent from the server to the device.

[0251] Audio output

[0252] The device's built-in ultra-directional speaker plays back the audio data received from the server, and by emitting audio in a specific direction, it provides audio guidance to users while protecting their privacy.

[0253] Specific examples

[0254] Example 1: Library guide

[0255] User: "Excuse me, where is the restaurant?" (in a slightly confused voice)

[0256] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0257] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[0258] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[0259] The server uses an emotion engine to recognize when the user is in trouble.

[0260] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[0261] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0262] The device's ultra-directional speaker provides gentle voice guidance, saying, "The restaurant is on the third floor, on the right. We have staff who can guide you, so please ask if you need any help."

[0263] Example 2: Providing event information

[0264] User: "What's happening today?" (excited)

[0265] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0266] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[0267] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[0268] The server uses an emotion engine to recognize when the user is excited.

[0269] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[0270] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0271] The device's ultra-directional speaker provides an excited voice prompt: "There's a concert starting at 3 o'clock today. I'm really looking forward to it!"

[0272] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in commercial facilities and public areas. Furthermore, emotion recognition capabilities enable more personalized responses, further improving visitor satisfaction.

[0273] The processing flow will be explained below.

[0274] Step 1:

[0275] A user approaches the device and asks, "Where is the bathroom?"

[0276] Step 2:

[0277] The device's ultra-directional microphone captures the user's voice and stores the voice data in a buffer.

[0278] Step 3:

[0279] The terminal generates an HTTP POST request to send the audio data to the server, and attaches the audio data.

[0280] Step 4:

[0281] The server receives the voice data and sends it to the voice recognition engine, specifically by calling the voice recognition engine's API.

[0282] Step 5:

[0283] The server receives text data (e.g., "Where is the toilet?") from the speech recognition engine.

[0284] Step 6:

[0285] The server sends the text data to the emotion engine for emotion analysis. Specifically, it calls the emotion engine's API to analyze the text data.

[0286] Step 7:

[0287] The server receives the analysis result (e.g., the user is in trouble) from the emotion engine.

[0288] Step 8:

[0289] The server sends the text data to the natural language processing engine. Specifically, it calls the API of the natural language processing engine and instructs it to analyze the text data.

[0290] Step 9:

[0291] The server receives the analysis results (e.g., the keyword "toilet" and a request to obtain location information) from the natural language processing engine.

[0292] Step 10:

[0293] The server accesses the database and searches for location information of facilities related to the keyword "toilet." For example, it retrieves information that the toilet is on the second floor, on the left.

[0294] Step 11:

[0295] Based on the location information acquired by the server and the analysis results of the emotion engine, a response message (e.g., "The toilet is on the second floor, on the left. We have staff who can guide you, so please let us know if you need help") is generated.

[0296] Step 12:

[0297] The server sends the response message to the speech synthesis engine to generate voice data. Specifically, it calls the speech synthesis engine's API to convert the text data into voice data.

[0298] Step 13:

[0299] The server returns the generated audio data to the terminal as an HTTP response.

[0300] Step 14:

[0301] The device receives the voice data and provides audio guidance to the user through an ultra-directional speaker: "The toilet is on the second floor, on the left. We have staff who can guide you, so please let us know if you need any help."

[0302] Example 2

[0303] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0304] In recent years, there has been a demand for systems that can provide visitors with information through natural conversation while protecting their privacy in commercial facilities and public areas. However, conventional systems have the following problems.

[0305] 1. If the recognition accuracy of voice input is low, it is difficult to accurately understand the user's intention.

[0306] 2. It is difficult for general audio output devices to provide directional audio output while protecting privacy.

[0307] 3. The lack of functionality to recognize user emotions and respond appropriately can lead to low visitor satisfaction.

[0308] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0309] In this invention, the server includes means including an audio input device that receives a user's voice input, means including a voice recognition device that converts the voice input into text data, means including a natural language processing device that analyzes the text data and understands the user's intention, means including a response generation device that generates an appropriate answer based on the intention, means including an emotion authentication device that recognizes the user's emotion, means for enhancing the response generation device that generates an appropriate answer based on the emotion, means including a voice synthesis device that converts the answer into voice data, means including an audio output device that outputs the voice data to the user, and communication means for communicating with the processing device via a communication network. This makes it possible to understand the user's intention and emotion through highly accurate recognition of the voice input and natural conversation, and to provide appropriate voice guidance while protecting privacy.

[0310] An "acoustic input device" is a device that collects sound from a specific direction, and includes a directional microphone.

[0311] A "voice recognition device" is a device that converts voice data into text data, and includes systems that perform highly accurate voice recognition using cloud services.

[0312] A "natural language processing device" is a device that analyzes text data and understands the user's intent, and uses a natural language processing engine.

[0313] A "response generation device" is a device that generates an appropriate response based on the user's intentions, and outputs the generated response as text data.

[0314] An "emotion authentication device" is a device that analyzes emotions from a user's voice and text data, and recognizes the user's emotional state.

[0315] The "means for strengthening the response generation device" is a means for enabling the response generation device to provide a response according to the user's emotions based on the results of the emotion authentication device.

[0316] A "speech synthesizer" is a device that converts character data into voice data, and includes a system that generates synthetic voice.

[0317] An "audio output device" is a device that outputs sound in a specific direction, and includes a directional speaker.

[0318] "Communication means" refers to a means for data communication between a terminal and a server, and includes a SIM card or module for communication via a mobile communication network.

[0319] The present invention combines a system for providing visitor guidance in commercial facilities and public areas through natural conversation while protecting privacy with an emotion engine that recognizes user emotions. Hereinafter, an embodiment of the present invention will be described in detail.

[0320] The system includes the following hardware and software: a device including an ultradirectional microphone for receiving a user's voice input, server means including a voice recognition engine for converting the voice input into text data, server means including a natural language processing engine for analyzing the text data and understanding the user's intention, response generation means for generating an appropriate response based on the user's intention, an emotion authentication device for recognizing the user's emotion, means for enhancing the response generation device for generating an appropriate response based on the emotion, server means including a voice synthesis engine for converting the response into voice data, a device including an ultradirectional speaker for outputting the voice data to the user, and a device including a SIM for communicating with the server means via a mobile communication network.

[0321] First, the user speaks a question or instruction to the system. For example, the user might say, "Excuse me, where is the restaurant?" This voice input is received by the device's superdirectional microphone, and the device transmits the captured voice data via the SIM card and the mobile communication network to a server. The server converts the voice data into text data using a cloud-based speech recognition engine (e.g., Google Cloud Speech-to-Text), and sends this text data to a natural language processing engine (e.g., Google Natural Language API) to analyze the user's intent.

[0322] Next, the server sends the recognized text data and voice data to an emotion authentication device to analyze the user's emotions. This emotion authentication device (e.g., Azure® Cognitive Services' Emotion Analysis API) is used to recognize emotions from the user's tone of voice and phrasing. For example, if the user is distressed or excited, the emotion is identified. The server then activates a response generation device to generate an appropriate response based on the results of the natural language processing engine and emotion authentication device. The response generation device retrieves information from a database and creates a response message, such as "The restaurant is on the third floor, on the right." The response generation device is also enhanced to generate responses in a tone appropriate to the emotion.

[0323] After the response message is generated, the server sends it to a speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) to generate voice data. The generated voice data is sent from the server to the device and played back by the device's super-directional speaker. This speaker emits voice in a specific direction, providing voice guidance to the user while protecting their privacy.

[0324] Example 1: Museum guide

[0325] User: "Excuse me, where is the restaurant?" (in a slightly confused voice)

[0326] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0327] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[0328] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[0329] The server uses an emotion authentication device to recognize that the user is in trouble.

[0330] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[0331] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0332] The device's ultra-directional speaker provides gentle voice guidance, saying, "The restaurant is on the third floor, on the right. We have staff who can guide you, so please ask if you need any help."

[0333] Example 2: Providing event information

[0334] User: "What's happening today?" (excited)

[0335] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0336] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[0337] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[0338] The server uses an emotion authentication device to recognize that the user is excited.

[0339] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[0340] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0341] The device's ultra-directional speaker provides an excited voice prompt: "There's a concert starting at 3 o'clock today. I'm really looking forward to it!"

[0342] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in commercial facilities and public areas. Furthermore, emotion recognition capabilities enable more personalized responses, further improving visitor satisfaction.

[0343] Example prompt sentence:

[0344] "How do we recognise when a visitor is in distress and design a system that can respond appropriately?"

[0345] "Please tell me an example of how emotion recognition is used in a guidance system for a commercial facility."

[0346] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0347] Step 1:

[0348] Accepting user voice input

[0349] The user speaks a question or command, for example, "Excuse me, where is the restaurant?"

[0350] Input: User's voice.

[0351] The device's ultra-directional microphone captures the audio.

[0352] Output: Audio data is generated on the device.

[0353] Step 2:

[0354] Sending audio data

[0355] The device transmits the received voice data to a server via the SIM and the mobile communication network.

[0356] Input: Audio data captured by a superdirectional microphone.

[0357] Output: The audio data sent to the server.

[0358] Step 3:

[0359] Speech Recognition Processing

[0360] The server inputs the received voice data into a voice recognition engine (cloud-based).

[0361] Input: The audio data sent to the server.

[0362] The speech recognition engine converts the voice data into text data, for example, generating the text data "Where is the restaurant?"

[0363] Output: Character data generated within the server.

[0364] Step 4:

[0365] Natural Language Processing

[0366] The server sends the generated character data to a natural language processing engine.

[0367] Input: The text data generated by the speech recognition engine.

[0368] A natural language processing engine analyzes text data to understand the user's intent, for example by extracting the keyword "restaurant."

[0369] Output: Data parsed for user intent.

[0370] Step 5:

[0371] Emotion Recognition Processing

[0372] The server inputs the character data and voice data into the emotion authentication device.

[0373] Input: The text data generated by the speech recognition engine and the original audio data.

[0374] Emotion recognition devices analyze emotions from voice tone and vocabulary, for example, recognizing when a user is distressed.

[0375] Output: Sentiment-analyzed data.

[0376] Step 6:

[0377] Proper response generation

[0378] The server generates an appropriate answer using a response generation means based on the results of the natural language processing engine and the emotion authentication device.

[0379] Input: User intent analyzed by natural language processing and user emotion analyzed by an emotion recognizer.

[0380] The response generation means retrieves information from the database and generates a response message such as "The restaurant is on the third floor, on the right."

[0381] Output: The generated response message.

[0382] Step 7:

[0383] Text-to-speech response messages

[0384] The server sends a response message to the speech synthesis engine.

[0385] Input: The response message generated by the response generator.

[0386] The speech synthesis engine converts text data into speech data, generating a voice file that says, for example, "The restaurant is on the third floor, on your right."

[0387] Output: The generated audio data.

[0388] Step 8:

[0389] Audio data output

[0390] The server transmits the generated voice data to the terminal.

[0391] Input: The speech data generated by the speech synthesis engine.

[0392] The device's ultra-directional speaker plays back the audio data.

[0393] As a specific example, the speaker provides voice guidance in a gentle tone, saying, "The restaurant is on the third floor on the right. We have staff who can guide you, so please ask if you need any help."

[0394] Output: A voice prompt of the processing results that the user hears.

[0395] (Application example 2)

[0396] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0397] In recent years, there has been a demand for automated visitor guidance systems in commercial facilities, public areas, and brick-and-mortar stores. However, existing guidance systems have difficulty protecting visitor privacy and are unable to provide personalized responses that take visitors' emotions into account. This makes it difficult for visitors to quickly and accurately obtain information that meets their needs, resulting in a decline in visitor experience and satisfaction. Given this background, there is a need for the development of a new guidance system that protects visitors' privacy while providing personalized responses that take their emotions into account.

[0398] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a superdirectional microphone that receives a user's voice input, means including a voice recognition engine that converts the voice input into text data, means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, means including a voice synthesis engine that converts the response into voice data, means including a superdirectional speaker that outputs the voice data to the user, means including a SIM that communicates with the server means via a mobile communication network, server means that includes an emotion engine that recognizes emotions from the user's voice input, and means for adjusting the response content based on the emotion recognized by the emotion engine. This enables personalized guidance that corresponds to the emotions of the situation while protecting the visitor's privacy.

[0399] A "super-directional microphone" is a microphone that has the ability to pick up sound only from a specific direction, and is a device that reduces surrounding noise and records the user's voice clearly.

[0400] A "voice recognition engine" is a software system that analyzes voice data and converts it into text data, and is the basic technology for understanding the user's voice.

[0401] A "natural language processing engine" is a software system that analyzes text data to understand the user's intent, and interprets the user's request based on the text data generated by a voice recognition engine.

[0402] The "response generation means" is a system that generates appropriate answers based on the user's intent analyzed by the natural language processing engine, and is responsible for providing responses that meet the user's requests.

[0403] A "speech synthesis engine" is a software system that converts text data into speech data, and is a technology for outputting the generated response as speech.

[0404] An "ultra-directional speaker" is a speaker that has the ability to emit sound in a specific direction, and is a device that provides audio to users while protecting their privacy.

[0405] An "emotion engine" is a software system that analyzes voice and text data to recognize a user's emotions, and understands emotions from the user's tone of voice and choice of words.

[0406] A "SIM" is a module for connecting a device to a cloud server means via a mobile communication network, and is hardware that enables communication between the server and the device.

[0407] "Cloud-based" refers to a technology form that utilizes remote servers on the Internet to process and store data, and does not require the device itself to have high computing power or large amounts of memory.

[0408] This invention is a system for providing guidance to visitors in commercial facilities, public areas, and brick-and-mortar stores, and aims to provide guidance in a natural, conversational style while preserving user privacy. It also aims to improve visitor satisfaction by recognizing the user's emotions and providing personalized responses based on those emotions.

[0409] The system includes the following elements:

[0410] 1. Ultra-directional microphone: A device that has the ability to pick up sound only from a specific direction, clearly recording the user's voice.

[0411] 2. A speech recognition engine included in the server means: A software system that converts a user's voice input into text data.

[0412] 3. Natural language processing engine included in the server means: A software system for analyzing character data and understanding the user's intent.

[0413] 4. Response generation means: A system that generates appropriate responses based on the user's intent.

[0414] 5. A speech synthesis engine included in the server means: A software system that converts the generated answers into speech data.

[0415] 6. Ultra-directional speaker: A speaker that emits sound in a specific direction, a device that provides audio guidance while maintaining privacy.

[0416] 7. Device including SIM: A module that performs communication between the server means and the device via a mobile communication network.

[0417] 8. Emotion engine: A software system that analyzes voice and text data to recognize the user's emotions.

[0418] Let us explain the system's operation through an example:

[0419] Example 1: Product location guidance

[0420] The user asks in a confused voice, "Where is the chocolate?" The device's superdirectional microphone receives the voice and sends the voice data to the server. The server's speech recognition engine converts the voice data into text data, and the natural language processing engine detects the keyword "chocolate" and analyzes the user's intention. Next, the emotion engine recognizes that the user is confused, and the response generation means generates a gentle response saying, "The chocolate is on the first floor, on the left. Shall I show you?" The response generated by the speech synthesis engine is converted into voice data, and the superdirectional speaker provides audio guidance to the user.

[0421] Example 2: Providing sales information

[0422] The user asks in an excited voice, "Are there any sales going on right now?" The device's superdirectional microphone receives the voice and sends the voice data to the server. The server's speech recognition engine converts the voice data into text data, and the natural language processing engine detects the keyword "sale" and analyzes the user's intention. Next, the emotion engine recognizes that the user is excited, and the response generation means generates a response in an excited tone: "Yes, there's a 20% off sale in the clothing department on the first floor!" The speech synthesis engine converts the generated response into voice data, and the superdirectional speaker provides audio guidance to the user.

[0423] As these examples show, this system can improve visitor satisfaction by providing personalized guidance based on emotions while protecting visitor privacy.

[0424] Prompt Sentence Examples

[0425] Prompt when user uses smart glasses to ask for product location:

[0426] "A user asked in a confused voice, 'Where is the chocolate?'

[0427] The voice recognition engine converted the text data. Next, the natural language processing engine analyzed the product location, and the emotion recognition engine recognized the user's confused emotion. The response generation engine generated a response in a gentle tone, and the speech synthesis engine converted it into voice data. The super-directional speaker said in a gentle tone, "The chocolate is on the first floor, on the left. Shall I show you?"

[0428] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0429] Step 1: Receiving Audio Input

[0430] This is the role of the device. The device's built-in superdirectional microphone receives the user's voice input and records it as voice data. This input includes what the user says. For example, the user may ask, "Where is the chocolate?"

[0431] Step 2: Sending audio data

[0432] This is the role of the device. It transmits the received voice data to a cloud-based server via a mobile communication network. The input in this step is the voice data recorded in step 1, and the output is the voice data received by the server.

[0433] Step 3: Voice Recognition

[0434] This is the role of the server. A cloud-based speech recognition engine analyzes the voice data received by the server and converts the voice data into text data. The input is voice data and the output is text data. Specifically, the speech recognition engine analyzes the utterance "Where is the chocolate?" and converts it into text data "Where is the chocolate?"

[0435] Step 4: Natural Language Processing

[0436] This is the role of the server. The natural language processing engine analyzes the converted text data and understands the user's intent. The input is text data, and the output is the analyzed intent data. Specifically, the natural language processing engine detects the keyword "chocolate" and understands that the user is asking where the chocolate is.

[0437] Step 5: Emotion Recognition

[0438] This is the role of the server. The emotion engine analyzes the text data obtained by the natural language processing engine and recognizes the user's emotions. The input is text data, and the output is recognized emotional data. Specifically, the emotion engine recognizes that the user is in a difficult situation.

[0439] Step 6: Response Generation

[0440] This is the role of the server. Based on the recognized emotion data and analyzed intent data, the response generation means generates an appropriate answer. The input is the analyzed intent data and emotion data, and the output is a response message. Specifically, the response generation means generates a gentle response message saying, "The chocolate is on the first floor on the left. Shall I show you?"

[0441] Step 7: Text-to-Speech

[0442] This is the role of the server. The generated response message is converted into voice data by the speech synthesis engine. The input is the response message, and the output is voice data. Specifically, the speech synthesis engine generates voice data saying, "The chocolate is on the first floor on the left. Shall I show you?"

[0443] Step 8: Audio Output

[0444] This is the role of the terminal. The generated voice data is transmitted to the user by the terminal's superdirectional speaker. The input is voice data, and the output is the user receiving the voice. In concrete terms, the terminal's superdirectional speaker provides voice guidance to the user, saying, "The chocolate is on the first floor, on the left. Shall I show you?"

[0445] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0446] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0447] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0448] [Second embodiment]

[0449] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0450] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0451] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0452] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0453] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0454] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0455] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0456] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0457] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0458] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0459] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0460] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0461] DETAILED DESCRIPTION OF THE INVENTION The present invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in commercial facilities and public areas. An embodiment of the present invention will be described in detail below.

[0462] The system includes a device including an ultradirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, and server means including a voice synthesis engine that converts the response into voice data, a device including an ultradirectional speaker that outputs the voice data to the user, and a device including a SIM that communicates with the server means via a mobile communication network.

[0463] Program processing details (natural language explanation)

[0464] User Interface Design

[0465] The devices in this system are designed to be easy for users to operate, with a simple and intuitive UI that accepts touch panels and voice input.

[0466] Building a voice recognition system

[0467] The device's onboard super-directional microphone captures the user's voice input and sends the voice data to a server, which then converts the voice data into text using a cloud-based speech recognition engine. For example, the server uses Google Cloud Speech-to-Text to convert the voice data into text.

[0468] Building a natural language processing system

[0469] The server then sends the converted text data to a natural language processing engine to analyze the user's intent. For example, it uses the Google Language API to properly interpret the user's question or instruction. The server then understands the user's request based on the analysis results and generates an appropriate response.

[0470] Building a speech synthesis system

[0471] The server sends the generated response message to a speech synthesis engine to generate voice data. For example, Google Text-to-Speech is used to convert the text data into voice. The generated voice data is then sent from the server to the device.

[0472] Audio output

[0473] The device's built-in ultra-directional speaker plays back the audio data received from the server, and by emitting audio in a specific direction, it provides audio guidance to users while protecting their privacy.

[0474] Specific examples

[0475] Example 1: Library guide

[0476] User: "Excuse me, where is the restaurant?"

[0477] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0478] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[0479] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[0480] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[0481] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0482] The device's ultra-directional speaker provides a voice prompt: "The restaurant is on the third floor, on your right."

[0483] Example 2: Providing event information

[0484] User: "What's happening today?"

[0485] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0486] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[0487] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[0488] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[0489] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0490] The device's ultra-directional speaker provides a voice prompt saying, "There will be a concert starting at 3 o'clock today."

[0491] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in the operation of commercial facilities and public areas.

[0492] The processing flow will be explained below.

[0493] Step 1:

[0494] A user approaches the device and asks, "Where is the bathroom?"

[0495] Step 2:

[0496] The device's ultra-directional microphone captures the user's voice and stores the voice data in a buffer.

[0497] Step 3:

[0498] The terminal generates an HTTP POST request to send the audio data to the server, and attaches the audio data.

[0499] Step 4:

[0500] The server receives the voice data and sends it to the voice recognition engine, specifically by calling the voice recognition engine's API.

[0501] Step 5:

[0502] The server receives text data (e.g., "Where is the toilet?") from the speech recognition engine.

[0503] Step 6:

[0504] The server sends the text data to the natural language processing engine. Specifically, it calls the API of the natural language processing engine and instructs it to analyze the text data.

[0505] Step 7:

[0506] The server receives the analysis results (e.g., the keyword "toilet" and a request to obtain location information) from the natural language processing engine.

[0507] Step 8:

[0508] The server accesses the database and searches for location information of facilities related to the keyword "toilet." For example, it retrieves information that the toilet is on the second floor, on the left.

[0509] Step 9:

[0510] Based on the location information obtained by the server, a response message is generated: "The toilet is on the second floor, on the left."

[0511] Step 10:

[0512] The server sends the response message to the speech synthesis engine to generate voice data. Specifically, it calls the speech synthesis engine's API to convert the text data into voice data.

[0513] Step 11:

[0514] The server returns the generated audio data to the terminal as an HTTP response.

[0515] Step 12:

[0516] The device receives the voice data and provides audible guidance to the user through a super-directional speaker, saying, "The toilet is on the second floor, on your left."

[0517] Example 1

[0518] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0519] Conventional guidance systems in commercial facilities and public areas have faced challenges in protecting privacy when providing information to visitors using voice input, and there has also been a lack of technology to create an interface that is intuitive and easy for visitors to operate.

[0520] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0521] In this invention, the server includes a terminal including a highly directional microphone that receives a user's voice input, an information processing device including a voice recognition engine that converts the voice input into text data, an information processing device including a natural language processing engine that analyzes the text data and understands the user's intention, an information processing device including a response generation means that generates an appropriate answer based on the intention, an information processing device including a voice synthesis engine that converts the answer into voice data, a terminal including a highly directional speaker that outputs the voice data to the user, and a communication module that communicates with the information processing device via a wireless communication network. This makes it possible to provide information to visitors through an intuitive and easy-to-use interface while protecting their privacy.

[0522] "User" refers to a person who uses the system to input voice information and obtain guidance information.

[0523] A "highly directional microphone" is a microphone that can efficiently capture sound from a specific direction.

[0524] "Terminal" refers to a device that a user can operate directly or use for voice input.

[0525] A "voice recognition engine" is software or a system for converting voice data into text data.

[0526] "Information processing device" refers to a computing device that includes a speech recognition engine, a natural language processing engine, a response generation means, and a speech synthesis engine.

[0527] A "natural language processing engine" is software or a system that analyzes text data and understands the user's intent.

[0528] A "response generation means" is software or a system that has the function of generating an appropriate response based on the user's intentions.

[0529] A "speech synthesis engine" is software or a system for converting text data into speech data.

[0530] A "highly directional speaker" is a speaker that can effectively transmit sound in a specific direction.

[0531] "Wireless communication network" refers to the communications infrastructure for transmitting and receiving data between a server and a terminal.

[0532] The term "communication module" refers to a device for communicating with an information processing device through a wireless communication network.

[0533] This invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in a commercial facility or public area. The system includes a terminal including a highly directional microphone that receives voice input from a user, an information processing device including a voice recognition engine that converts the voice input into text data, an information processing device including a natural language processing engine that analyzes the text data and understands the user's intent, an information processing device including a response generation means that generates an appropriate response based on the intent, an information processing device including a voice synthesis engine that converts the response into voice data, a terminal including a highly directional speaker that outputs the voice data to the user, and a communication module that communicates with the information processing device via a wireless communication network.

[0534] Device and Technology Configuration

[0535] Terminal

[0536] Users input questions and instructions into the device by voice. The device is equipped with a highly directional microphone, which can capture voice input from a specific direction with high accuracy. The device then transmits the received voice data to an information processing device via a wireless communication network.

[0537] Information processing device

[0538] The information processing device includes a speech recognition engine, a natural language processing engine, a response generation means, and a speech synthesis engine. Received speech data is first converted into text data using the speech recognition engine. For example, Google Cloud Speech-to-Text is used to convert the speech data into text data. The text data is then sent to the natural language processing engine, which analyzes the user's intent. The Google Language API is used to analyze the user's question or instruction and generate an appropriate response. The response generation means creates an appropriate answer based on the analysis results, and the speech synthesis engine converts the answer into speech data. Text data is converted into speech data using Google Text-to-Speech.

[0539] Audio Output

[0540] The device outputs the received audio data to the user using a highly directional speaker, which transmits audio in a specific direction, preventing it from being heard by those around it and providing the user with audio guidance that protects their privacy.

[0541] Specific examples

[0542] Example 1: Library guide

[0543] User: "Excuse me, where is the restaurant?"

[0544] The terminal's highly directional microphone captures the voice and transmits the voice data to an information processing device.

[0545] The information processing device receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[0546] The information processing device analyzes the keyword "restaurant" using a natural language processing engine.

[0547] The information processing device retrieves optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on your right."

[0548] The information processing device converts the response message into voice data using a voice synthesis engine and transmits the voice data to the terminal.

[0549] The device's highly directional speaker provides voice guidance: "The restaurant is on the third floor, on your right."

[0550] Example 2: Providing event information

[0551] User: "What's happening today?"

[0552] The terminal's highly directional microphone captures the voice and transmits the voice data to an information processing device.

[0553] The information processing device receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[0554] The information processing device analyzes the keyword "today's events" using a natural language processing engine.

[0555] The information processing device acquires the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[0556] The information processing device converts the response message into voice data using a voice synthesis engine and transmits the voice data to the terminal.

[0557] The device's highly directional speaker provides a voice prompt saying, "There will be a concert starting at 3 o'clock today."

[0558] In this way, the invention allows visitors to easily obtain the information they need, while also alleviating the labor shortage in the operation of commercial facilities and public areas.

[0559] Prompt Sentence Examples

[0560] "Generate a program to design a system that provides natural conversational guidance to visitors in a commercial establishment."

[0561] "Please provide detailed instructions in natural language for building a system that recognizes input speech and returns an appropriate response."

[0562] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0563] Step 1: Receiving Audio Input

[0564] The user inputs questions and instructions into the terminal by voice.

[0565] The device's highly directional microphone captures the user's voice input.

[0566] The terminal processes the captured audio data to digitize it and transmits it to a server over a wireless communication network.

[0567] Input: User's spoken command (e.g., "Where is the restaurant?")

[0568] Output: Audio data sent to the server

[0569] Step 2: Convert audio data to text

[0570] The server receives the voice data sent from the terminal.

[0571] The server runs the voice data through a voice recognition engine, specifically a cloud-based voice recognition engine (e.g., a general-purpose voice recognition engine).

[0572] The server converts the voice data into character data (text information) using a voice recognition engine.

[0573] Input: Audio data sent from the device

[0574] Output: Text data (e.g., "Where is the restaurant?")

[0575] Step 3: Parsing character data

[0576] The server sends the generated character data to a natural language processing engine.

[0577] The server uses a natural language processing engine to analyze the text data and interpret the user's intent. Specifically, it uses a "text analysis engine."

[0578] Based on the analysis results, the server identifies the user's requests and questions.

[0579] Input: Text data (e.g., "Where is the restaurant?")

[0580] Output: Interpreted intent (e.g., "asking for restaurant locations")

[0581] Step 4: Generate an appropriate response

[0582] The server generates an appropriate response message based on the user's intent.

[0583] The server retrieves the necessary information from a database or external API. For example, it retrieves the "location of the restaurant" from the facility's floor map information.

[0584] The server uses a "response generation engine" to create a response message based on the information it has obtained.

[0585] Input: Interpreted intent (e.g., "asking for restaurant locations")

[0586] Output: Response message (e.g., "The restaurant is on the third floor, on the right.")

[0587] Step 5: Convert the response message to audio data

[0588] The server sends the generated response message to the speech synthesis engine.

[0589] The server uses a speech synthesis engine to generate voice data from text data. Specifically, it uses a "voice conversion engine."

[0590] The server transmits the generated voice data to the terminal.

[0591] Input: Response message (character data)

[0592] Output: Speech data (e.g., "The restaurant is on the third floor, on the right.")

[0593] Step 6: Outputting audio

[0594] The terminal receives the voice data sent from the server.

[0595] The terminal uses a highly directional speaker to play back the received audio data, allowing users to receive guidance information in a manner that protects their privacy without being seen by other visitors.

[0596] Input: Audio data sent from the server

[0597] Output: Spoken directions provided to the user (e.g., "The restaurant is on the third floor, on your right.")

[0598] Through these steps, the system can provide visitors with appropriate information while functioning efficiently and protecting their privacy.

[0599] (Application example 1)

[0600] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0601] In recent years, there has been a demand for methods to quickly and accurately provide visitors with the information they need in commercial facilities and public areas. However, previous guidance systems have had problems such as complex user interfaces and insufficient privacy protection. Furthermore, providing highly accurate information and guidance in real time has been difficult. In brick-and-mortar stores in particular, the inability to smoothly provide product information and location information has led to issues that can lead to lower customer satisfaction and reduced business efficiency.

[0602] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0603] In this invention, the server includes a device including a superdirectional microphone that receives a user's voice input, means including a voice recognition engine that converts the voice input into text data, means including a natural language processing engine that analyzes the text data and understands the user's intent, response generation means that generates a response message, means including a voice synthesis engine that converts the response into voice data, and means that provides the voice data to the user via a mobile information terminal with a built-in speaker or a mobile machine with a superdirectional speaker. This enables real-time, highly accurate, and privacy-protected guidance via voice input. Furthermore, use in commercial facilities and brick-and-mortar stores can improve customer convenience and streamline store operations.

[0604] 1. A "superdirectional microphone" is a microphone that has the ability to capture only sounds from a specific direction, and is a device that can accurately capture specific sounds by eliminating surrounding noise.

[0605] 2. "Speech Recognition Engine" means a software or hardware component that analyzes and converts voice input into text data.

[0606] 3. "Text data" means text data converted from speech by a speech recognition engine.

[0607] 4. A "natural language processing engine" is an algorithm or system that analyzes input text data and understands the user's intent and meaning.

[0608] 5. "Response generator" means a system or algorithm that creates an appropriate response message based on the user's intent as analyzed by the natural language processing engine.

[0609] 6. "Speech synthesis engine" means a software or hardware component that has the function of converting text data into speech data.

[0610] 7. An "ultra-directional speaker" is a speaker that has the ability to emit sound in a specific direction and is a device designed so that only specific users can hear the sound.

[0611] 8. "Personal digital assistant" means a portable device (e.g., smartphone, tablet) with a built-in speaker for providing audio data to the user.

[0612] 9. "Mobile Machine" means a robot or guide device that contains an ultradirectional speaker and is capable of moving to provide audio data to a user.

[0613] 10. "Mobile communication network" means a network that provides mobile communications for data communication between the entire system and the server.

[0614] 11. "SIM" means a subscriber identity module for use in a mobile communications network.

[0615] DETAILED DESCRIPTION OF THE INVENTION The present invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in commercial facilities and public areas. An embodiment of the present invention will be described in detail below.

[0616] The system includes a device including a superdirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intent, response generation means that generates an appropriate response based on the intent, server means including a voice synthesis engine that converts the response into voice data, and a device including a superdirectional speaker that outputs the voice data to the user, or a mobile information terminal with a built-in speaker. These components also comprise a device including a SIM that communicates with the server means via a mobile communication network.

[0617] Program processing details (natural language explanation)

[0618] User Interface Design

[0619] The smartphones and robots used as terminals are equipped with a user interface that allows voice input and simple touch operations, making it easy for users to use the guidance service.

[0620] Building a voice recognition system

[0621] The device's built-in or super-directional microphone captures the user's voice input, which is then sent to a server, where it converts the speech into text using a cloud-based speech recognition engine such as Google Cloud Speech-to-Text.

[0622] Building a natural language processing system

[0623] The server then sends the converted text data to a natural language processing engine such as Google Language API to analyze the user's intent, allowing it to properly understand the user's question or instruction and generate a response.

[0624] Building a speech synthesis system

[0625] The server sends the generated response message to a speech synthesis engine such as Google Text-to-Speech to generate voice data, which is then sent to the device.

[0626] Audio output

[0627] Voice data synthesized on the server is played from the device's ultra-directional speaker or built-in speaker, providing guidance to the user while protecting their privacy.

[0628] Specific examples

[0629] Example 1: Checking product location

[0630] User: "Excuse me, where is the tea?"

[0631] The device's ultra-directional microphone or built-in microphone captures the sound and transmits the audio data to the server.

[0632] The server converts the voice data into text data and parses it as "Where is the tea?"

[0633] The server identifies the keyword "tea" and retrieves the most appropriate information from a product location database.

[0634] The server generates a response message saying, "Tea is in aisle 3 of the beverage section."

[0635] The server converts this response message into voice data and transmits it to the terminal.

[0636] The device's ultra-directional speaker or built-in speaker announces, "Tea is located in aisle 3 of the beverage section."

[0637] Examples of prompt statements

[0638] Example prompt sentence:

[0639] Please provide a sample audio of a user asking, "Where is the tea?"

[0640] In this way, the present invention allows visitors to easily obtain the information they need, and also improves the efficiency of operations in commercial facilities and public areas, thereby alleviating labor shortages.

[0641] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0642] Step 1:

[0643] This is a situation where a user performs voice input. The user asks a question to a device (e.g., a smartphone or a robot) such as "Where is the tea?" The input is the user's voice, and the output is an analog voice signal.

[0644] Step 2:

[0645] The device's superdirectional microphone or built-in microphone receives the user's voice. The received voice is converted into digital voice data and sent to the server. The input is an analog voice signal, and the output is digital voice data.

[0646] Step 3:

[0647] The server sends the received voice data to a speech recognition engine such as Google Cloud Speech-to-Text, where it is converted into text data. At this stage, the voice data is analyzed and output as text data. The input is digital voice data, and the output is text data.

[0648] Step 4:

[0649] The server sends the text data to a natural language processing engine such as Google Language API to analyze the user's intent. The server extracts the keyword "tea" from the text data "Where is the tea?" and uses this keyword to search the appropriate database. The input is text data, and the output is the search query and search results.

[0650] Step 5:

[0651] The server searches the database based on the analyzed keywords and generates a response message such as "Tea is in aisle 3 of the beverage section." The response generator creates an appropriate answer in text format. The input is the search results, and the output is the text response message.

[0652] Step 6:

[0653] The server sends the generated response message to a speech synthesis engine such as Google Text-to-Speech to generate audio data. The server converts the text data into audio data and sends it to the device. The input is the text response message, and the output is digital audio data.

[0654] Step 7:

[0655] The device's ultra-directional speaker or built-in speaker plays the audio data sent from the server. The user receives a voice prompt saying, "Tea is in aisle 3 in the beverage section." The input is digital audio data, and the output is the audio being played.

[0656] This will enable users to obtain necessary information in real time through voice guidance in commercial facilities and public areas.

[0657] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0658] The present invention combines a system for providing visitor guidance in commercial facilities and public areas through natural conversation while protecting privacy with an emotion engine that recognizes user emotions. Hereinafter, an embodiment of the present invention will be described in detail.

[0659] This system comprises a device including an ultradirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, server means including a voice synthesis engine that converts the response into voice data, a device including an ultradirectional speaker that outputs the voice data to the user, and a device including a SIM that communicates with the server means via a mobile communication network.Furthermore, this invention incorporates an emotion engine that recognizes emotions from the user's voice input.

[0660] Program processing details (natural language explanation)

[0661] User Interface Design

[0662] The devices in this system are designed to be easy for users to operate, with a simple and intuitive UI that accepts touch panels and voice input.

[0663] Building a voice recognition system

[0664] The device's onboard super-directional microphone captures the user's voice input and transmits the voice data to a server, which then converts the voice data into text using a cloud-based speech recognition engine. For example, the server may use a cloud speech recognition service to convert the voice data into text.

[0665] Building a natural language processing system

[0666] The server then sends the converted text data to a natural language processing engine to analyze the user's intent. For example, it uses a natural language processing service to properly interpret the user's questions and instructions. The server then understands the user's request based on the analysis results and generates an appropriate response.

[0667] Building an Emotion Engine

[0668] The server sends the text data generated by speech recognition to an emotion engine, which analyzes the user's emotions. For example, in cooperation with a natural language processing engine, the emotion engine can recognize emotions from the user's tone of voice and vocabulary. For example, if the user is angry or in distress, it can respond accordingly.

[0669] Enhanced response generation

[0670] The server uses the emotion engine to generate a response that matches the user's emotion. For example, if the user is in trouble, the response generator will use a gentle tone. This allows for more personalized responses.

[0671] Building a speech synthesis system

[0672] The server sends the generated response message to a speech synthesis engine to generate voice data. For example, a speech synthesis service is used to convert text data into voice. The generated voice data is then sent from the server to the device.

[0673] Audio output

[0674] The device's built-in ultra-directional speaker plays back the audio data received from the server, and by emitting audio in a specific direction, it provides audio guidance to users while protecting their privacy.

[0675] Specific examples

[0676] Example 1: Library guide

[0677] User: "Excuse me, where is the restaurant?" (in a slightly confused voice)

[0678] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0679] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[0680] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[0681] The server uses an emotion engine to recognize when the user is in trouble.

[0682] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[0683] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0684] The device's ultra-directional speaker provides gentle voice guidance, saying, "The restaurant is on the third floor, on the right. We have staff who can guide you, so please ask if you need any help."

[0685] Example 2: Providing event information

[0686] User: "What's happening today?" (excited)

[0687] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0688] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[0689] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[0690] The server uses an emotion engine to recognize when the user is excited.

[0691] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[0692] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0693] The device's ultra-directional speaker provides an excited voice prompt: "There's a concert starting at 3 o'clock today. I'm really looking forward to it!"

[0694] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in commercial facilities and public areas. Furthermore, emotion recognition capabilities enable more personalized responses, further improving visitor satisfaction.

[0695] The processing flow will be explained below.

[0696] Step 1:

[0697] A user approaches the device and asks, "Where is the bathroom?"

[0698] Step 2:

[0699] The device's ultra-directional microphone captures the user's voice and stores the voice data in a buffer.

[0700] Step 3:

[0701] The terminal generates an HTTP POST request to send the audio data to the server, and attaches the audio data.

[0702] Step 4:

[0703] The server receives the voice data and sends it to the voice recognition engine, specifically by calling the voice recognition engine's API.

[0704] Step 5:

[0705] The server receives text data (e.g., "Where is the toilet?") from the speech recognition engine.

[0706] Step 6:

[0707] The server sends the text data to the emotion engine for emotion analysis. Specifically, it calls the emotion engine's API to analyze the text data.

[0708] Step 7:

[0709] The server receives the analysis result (e.g., the user is in trouble) from the emotion engine.

[0710] Step 8:

[0711] The server sends the text data to the natural language processing engine. Specifically, it calls the API of the natural language processing engine and instructs it to analyze the text data.

[0712] Step 9:

[0713] The server receives the analysis results (e.g., the keyword "toilet" and a request to obtain location information) from the natural language processing engine.

[0714] Step 10:

[0715] The server accesses the database and searches for location information of facilities related to the keyword "toilet." For example, it retrieves information that the toilet is on the second floor, on the left.

[0716] Step 11:

[0717] Based on the location information acquired by the server and the analysis results of the emotion engine, a response message (e.g., "The toilet is on the second floor, on the left. We have staff who can guide you, so please let us know if you need help") is generated.

[0718] Step 12:

[0719] The server sends the response message to the speech synthesis engine to generate voice data. Specifically, it calls the speech synthesis engine's API to convert the text data into voice data.

[0720] Step 13:

[0721] The server returns the generated audio data to the terminal as an HTTP response.

[0722] Step 14:

[0723] The device receives the voice data and provides audio guidance to the user through an ultra-directional speaker: "The toilet is on the second floor, on the left. We have staff who can guide you, so please let us know if you need any help."

[0724] Example 2

[0725] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0726] In recent years, there has been a demand for systems that can provide visitors with information through natural conversation while protecting their privacy in commercial facilities and public areas. However, conventional systems have the following problems.

[0727] 1. If the recognition accuracy of voice input is low, it is difficult to accurately understand the user's intention.

[0728] 2. It is difficult for general audio output devices to provide directional audio output while protecting privacy.

[0729] 3. The lack of functionality to recognize user emotions and respond appropriately can lead to low visitor satisfaction.

[0730] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0731] In this invention, the server includes means including an audio input device that receives a user's voice input, means including a voice recognition device that converts the voice input into text data, means including a natural language processing device that analyzes the text data and understands the user's intention, means including a response generation device that generates an appropriate answer based on the intention, means including an emotion authentication device that recognizes the user's emotion, means for enhancing the response generation device that generates an appropriate answer based on the emotion, means including a voice synthesis device that converts the answer into voice data, means including an audio output device that outputs the voice data to the user, and communication means for communicating with the processing device via a communication network. This makes it possible to understand the user's intention and emotion through highly accurate recognition of the voice input and natural conversation, and to provide appropriate voice guidance while protecting privacy.

[0732] An "acoustic input device" is a device that collects sound from a specific direction, and includes a directional microphone.

[0733] A "voice recognition device" is a device that converts voice data into text data, and includes systems that perform highly accurate voice recognition using cloud services.

[0734] A "natural language processing device" is a device that analyzes text data and understands the user's intent, and uses a natural language processing engine.

[0735] A "response generation device" is a device that generates an appropriate response based on the user's intentions, and outputs the generated response as text data.

[0736] An "emotion authentication device" is a device that analyzes emotions from a user's voice and text data, and recognizes the user's emotional state.

[0737] The "means for strengthening the response generation device" is a means for enabling the response generation device to provide a response according to the user's emotions based on the results of the emotion authentication device.

[0738] A "speech synthesizer" is a device that converts character data into voice data, and includes a system that generates synthetic voice.

[0739] An "audio output device" is a device that outputs sound in a specific direction, and includes a directional speaker.

[0740] "Communication means" refers to a means for data communication between a terminal and a server, and includes a SIM card or module for communication via a mobile communication network.

[0741] The present invention combines a system for providing visitor guidance in commercial facilities and public areas through natural conversation while protecting privacy with an emotion engine that recognizes user emotions. Hereinafter, an embodiment of the present invention will be described in detail.

[0742] The system includes the following hardware and software: a device including an ultradirectional microphone for receiving a user's voice input, server means including a voice recognition engine for converting the voice input into text data, server means including a natural language processing engine for analyzing the text data and understanding the user's intention, response generation means for generating an appropriate response based on the user's intention, an emotion authentication device for recognizing the user's emotion, means for enhancing the response generation device for generating an appropriate response based on the emotion, server means including a voice synthesis engine for converting the response into voice data, a device including an ultradirectional speaker for outputting the voice data to the user, and a device including a SIM for communicating with the server means via a mobile communication network.

[0743] First, the user speaks a question or instruction to the system. For example, the user might say, "Excuse me, where is the restaurant?" This voice input is received by the device's superdirectional microphone, and the device transmits the captured voice data via the SIM card and the mobile communication network to a server. The server converts the voice data into text data using a cloud-based speech recognition engine (e.g., Google Cloud Speech-to-Text), and sends this text data to a natural language processing engine (e.g., Google Natural Language API) to analyze the user's intent.

[0744] Next, the server sends the recognized text data and voice data to an emotion recognition device, which analyzes the user's emotions. This emotion recognition device (for example, Azure Cognitive Services' Emotion Analysis API) is used to recognize emotions from the user's tone of voice and phrasing. For example, if the user is distressed or excited, the emotion is identified. The server then activates a response generation device to generate an appropriate response based on the results of the natural language processing engine and emotion recognition device. The response generation device retrieves information from a database and creates a response message, such as "The restaurant is on the third floor, on the right." The response generation device is also enhanced to generate responses in a tone that corresponds to the emotion.

[0745] After the response message is generated, the server sends it to a speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) to generate voice data. The generated voice data is sent from the server to the device and played back by the device's super-directional speaker. This speaker emits voice in a specific direction, providing voice guidance to the user while protecting their privacy.

[0746] Example 1: Museum guide

[0747] User: "Excuse me, where is the restaurant?" (in a slightly confused voice)

[0748] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0749] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[0750] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[0751] The server uses an emotion authentication device to recognize that the user is in trouble.

[0752] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[0753] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0754] The device's ultra-directional speaker provides gentle voice guidance, saying, "The restaurant is on the third floor, on the right. We have staff who can guide you, so please ask if you need any help."

[0755] Example 2: Providing event information

[0756] User: "What's happening today?" (excited)

[0757] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0758] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[0759] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[0760] The server uses an emotion authentication device to recognize that the user is excited.

[0761] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[0762] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0763] The device's ultra-directional speaker provides an excited voice prompt: "There's a concert starting at 3 o'clock today. I'm really looking forward to it!"

[0764] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in commercial facilities and public areas. Furthermore, emotion recognition capabilities enable more personalized responses, further improving visitor satisfaction.

[0765] Example prompt sentence:

[0766] "How do we recognise when a visitor is in distress and design a system that can respond appropriately?"

[0767] "Please tell me an example of how emotion recognition is used in a guidance system for a commercial facility."

[0768] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0769] Step 1:

[0770] Accepting user voice input

[0771] The user speaks a question or command, for example, "Excuse me, where is the restaurant?"

[0772] Input: User's voice.

[0773] The device's ultra-directional microphone captures the audio.

[0774] Output: Audio data is generated on the device.

[0775] Step 2:

[0776] Sending audio data

[0777] The device transmits the received voice data to a server via the SIM and the mobile communication network.

[0778] Input: Audio data captured by a superdirectional microphone.

[0779] Output: The audio data sent to the server.

[0780] Step 3:

[0781] Speech Recognition Processing

[0782] The server inputs the received voice data into a voice recognition engine (cloud-based).

[0783] Input: The audio data sent to the server.

[0784] The speech recognition engine converts the voice data into text data, for example, generating the text data "Where is the restaurant?"

[0785] Output: Character data generated within the server.

[0786] Step 4:

[0787] Natural Language Processing

[0788] The server sends the generated character data to a natural language processing engine.

[0789] Input: The text data generated by the speech recognition engine.

[0790] A natural language processing engine analyzes text data to understand the user's intent, for example by extracting the keyword "restaurant."

[0791] Output: Data parsed for user intent.

[0792] Step 5:

[0793] Emotion Recognition Processing

[0794] The server inputs the character data and voice data into the emotion authentication device.

[0795] Input: The text data generated by the speech recognition engine and the original audio data.

[0796] Emotion recognition devices analyze emotions from voice tone and vocabulary, for example, recognizing when a user is distressed.

[0797] Output: Sentiment-analyzed data.

[0798] Step 6:

[0799] Proper response generation

[0800] The server generates an appropriate answer using a response generation means based on the results of the natural language processing engine and the emotion authentication device.

[0801] Input: User intent analyzed by natural language processing and user emotion analyzed by an emotion recognizer.

[0802] The response generation means retrieves information from the database and generates a response message such as "The restaurant is on the third floor, on the right."

[0803] Output: The generated response message.

[0804] Step 7:

[0805] Text-to-speech response messages

[0806] The server sends a response message to the speech synthesis engine.

[0807] Input: The response message generated by the response generator.

[0808] The speech synthesis engine converts text data into speech data, generating a voice file that says, for example, "The restaurant is on the third floor, on your right."

[0809] Output: The generated audio data.

[0810] Step 8:

[0811] Audio data output

[0812] The server transmits the generated voice data to the terminal.

[0813] Input: The speech data generated by the speech synthesis engine.

[0814] The device's ultra-directional speaker plays back the audio data.

[0815] As a specific example, the speaker provides voice guidance in a gentle tone, saying, "The restaurant is on the third floor on the right. We have staff who can guide you, so please ask if you need any help."

[0816] Output: A voice prompt of the processing results that the user hears.

[0817] (Application example 2)

[0818] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0819] In recent years, there has been a demand for automated visitor guidance systems in commercial facilities, public areas, and brick-and-mortar stores. However, existing guidance systems have difficulty protecting visitor privacy and are unable to provide personalized responses that take visitors' emotions into account. This makes it difficult for visitors to quickly and accurately obtain information that meets their needs, resulting in a decline in visitor experience and satisfaction. Given this background, there is a need for the development of a new guidance system that protects visitors' privacy while providing personalized responses that take their emotions into account.

[0820] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a superdirectional microphone that receives a user's voice input, means including a voice recognition engine that converts the voice input into text data, means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, means including a voice synthesis engine that converts the response into voice data, means including a superdirectional speaker that outputs the voice data to the user, means including a SIM that communicates with the server means via a mobile communication network, server means that includes an emotion engine that recognizes emotions from the user's voice input, and means for adjusting the response content based on the emotion recognized by the emotion engine. This enables personalized guidance that corresponds to the emotions of the situation while protecting the visitor's privacy.

[0821] A "super-directional microphone" is a microphone that has the ability to pick up sound only from a specific direction, and is a device that reduces surrounding noise and records the user's voice clearly.

[0822] A "voice recognition engine" is a software system that analyzes voice data and converts it into text data, and is the basic technology for understanding the user's voice.

[0823] A "natural language processing engine" is a software system that analyzes text data to understand the user's intent, and interprets the user's request based on the text data generated by a voice recognition engine.

[0824] The "response generation means" is a system that generates appropriate answers based on the user's intent analyzed by the natural language processing engine, and is responsible for providing responses that meet the user's requests.

[0825] A "speech synthesis engine" is a software system that converts text data into speech data, and is a technology for outputting the generated response as speech.

[0826] An "ultra-directional speaker" is a speaker that has the ability to emit sound in a specific direction, and is a device that provides audio to users while protecting their privacy.

[0827] An "emotion engine" is a software system that analyzes voice and text data to recognize a user's emotions, and understands emotions from the user's tone of voice and choice of words.

[0828] A "SIM" is a module for connecting a device to a cloud server means via a mobile communication network, and is hardware that enables communication between the server and the device.

[0829] "Cloud-based" refers to a technology form that utilizes remote servers on the Internet to process and store data, and does not require the device itself to have high computing power or large amounts of memory.

[0830] This invention is a system for providing guidance to visitors in commercial facilities, public areas, and brick-and-mortar stores, and aims to provide guidance in a natural, conversational style while preserving user privacy. It also aims to improve visitor satisfaction by recognizing the user's emotions and providing personalized responses based on those emotions.

[0831] The system includes the following elements:

[0832] 1. Ultra-directional microphone: A device that has the ability to pick up sound only from a specific direction, clearly recording the user's voice.

[0833] 2. A speech recognition engine included in the server means: A software system that converts a user's voice input into text data.

[0834] 3. Natural language processing engine included in the server means: A software system for analyzing character data and understanding the user's intent.

[0835] 4. Response generation means: A system that generates appropriate responses based on the user's intent.

[0836] 5. A speech synthesis engine included in the server means: A software system that converts the generated answers into speech data.

[0837] 6. Ultra-directional speaker: A speaker that emits sound in a specific direction, a device that provides audio guidance while maintaining privacy.

[0838] 7. Device including SIM: A module that performs communication between the server means and the device via a mobile communication network.

[0839] 8. Emotion engine: A software system that analyzes voice and text data to recognize the user's emotions.

[0840] Let us explain the system's operation through an example:

[0841] Example 1: Product location guidance

[0842] The user asks in a confused voice, "Where is the chocolate?" The device's superdirectional microphone receives the voice and sends the voice data to the server. The server's speech recognition engine converts the voice data into text data, and the natural language processing engine detects the keyword "chocolate" and analyzes the user's intention. Next, the emotion engine recognizes that the user is confused, and the response generation means generates a gentle response saying, "The chocolate is on the first floor, on the left. Shall I show you?" The response generated by the speech synthesis engine is converted into voice data, and the superdirectional speaker provides audio guidance to the user.

[0843] Example 2: Providing sales information

[0844] The user asks in an excited voice, "Are there any sales going on right now?" The device's superdirectional microphone receives the voice and sends the voice data to the server. The server's speech recognition engine converts the voice data into text data, and the natural language processing engine detects the keyword "sale" and analyzes the user's intention. Next, the emotion engine recognizes that the user is excited, and the response generation means generates a response in an excited tone: "Yes, there's a 20% off sale in the clothing department on the first floor!" The speech synthesis engine converts the generated response into voice data, and the superdirectional speaker provides audio guidance to the user.

[0845] As these examples show, this system can improve visitor satisfaction by providing personalized guidance based on emotions while protecting visitor privacy.

[0846] Prompt Sentence Examples

[0847] Prompt when user uses smart glasses to ask for product location:

[0848] "A user asked in a confused voice, 'Where is the chocolate?'

[0849] The voice recognition engine converted the text data. Next, the natural language processing engine analyzed the product location, and the emotion recognition engine recognized the user's confused emotion. The response generation engine generated a response in a gentle tone, and the speech synthesis engine converted it into voice data. The super-directional speaker said in a gentle tone, "The chocolate is on the first floor, on the left. Shall I show you?"

[0850] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0851] Step 1: Receiving Audio Input

[0852] This is the role of the device. The device's built-in superdirectional microphone receives the user's voice input and records it as voice data. This input includes what the user says. For example, the user may ask, "Where is the chocolate?"

[0853] Step 2: Sending audio data

[0854] This is the role of the device. It transmits the received voice data to a cloud-based server via a mobile communication network. The input in this step is the voice data recorded in step 1, and the output is the voice data received by the server.

[0855] Step 3: Voice Recognition

[0856] This is the role of the server. A cloud-based speech recognition engine analyzes the voice data received by the server and converts the voice data into text data. The input is voice data and the output is text data. Specifically, the speech recognition engine analyzes the utterance "Where is the chocolate?" and converts it into text data "Where is the chocolate?"

[0857] Step 4: Natural Language Processing

[0858] This is the role of the server. The natural language processing engine analyzes the converted text data and understands the user's intent. The input is text data, and the output is the analyzed intent data. Specifically, the natural language processing engine detects the keyword "chocolate" and understands that the user is asking where the chocolate is.

[0859] Step 5: Emotion Recognition

[0860] This is the role of the server. The emotion engine analyzes the text data obtained by the natural language processing engine and recognizes the user's emotions. The input is text data, and the output is recognized emotional data. Specifically, the emotion engine recognizes that the user is in a difficult situation.

[0861] Step 6: Response Generation

[0862] This is the role of the server. Based on the recognized emotion data and analyzed intent data, the response generation means generates an appropriate answer. The input is the analyzed intent data and emotion data, and the output is a response message. Specifically, the response generation means generates a gentle response message saying, "The chocolate is on the first floor on the left. Shall I show you?"

[0863] Step 7: Text-to-Speech

[0864] This is the role of the server. The generated response message is converted into voice data by the speech synthesis engine. The input is the response message, and the output is voice data. Specifically, the speech synthesis engine generates voice data saying, "The chocolate is on the first floor on the left. Shall I show you?"

[0865] Step 8: Audio Output

[0866] This is the role of the terminal. The generated voice data is transmitted to the user by the terminal's superdirectional speaker. The input is voice data, and the output is the user receiving the voice. In concrete terms, the terminal's superdirectional speaker provides voice guidance to the user, saying, "The chocolate is on the first floor, on the left. Shall I show you?"

[0867] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0868] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0869] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0870] [Third embodiment]

[0871] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0872] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0873] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0874] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0875] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0876] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0877] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0878] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0879] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0880] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0881] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0882] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0883] DETAILED DESCRIPTION OF THE INVENTION The present invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in commercial facilities and public areas. An embodiment of the present invention will be described in detail below.

[0884] The system includes a device including an ultradirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, and server means including a voice synthesis engine that converts the response into voice data, a device including an ultradirectional speaker that outputs the voice data to the user, and a device including a SIM that communicates with the server means via a mobile communication network.

[0885] Program processing details (natural language explanation)

[0886] User Interface Design

[0887] The devices in this system are designed to be easy for users to operate, with a simple and intuitive UI that accepts touch panels and voice input.

[0888] Building a voice recognition system

[0889] The device's onboard super-directional microphone captures the user's voice input and sends the voice data to a server, which then converts the voice data into text using a cloud-based speech recognition engine. For example, the server uses Google Cloud Speech-to-Text to convert the voice data into text.

[0890] Building a natural language processing system

[0891] The server then sends the converted text data to a natural language processing engine to analyze the user's intent. For example, it uses the Google Language API to properly interpret the user's question or instruction. The server then understands the user's request based on the analysis results and generates an appropriate response.

[0892] Building a speech synthesis system

[0893] The server sends the generated response message to a speech synthesis engine to generate voice data. For example, Google Text-to-Speech is used to convert the text data into voice. The generated voice data is then sent from the server to the device.

[0894] Audio output

[0895] The device's built-in ultra-directional speaker plays back the audio data received from the server, and by emitting audio in a specific direction, it provides audio guidance to users while protecting their privacy.

[0896] Specific examples

[0897] Example 1: Library guide

[0898] User: "Excuse me, where is the restaurant?"

[0899] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0900] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[0901] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[0902] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[0903] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0904] The device's ultra-directional speaker provides a voice prompt: "The restaurant is on the third floor, on your right."

[0905] Example 2: Providing event information

[0906] User: "What's happening today?"

[0907] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[0908] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[0909] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[0910] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[0911] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[0912] The device's ultra-directional speaker provides a voice prompt saying, "There will be a concert starting at 3 o'clock today."

[0913] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in the operation of commercial facilities and public areas.

[0914] The processing flow will be explained below.

[0915] Step 1:

[0916] A user approaches the device and asks, "Where is the bathroom?"

[0917] Step 2:

[0918] The device's ultra-directional microphone captures the user's voice and stores the voice data in a buffer.

[0919] Step 3:

[0920] The terminal generates an HTTP POST request to send the audio data to the server, and attaches the audio data.

[0921] Step 4:

[0922] The server receives the voice data and sends it to the voice recognition engine, specifically by calling the voice recognition engine's API.

[0923] Step 5:

[0924] The server receives text data (e.g., "Where is the toilet?") from the speech recognition engine.

[0925] Step 6:

[0926] The server sends the text data to the natural language processing engine. Specifically, it calls the API of the natural language processing engine and instructs it to analyze the text data.

[0927] Step 7:

[0928] The server receives the analysis results (e.g., the keyword "toilet" and a request to obtain location information) from the natural language processing engine.

[0929] Step 8:

[0930] The server accesses the database and searches for location information of facilities related to the keyword "toilet." For example, it retrieves information that the toilet is on the second floor, on the left.

[0931] Step 9:

[0932] Based on the location information obtained by the server, a response message is generated: "The toilet is on the second floor, on the left."

[0933] Step 10:

[0934] The server sends the response message to the speech synthesis engine to generate voice data. Specifically, it calls the speech synthesis engine's API to convert the text data into voice data.

[0935] Step 11:

[0936] The server returns the generated audio data to the terminal as an HTTP response.

[0937] Step 12:

[0938] The device receives the voice data and provides audible guidance to the user through a super-directional speaker, saying, "The toilet is on the second floor, on your left."

[0939] Example 1

[0940] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0941] Conventional guidance systems in commercial facilities and public areas have faced challenges in protecting privacy when providing information to visitors using voice input, and there has also been a lack of technology to create an interface that is intuitive and easy for visitors to operate.

[0942] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0943] In this invention, the server includes a terminal including a highly directional microphone that receives a user's voice input, an information processing device including a voice recognition engine that converts the voice input into text data, an information processing device including a natural language processing engine that analyzes the text data and understands the user's intention, an information processing device including a response generation means that generates an appropriate answer based on the intention, an information processing device including a voice synthesis engine that converts the answer into voice data, a terminal including a highly directional speaker that outputs the voice data to the user, and a communication module that communicates with the information processing device via a wireless communication network. This makes it possible to provide information to visitors through an intuitive and easy-to-use interface while protecting their privacy.

[0944] "User" refers to a person who uses the system to input voice information and obtain guidance information.

[0945] A "highly directional microphone" is a microphone that can efficiently capture sound from a specific direction.

[0946] "Terminal" refers to a device that a user can operate directly or use for voice input.

[0947] A "voice recognition engine" is software or a system for converting voice data into text data.

[0948] "Information processing device" refers to a computing device that includes a speech recognition engine, a natural language processing engine, a response generation means, and a speech synthesis engine.

[0949] A "natural language processing engine" is software or a system that analyzes text data and understands the user's intent.

[0950] A "response generation means" is software or a system that has the function of generating an appropriate response based on the user's intentions.

[0951] A "speech synthesis engine" is software or a system for converting text data into speech data.

[0952] A "highly directional speaker" is a speaker that can effectively transmit sound in a specific direction.

[0953] "Wireless communication network" refers to the communications infrastructure for transmitting and receiving data between a server and a terminal.

[0954] The term "communication module" refers to a device for communicating with an information processing device through a wireless communication network.

[0955] This invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in a commercial facility or public area. The system includes a terminal including a highly directional microphone that receives voice input from a user, an information processing device including a voice recognition engine that converts the voice input into text data, an information processing device including a natural language processing engine that analyzes the text data and understands the user's intent, an information processing device including a response generation means that generates an appropriate response based on the intent, an information processing device including a voice synthesis engine that converts the response into voice data, a terminal including a highly directional speaker that outputs the voice data to the user, and a communication module that communicates with the information processing device via a wireless communication network.

[0956] Device and Technology Configuration

[0957] Terminal

[0958] Users input questions and instructions into the device by voice. The device is equipped with a highly directional microphone, which can capture voice input from a specific direction with high accuracy. The device then transmits the received voice data to an information processing device via a wireless communication network.

[0959] Information processing device

[0960] The information processing device includes a speech recognition engine, a natural language processing engine, a response generation means, and a speech synthesis engine. Received speech data is first converted into text data using the speech recognition engine. For example, Google Cloud Speech-to-Text is used to convert the speech data into text data. The text data is then sent to the natural language processing engine, which analyzes the user's intent. The Google Language API is used to analyze the user's question or instruction and generate an appropriate response. The response generation means creates an appropriate answer based on the analysis results, and the speech synthesis engine converts the answer into speech data. Text data is converted into speech data using Google Text-to-Speech.

[0961] Audio Output

[0962] The device outputs the received audio data to the user using a highly directional speaker, which transmits audio in a specific direction, preventing it from being heard by those around it and providing the user with audio guidance that protects their privacy.

[0963] Specific examples

[0964] Example 1: Library guide

[0965] User: "Excuse me, where is the restaurant?"

[0966] The terminal's highly directional microphone captures the voice and transmits the voice data to an information processing device.

[0967] The information processing device receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[0968] The information processing device analyzes the keyword "restaurant" using a natural language processing engine.

[0969] The information processing device retrieves optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on your right."

[0970] The information processing device converts the response message into voice data using a voice synthesis engine and transmits the voice data to the terminal.

[0971] The device's highly directional speaker provides voice guidance: "The restaurant is on the third floor, on your right."

[0972] Example 2: Providing event information

[0973] User: "What's happening today?"

[0974] The terminal's highly directional microphone captures the voice and transmits the voice data to an information processing device.

[0975] The information processing device receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[0976] The information processing device analyzes the keyword "today's events" using a natural language processing engine.

[0977] The information processing device acquires the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[0978] The information processing device converts the response message into voice data using a voice synthesis engine and transmits the voice data to the terminal.

[0979] The device's highly directional speaker provides a voice prompt saying, "There will be a concert starting at 3 o'clock today."

[0980] In this way, the invention allows visitors to easily obtain the information they need, while also alleviating the labor shortage in the operation of commercial facilities and public areas.

[0981] Prompt Sentence Examples

[0982] "Generate a program to design a system that provides natural conversational guidance to visitors in a commercial establishment."

[0983] "Please provide detailed instructions in natural language for building a system that recognizes input speech and returns an appropriate response."

[0984] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0985] Step 1: Receiving Audio Input

[0986] The user inputs questions and instructions into the terminal by voice.

[0987] The device's highly directional microphone captures the user's voice input.

[0988] The terminal processes the captured audio data to digitize it and transmits it to a server over a wireless communication network.

[0989] Input: User's spoken command (e.g., "Where is the restaurant?")

[0990] Output: Audio data sent to the server

[0991] Step 2: Convert audio data to text

[0992] The server receives the voice data sent from the terminal.

[0993] The server runs the voice data through a voice recognition engine, specifically a cloud-based voice recognition engine (e.g., a general-purpose voice recognition engine).

[0994] The server converts the voice data into character data (text information) using a voice recognition engine.

[0995] Input: Audio data sent from the device

[0996] Output: Text data (e.g., "Where is the restaurant?")

[0997] Step 3: Parsing character data

[0998] The server sends the generated character data to a natural language processing engine.

[0999] The server uses a natural language processing engine to analyze the text data and interpret the user's intent. Specifically, it uses a "text analysis engine."

[1000] Based on the analysis results, the server identifies the user's requests and questions.

[1001] Input: Text data (e.g., "Where is the restaurant?")

[1002] Output: Interpreted intent (e.g., "asking for restaurant locations")

[1003] Step 4: Generate an appropriate response

[1004] The server generates an appropriate response message based on the user's intent.

[1005] The server retrieves the necessary information from a database or external API. For example, it retrieves the "location of the restaurant" from the facility's floor map information.

[1006] The server uses a "response generation engine" to create a response message based on the information it has obtained.

[1007] Input: Interpreted intent (e.g., "asking for restaurant locations")

[1008] Output: Response message (e.g., "The restaurant is on the third floor, on the right.")

[1009] Step 5: Convert the response message to audio data

[1010] The server sends the generated response message to the speech synthesis engine.

[1011] The server uses a speech synthesis engine to generate voice data from text data. Specifically, it uses a "voice conversion engine."

[1012] The server transmits the generated voice data to the terminal.

[1013] Input: Response message (character data)

[1014] Output: Speech data (e.g., "The restaurant is on the third floor, on the right.")

[1015] Step 6: Outputting audio

[1016] The terminal receives the voice data sent from the server.

[1017] The terminal uses a highly directional speaker to play back the received audio data, allowing users to receive guidance information in a manner that protects their privacy without being seen by other visitors.

[1018] Input: Audio data sent from the server

[1019] Output: Spoken directions provided to the user (e.g., "The restaurant is on the third floor, on your right.")

[1020] Through these steps, the system can provide visitors with appropriate information while functioning efficiently and protecting their privacy.

[1021] (Application example 1)

[1022] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1023] In recent years, there has been a demand for methods to quickly and accurately provide visitors with the information they need in commercial facilities and public areas. However, previous guidance systems have had problems such as complex user interfaces and insufficient privacy protection. Furthermore, providing highly accurate information and guidance in real time has been difficult. In brick-and-mortar stores in particular, the inability to smoothly provide product information and location information has led to issues that can lead to lower customer satisfaction and reduced business efficiency.

[1024] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1025] In this invention, the server includes a device including a superdirectional microphone that receives a user's voice input, means including a voice recognition engine that converts the voice input into text data, means including a natural language processing engine that analyzes the text data and understands the user's intent, response generation means that generates a response message, means including a voice synthesis engine that converts the response into voice data, and means that provides the voice data to the user via a mobile information terminal with a built-in speaker or a mobile machine with a superdirectional speaker. This enables real-time, highly accurate, and privacy-protected guidance via voice input. Furthermore, use in commercial facilities and brick-and-mortar stores can improve customer convenience and streamline store operations.

[1026] 1. A "superdirectional microphone" is a microphone that has the ability to capture only sounds from a specific direction, and is a device that can accurately capture specific sounds by eliminating surrounding noise.

[1027] 2. "Speech Recognition Engine" means a software or hardware component that analyzes and converts voice input into text data.

[1028] 3. "Text data" means text data converted from speech by a speech recognition engine.

[1029] 4. A "natural language processing engine" is an algorithm or system that analyzes input text data and understands the user's intent and meaning.

[1030] 5. "Response generator" means a system or algorithm that creates an appropriate response message based on the user's intent as analyzed by the natural language processing engine.

[1031] 6. "Speech synthesis engine" means a software or hardware component that has the function of converting text data into speech data.

[1032] 7. An "ultra-directional speaker" is a speaker that has the ability to emit sound in a specific direction and is a device designed so that only specific users can hear the sound.

[1033] 8. "Personal digital assistant" means a portable device (e.g., smartphone, tablet) with a built-in speaker for providing audio data to the user.

[1034] 9. "Mobile Machine" means a robot or guide device that contains an ultradirectional speaker and is capable of moving to provide audio data to a user.

[1035] 10. "Mobile communication network" means a network that provides mobile communications for data communication between the entire system and the server.

[1036] 11. "SIM" means a subscriber identity module for use in a mobile communications network.

[1037] DETAILED DESCRIPTION OF THE INVENTION The present invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in commercial facilities and public areas. An embodiment of the present invention will be described in detail below.

[1038] The system includes a device including a superdirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intent, response generation means that generates an appropriate response based on the intent, server means including a voice synthesis engine that converts the response into voice data, and a device including a superdirectional speaker that outputs the voice data to the user, or a mobile information terminal with a built-in speaker. These components also comprise a device including a SIM that communicates with the server means via a mobile communication network.

[1039] Program processing details (natural language explanation)

[1040] User Interface Design

[1041] The smartphones and robots used as terminals are equipped with a user interface that allows voice input and simple touch operations, making it easy for users to use the guidance service.

[1042] Building a voice recognition system

[1043] The device's built-in or super-directional microphone captures the user's voice input, which is then sent to a server, where it converts the speech into text using a cloud-based speech recognition engine such as Google Cloud Speech-to-Text.

[1044] Building a natural language processing system

[1045] The server then sends the converted text data to a natural language processing engine such as Google Language API to analyze the user's intent, allowing it to properly understand the user's question or instruction and generate a response.

[1046] Building a speech synthesis system

[1047] The server sends the generated response message to a speech synthesis engine such as Google Text-to-Speech to generate voice data, which is then sent to the device.

[1048] Audio output

[1049] Voice data synthesized on the server is played from the device's ultra-directional speaker or built-in speaker, providing guidance to the user while protecting their privacy.

[1050] Specific examples

[1051] Example 1: Checking product location

[1052] User: "Excuse me, where is the tea?"

[1053] The device's ultra-directional microphone or built-in microphone captures the sound and transmits the audio data to the server.

[1054] The server converts the voice data into text data and parses it as "Where is the tea?"

[1055] The server identifies the keyword "tea" and retrieves the most appropriate information from a product location database.

[1056] The server generates a response message saying, "Tea is in aisle 3 of the beverage section."

[1057] The server converts this response message into voice data and transmits it to the terminal.

[1058] The device's ultra-directional speaker or built-in speaker announces, "Tea is located in aisle 3 of the beverage section."

[1059] Examples of prompt statements

[1060] Example prompt sentence:

[1061] Please provide a sample audio of a user asking, "Where is the tea?"

[1062] In this way, the present invention allows visitors to easily obtain the information they need, and also improves the efficiency of operations in commercial facilities and public areas, thereby alleviating labor shortages.

[1063] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1064] Step 1:

[1065] This is a situation where a user performs voice input. The user asks a question to a device (e.g., a smartphone or a robot) such as "Where is the tea?" The input is the user's voice, and the output is an analog voice signal.

[1066] Step 2:

[1067] The device's superdirectional microphone or built-in microphone receives the user's voice. The received voice is converted into digital voice data and sent to the server. The input is an analog voice signal, and the output is digital voice data.

[1068] Step 3:

[1069] The server sends the received voice data to a speech recognition engine such as Google Cloud Speech-to-Text, where it is converted into text data. At this stage, the voice data is analyzed and output as text data. The input is digital voice data, and the output is text data.

[1070] Step 4:

[1071] The server sends the text data to a natural language processing engine such as Google Language API to analyze the user's intent. The server extracts the keyword "tea" from the text data "Where is the tea?" and uses this keyword to search the appropriate database. The input is text data, and the output is the search query and search results.

[1072] Step 5:

[1073] The server searches the database based on the analyzed keywords and generates a response message such as "Tea is in aisle 3 of the beverage section." The response generator creates an appropriate answer in text format. The input is the search results, and the output is the text response message.

[1074] Step 6:

[1075] The server sends the generated response message to a speech synthesis engine such as Google Text-to-Speech to generate audio data. The server converts the text data into audio data and sends it to the device. The input is the text response message, and the output is digital audio data.

[1076] Step 7:

[1077] The device's ultra-directional speaker or built-in speaker plays the audio data sent from the server. The user receives a voice prompt saying, "Tea is in aisle 3 in the beverage section." The input is digital audio data, and the output is the audio being played.

[1078] This will enable users to obtain necessary information in real time through voice guidance in commercial facilities and public areas.

[1079] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1080] The present invention combines a system for providing visitor guidance in commercial facilities and public areas through natural conversation while protecting privacy with an emotion engine that recognizes user emotions. Hereinafter, an embodiment of the present invention will be described in detail.

[1081] This system comprises a device including an ultradirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, server means including a voice synthesis engine that converts the response into voice data, a device including an ultradirectional speaker that outputs the voice data to the user, and a device including a SIM that communicates with the server means via a mobile communication network.Furthermore, this invention incorporates an emotion engine that recognizes emotions from the user's voice input.

[1082] Program processing details (natural language explanation)

[1083] User Interface Design

[1084] The devices in this system are designed to be easy for users to operate, with a simple and intuitive UI that accepts touch panels and voice input.

[1085] Building a voice recognition system

[1086] The device's onboard super-directional microphone captures the user's voice input and transmits the voice data to a server, which then converts the voice data into text using a cloud-based speech recognition engine. For example, the server may use a cloud speech recognition service to convert the voice data into text.

[1087] Building a natural language processing system

[1088] The server then sends the converted text data to a natural language processing engine to analyze the user's intent. For example, it uses a natural language processing service to properly interpret the user's questions and instructions. The server then understands the user's request based on the analysis results and generates an appropriate response.

[1089] Building an Emotion Engine

[1090] The server sends the text data generated by speech recognition to an emotion engine, which analyzes the user's emotions. For example, in cooperation with a natural language processing engine, the emotion engine can recognize emotions from the user's tone of voice and vocabulary. For example, if the user is angry or in distress, it can respond accordingly.

[1091] Enhanced response generation

[1092] The server uses the emotion engine to generate a response that matches the user's emotion. For example, if the user is in trouble, the response generator will use a gentle tone. This allows for more personalized responses.

[1093] Building a speech synthesis system

[1094] The server sends the generated response message to a speech synthesis engine to generate voice data. For example, a speech synthesis service is used to convert text data into voice. The generated voice data is then sent from the server to the device.

[1095] Audio output

[1096] The device's built-in ultra-directional speaker plays back the audio data received from the server, and by emitting audio in a specific direction, it provides audio guidance to users while protecting their privacy.

[1097] Specific examples

[1098] Example 1: Library guide

[1099] User: "Excuse me, where is the restaurant?" (in a slightly confused voice)

[1100] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[1101] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[1102] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[1103] The server uses an emotion engine to recognize when the user is in trouble.

[1104] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[1105] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[1106] The device's ultra-directional speaker provides gentle voice guidance, saying, "The restaurant is on the third floor, on the right. We have staff who can guide you, so please ask if you need any help."

[1107] Example 2: Providing event information

[1108] User: "What's happening today?" (excited)

[1109] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[1110] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[1111] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[1112] The server uses an emotion engine to recognize when the user is excited.

[1113] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[1114] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[1115] The device's ultra-directional speaker provides an excited voice prompt: "There's a concert starting at 3 o'clock today. I'm really looking forward to it!"

[1116] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in commercial facilities and public areas. Furthermore, emotion recognition capabilities enable more personalized responses, further improving visitor satisfaction.

[1117] The processing flow will be explained below.

[1118] Step 1:

[1119] A user approaches the device and asks, "Where is the bathroom?"

[1120] Step 2:

[1121] The device's ultra-directional microphone captures the user's voice and stores the voice data in a buffer.

[1122] Step 3:

[1123] The terminal generates an HTTP POST request to send the audio data to the server, and attaches the audio data.

[1124] Step 4:

[1125] The server receives the voice data and sends it to the voice recognition engine, specifically by calling the voice recognition engine's API.

[1126] Step 5:

[1127] The server receives text data (e.g., "Where is the toilet?") from the speech recognition engine.

[1128] Step 6:

[1129] The server sends the text data to the emotion engine for emotion analysis. Specifically, it calls the emotion engine's API to analyze the text data.

[1130] Step 7:

[1131] The server receives the analysis result (e.g., the user is in trouble) from the emotion engine.

[1132] Step 8:

[1133] The server sends the text data to the natural language processing engine. Specifically, it calls the API of the natural language processing engine and instructs it to analyze the text data.

[1134] Step 9:

[1135] The server receives the analysis results (e.g., the keyword "toilet" and a request to obtain location information) from the natural language processing engine.

[1136] Step 10:

[1137] The server accesses the database and searches for location information of facilities related to the keyword "toilet." For example, it retrieves information that the toilet is on the second floor, on the left.

[1138] Step 11:

[1139] Based on the location information acquired by the server and the analysis results of the emotion engine, a response message (e.g., "The toilet is on the second floor, on the left. We have staff who can guide you, so please let us know if you need help") is generated.

[1140] Step 12:

[1141] The server sends the response message to the speech synthesis engine to generate voice data. Specifically, it calls the speech synthesis engine's API to convert the text data into voice data.

[1142] Step 13:

[1143] The server returns the generated audio data to the terminal as an HTTP response.

[1144] Step 14:

[1145] The device receives the voice data and provides audio guidance to the user through an ultra-directional speaker: "The toilet is on the second floor, on the left. We have staff who can guide you, so please let us know if you need any help."

[1146] Example 2

[1147] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1148] In recent years, there has been a demand for systems that can provide visitors with information through natural conversation while protecting their privacy in commercial facilities and public areas. However, conventional systems have the following problems.

[1149] 1. If the recognition accuracy of voice input is low, it is difficult to accurately understand the user's intention.

[1150] 2. It is difficult for general audio output devices to provide directional audio output while protecting privacy.

[1151] 3. The lack of functionality to recognize user emotions and respond appropriately can lead to low visitor satisfaction.

[1152] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1153] In this invention, the server includes means including an audio input device that receives a user's voice input, means including a voice recognition device that converts the voice input into text data, means including a natural language processing device that analyzes the text data and understands the user's intention, means including a response generation device that generates an appropriate answer based on the intention, means including an emotion authentication device that recognizes the user's emotion, means for enhancing the response generation device that generates an appropriate answer based on the emotion, means including a voice synthesis device that converts the answer into voice data, means including an audio output device that outputs the voice data to the user, and communication means for communicating with the processing device via a communication network. This makes it possible to understand the user's intention and emotion through highly accurate recognition of the voice input and natural conversation, and to provide appropriate voice guidance while protecting privacy.

[1154] An "acoustic input device" is a device that collects sound from a specific direction, and includes a directional microphone.

[1155] A "voice recognition device" is a device that converts voice data into text data, and includes systems that perform highly accurate voice recognition using cloud services.

[1156] A "natural language processing device" is a device that analyzes text data and understands the user's intent, and uses a natural language processing engine.

[1157] A "response generation device" is a device that generates an appropriate response based on the user's intentions, and outputs the generated response as text data.

[1158] An "emotion authentication device" is a device that analyzes emotions from a user's voice and text data, and recognizes the user's emotional state.

[1159] The "means for strengthening the response generation device" is a means for enabling the response generation device to provide a response according to the user's emotions based on the results of the emotion authentication device.

[1160] A "speech synthesizer" is a device that converts character data into voice data, and includes a system that generates synthetic voice.

[1161] An "audio output device" is a device that outputs sound in a specific direction, and includes a directional speaker.

[1162] "Communication means" refers to a means for data communication between a terminal and a server, and includes a SIM card or module for communication via a mobile communication network.

[1163] The present invention combines a system for providing visitor guidance in commercial facilities and public areas through natural conversation while protecting privacy with an emotion engine that recognizes user emotions. Hereinafter, an embodiment of the present invention will be described in detail.

[1164] The system includes the following hardware and software: a device including an ultradirectional microphone for receiving a user's voice input, server means including a voice recognition engine for converting the voice input into text data, server means including a natural language processing engine for analyzing the text data and understanding the user's intention, response generation means for generating an appropriate response based on the user's intention, an emotion authentication device for recognizing the user's emotion, means for enhancing the response generation device for generating an appropriate response based on the emotion, server means including a voice synthesis engine for converting the response into voice data, a device including an ultradirectional speaker for outputting the voice data to the user, and a device including a SIM for communicating with the server means via a mobile communication network.

[1165] First, the user speaks a question or instruction to the system. For example, the user might say, "Excuse me, where is the restaurant?" This voice input is received by the device's superdirectional microphone, and the device transmits the captured voice data via the SIM card and the mobile communication network to a server. The server converts the voice data into text data using a cloud-based speech recognition engine (e.g., Google Cloud Speech-to-Text), and sends this text data to a natural language processing engine (e.g., Google Natural Language API) to analyze the user's intent.

[1166] Next, the server sends the recognized text data and voice data to an emotion recognition device, which analyzes the user's emotions. This emotion recognition device (for example, Azure Cognitive Services' Emotion Analysis API) is used to recognize emotions from the user's tone of voice and phrasing. For example, if the user is distressed or excited, the emotion is identified. The server then activates a response generation device to generate an appropriate response based on the results of the natural language processing engine and emotion recognition device. The response generation device retrieves information from a database and creates a response message, such as "The restaurant is on the third floor, on the right." The response generation device is also enhanced to generate responses in a tone that corresponds to the emotion.

[1167] After the response message is generated, the server sends it to a speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) to generate voice data. The generated voice data is sent from the server to the device and played back by the device's super-directional speaker. This speaker emits voice in a specific direction, providing voice guidance to the user while protecting their privacy.

[1168] Example 1: Museum guide

[1169] User: "Excuse me, where is the restaurant?" (in a slightly confused voice)

[1170] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[1171] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[1172] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[1173] The server uses an emotion authentication device to recognize that the user is in trouble.

[1174] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[1175] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[1176] The device's ultra-directional speaker provides gentle voice guidance, saying, "The restaurant is on the third floor, on the right. We have staff who can guide you, so please ask if you need any help."

[1177] Example 2: Providing event information

[1178] User: "What's happening today?" (excited)

[1179] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[1180] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[1181] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[1182] The server uses an emotion authentication device to recognize that the user is excited.

[1183] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[1184] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[1185] The device's ultra-directional speaker provides an excited voice prompt: "There's a concert starting at 3 o'clock today. I'm really looking forward to it!"

[1186] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in commercial facilities and public areas. Furthermore, emotion recognition capabilities enable more personalized responses, further improving visitor satisfaction.

[1187] Example prompt sentence:

[1188] "How do we recognise when a visitor is in distress and design a system that can respond appropriately?"

[1189] "Please tell me an example of how emotion recognition is used in a guidance system for a commercial facility."

[1190] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1191] Step 1:

[1192] Accepting user voice input

[1193] The user speaks a question or command, for example, "Excuse me, where is the restaurant?"

[1194] Input: User's voice.

[1195] The device's ultra-directional microphone captures the audio.

[1196] Output: Audio data is generated on the device.

[1197] Step 2:

[1198] Sending audio data

[1199] The device transmits the received voice data to a server via the SIM and the mobile communication network.

[1200] Input: Audio data captured by a superdirectional microphone.

[1201] Output: The audio data sent to the server.

[1202] Step 3:

[1203] Speech Recognition Processing

[1204] The server inputs the received voice data into a voice recognition engine (cloud-based).

[1205] Input: The audio data sent to the server.

[1206] The speech recognition engine converts the voice data into text data, for example, generating the text data "Where is the restaurant?"

[1207] Output: Character data generated within the server.

[1208] Step 4:

[1209] Natural Language Processing

[1210] The server sends the generated character data to a natural language processing engine.

[1211] Input: The text data generated by the speech recognition engine.

[1212] A natural language processing engine analyzes text data to understand the user's intent, for example by extracting the keyword "restaurant."

[1213] Output: Data parsed for user intent.

[1214] Step 5:

[1215] Emotion Recognition Processing

[1216] The server inputs the character data and voice data into the emotion authentication device.

[1217] Input: The text data generated by the speech recognition engine and the original audio data.

[1218] Emotion recognition devices analyze emotions from voice tone and vocabulary, for example, recognizing when a user is distressed.

[1219] Output: Sentiment-analyzed data.

[1220] Step 6:

[1221] Proper response generation

[1222] The server generates an appropriate answer using a response generation means based on the results of the natural language processing engine and the emotion authentication device.

[1223] Input: User intent analyzed by natural language processing and user emotion analyzed by an emotion recognizer.

[1224] The response generation means retrieves information from the database and generates a response message such as "The restaurant is on the third floor, on the right."

[1225] Output: The generated response message.

[1226] Step 7:

[1227] Text-to-speech response messages

[1228] The server sends a response message to the speech synthesis engine.

[1229] Input: The response message generated by the response generator.

[1230] The speech synthesis engine converts text data into speech data, generating a voice file that says, for example, "The restaurant is on the third floor, on your right."

[1231] Output: The generated audio data.

[1232] Step 8:

[1233] Audio data output

[1234] The server transmits the generated voice data to the terminal.

[1235] Input: The speech data generated by the speech synthesis engine.

[1236] The device's ultra-directional speaker plays back the audio data.

[1237] As a specific example, the speaker provides voice guidance in a gentle tone, saying, "The restaurant is on the third floor on the right. We have staff who can guide you, so please ask if you need any help."

[1238] Output: A voice prompt of the processing results that the user hears.

[1239] (Application example 2)

[1240] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1241] In recent years, there has been a demand for automated visitor guidance systems in commercial facilities, public areas, and brick-and-mortar stores. However, existing guidance systems have difficulty protecting visitor privacy and are unable to provide personalized responses that take visitors' emotions into account. This makes it difficult for visitors to quickly and accurately obtain information that meets their needs, resulting in a decline in visitor experience and satisfaction. Given this background, there is a need for the development of a new guidance system that protects visitors' privacy while providing personalized responses that take their emotions into account.

[1242] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a superdirectional microphone that receives a user's voice input, means including a voice recognition engine that converts the voice input into text data, means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, means including a voice synthesis engine that converts the response into voice data, means including a superdirectional speaker that outputs the voice data to the user, means including a SIM that communicates with the server means via a mobile communication network, server means that includes an emotion engine that recognizes emotions from the user's voice input, and means for adjusting the response content based on the emotion recognized by the emotion engine. This enables personalized guidance that corresponds to the emotions of the situation while protecting the visitor's privacy.

[1243] A "super-directional microphone" is a microphone that has the ability to pick up sound only from a specific direction, and is a device that reduces surrounding noise and records the user's voice clearly.

[1244] A "voice recognition engine" is a software system that analyzes voice data and converts it into text data, and is the basic technology for understanding the user's voice.

[1245] A "natural language processing engine" is a software system that analyzes text data to understand the user's intent, and interprets the user's request based on the text data generated by a voice recognition engine.

[1246] The "response generation means" is a system that generates appropriate answers based on the user's intent analyzed by the natural language processing engine, and is responsible for providing responses that meet the user's requests.

[1247] A "speech synthesis engine" is a software system that converts text data into speech data, and is a technology for outputting the generated response as speech.

[1248] An "ultra-directional speaker" is a speaker that has the ability to emit sound in a specific direction, and is a device that provides audio to users while protecting their privacy.

[1249] An "emotion engine" is a software system that analyzes voice and text data to recognize a user's emotions, and understands emotions from the user's tone of voice and choice of words.

[1250] A "SIM" is a module for connecting a device to a cloud server means via a mobile communication network, and is hardware that enables communication between the server and the device.

[1251] "Cloud-based" refers to a technology form that utilizes remote servers on the Internet to process and store data, and does not require the device itself to have high computing power or large amounts of memory.

[1252] This invention is a system for providing guidance to visitors in commercial facilities, public areas, and brick-and-mortar stores, and aims to provide guidance in a natural, conversational style while preserving user privacy. It also aims to improve visitor satisfaction by recognizing the user's emotions and providing personalized responses based on those emotions.

[1253] The system includes the following elements:

[1254] 1. Ultra-directional microphone: A device that has the ability to pick up sound only from a specific direction, clearly recording the user's voice.

[1255] 2. A speech recognition engine included in the server means: A software system that converts a user's voice input into text data.

[1256] 3. Natural language processing engine included in the server means: A software system for analyzing character data and understanding the user's intent.

[1257] 4. Response generation means: A system that generates appropriate responses based on the user's intent.

[1258] 5. A speech synthesis engine included in the server means: A software system that converts the generated answers into speech data.

[1259] 6. Ultra-directional speaker: A speaker that emits sound in a specific direction, a device that provides audio guidance while maintaining privacy.

[1260] 7. Device including SIM: A module that performs communication between the server means and the device via a mobile communication network.

[1261] 8. Emotion engine: A software system that analyzes voice and text data to recognize the user's emotions.

[1262] Let us explain the system's operation through an example:

[1263] Example 1: Product location guidance

[1264] The user asks in a confused voice, "Where is the chocolate?" The device's superdirectional microphone receives the voice and sends the voice data to the server. The server's speech recognition engine converts the voice data into text data, and the natural language processing engine detects the keyword "chocolate" and analyzes the user's intention. Next, the emotion engine recognizes that the user is confused, and the response generation means generates a gentle response saying, "The chocolate is on the first floor, on the left. Shall I show you?" The response generated by the speech synthesis engine is converted into voice data, and the superdirectional speaker provides audio guidance to the user.

[1265] Example 2: Providing sales information

[1266] The user asks in an excited voice, "Are there any sales going on right now?" The device's superdirectional microphone receives the voice and sends the voice data to the server. The server's speech recognition engine converts the voice data into text data, and the natural language processing engine detects the keyword "sale" and analyzes the user's intention. Next, the emotion engine recognizes that the user is excited, and the response generation means generates a response in an excited tone: "Yes, there's a 20% off sale in the clothing department on the first floor!" The speech synthesis engine converts the generated response into voice data, and the superdirectional speaker provides audio guidance to the user.

[1267] As these examples show, this system can improve visitor satisfaction by providing personalized guidance based on emotions while protecting visitor privacy.

[1268] Prompt Sentence Examples

[1269] Prompt when user uses smart glasses to ask for product location:

[1270] "A user asked in a confused voice, 'Where is the chocolate?'

[1271] The voice recognition engine converted the text data. Next, the natural language processing engine analyzed the product location, and the emotion recognition engine recognized the user's confused emotion. The response generation engine generated a response in a gentle tone, and the speech synthesis engine converted it into voice data. The super-directional speaker said in a gentle tone, "The chocolate is on the first floor, on the left. Shall I show you?"

[1272] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1273] Step 1: Receiving Audio Input

[1274] This is the role of the device. The device's built-in superdirectional microphone receives the user's voice input and records it as voice data. This input includes what the user says. For example, the user may ask, "Where is the chocolate?"

[1275] Step 2: Sending audio data

[1276] This is the role of the device. It transmits the received voice data to a cloud-based server via a mobile communication network. The input in this step is the voice data recorded in step 1, and the output is the voice data received by the server.

[1277] Step 3: Voice Recognition

[1278] This is the role of the server. A cloud-based speech recognition engine analyzes the voice data received by the server and converts the voice data into text data. The input is voice data and the output is text data. Specifically, the speech recognition engine analyzes the utterance "Where is the chocolate?" and converts it into text data "Where is the chocolate?"

[1279] Step 4: Natural Language Processing

[1280] This is the role of the server. The natural language processing engine analyzes the converted text data and understands the user's intent. The input is text data, and the output is the analyzed intent data. Specifically, the natural language processing engine detects the keyword "chocolate" and understands that the user is asking where the chocolate is.

[1281] Step 5: Emotion Recognition

[1282] This is the role of the server. The emotion engine analyzes the text data obtained by the natural language processing engine and recognizes the user's emotions. The input is text data, and the output is recognized emotional data. Specifically, the emotion engine recognizes that the user is in a difficult situation.

[1283] Step 6: Response Generation

[1284] This is the role of the server. Based on the recognized emotion data and analyzed intent data, the response generation means generates an appropriate answer. The input is the analyzed intent data and emotion data, and the output is a response message. Specifically, the response generation means generates a gentle response message saying, "The chocolate is on the first floor on the left. Shall I show you?"

[1285] Step 7: Text-to-Speech

[1286] This is the role of the server. The generated response message is converted into voice data by the speech synthesis engine. The input is the response message, and the output is voice data. Specifically, the speech synthesis engine generates voice data saying, "The chocolate is on the first floor on the left. Shall I show you?"

[1287] Step 8: Audio Output

[1288] This is the role of the terminal. The generated voice data is transmitted to the user by the terminal's superdirectional speaker. The input is voice data, and the output is the user receiving the voice. In concrete terms, the terminal's superdirectional speaker provides voice guidance to the user, saying, "The chocolate is on the first floor, on the left. Shall I show you?"

[1289] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1290] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1291] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1292] [Fourth embodiment]

[1293] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1294] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1295] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1296] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1297] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1298] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1299] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1300] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1301] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1302] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1303] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1304] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1305] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1306] DETAILED DESCRIPTION OF THE INVENTION The present invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in commercial facilities and public areas. An embodiment of the present invention will be described in detail below.

[1307] The system includes a device including an ultradirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, and server means including a voice synthesis engine that converts the response into voice data, a device including an ultradirectional speaker that outputs the voice data to the user, and a device including a SIM that communicates with the server means via a mobile communication network.

[1308] Program processing details (natural language explanation)

[1309] User Interface Design

[1310] The devices in this system are designed to be easy for users to operate, with a simple and intuitive UI that accepts touch panels and voice input.

[1311] Building a voice recognition system

[1312] The device's onboard super-directional microphone captures the user's voice input and sends the voice data to a server, which then converts the voice data into text using a cloud-based speech recognition engine. For example, the server uses Google Cloud Speech-to-Text to convert the voice data into text.

[1313] Building a natural language processing system

[1314] The server then sends the converted text data to a natural language processing engine to analyze the user's intent. For example, it uses the Google Language API to properly interpret the user's question or instruction. The server then understands the user's request based on the analysis results and generates an appropriate response.

[1315] Building a speech synthesis system

[1316] The server sends the generated response message to a speech synthesis engine to generate voice data. For example, Google Text-to-Speech is used to convert the text data into voice. The generated voice data is then sent from the server to the device.

[1317] Audio output

[1318] The device's built-in ultra-directional speaker plays back the audio data received from the server, and by emitting audio in a specific direction, it provides audio guidance to users while protecting their privacy.

[1319] Specific examples

[1320] Example 1: Library guide

[1321] User: "Excuse me, where is the restaurant?"

[1322] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[1323] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[1324] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[1325] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[1326] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[1327] The device's ultra-directional speaker provides a voice prompt: "The restaurant is on the third floor, on your right."

[1328] Example 2: Providing event information

[1329] User: "What's happening today?"

[1330] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[1331] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[1332] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[1333] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[1334] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[1335] The device's ultra-directional speaker provides a voice prompt saying, "There will be a concert starting at 3 o'clock today."

[1336] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in the operation of commercial facilities and public areas.

[1337] The processing flow will be explained below.

[1338] Step 1:

[1339] A user approaches the device and asks, "Where is the bathroom?"

[1340] Step 2:

[1341] The device's ultra-directional microphone captures the user's voice and stores the voice data in a buffer.

[1342] Step 3:

[1343] The terminal generates an HTTP POST request to send the audio data to the server, and attaches the audio data.

[1344] Step 4:

[1345] The server receives the voice data and sends it to the voice recognition engine, specifically by calling the voice recognition engine's API.

[1346] Step 5:

[1347] The server receives text data (e.g., "Where is the toilet?") from the speech recognition engine.

[1348] Step 6:

[1349] The server sends the text data to the natural language processing engine. Specifically, it calls the API of the natural language processing engine and instructs it to analyze the text data.

[1350] Step 7:

[1351] The server receives the analysis results (e.g., the keyword "toilet" and a request to obtain location information) from the natural language processing engine.

[1352] Step 8:

[1353] The server accesses the database and searches for location information of facilities related to the keyword "toilet." For example, it retrieves information that the toilet is on the second floor, on the left.

[1354] Step 9:

[1355] Based on the location information obtained by the server, a response message is generated: "The toilet is on the second floor, on the left."

[1356] Step 10:

[1357] The server sends the response message to the speech synthesis engine to generate voice data. Specifically, it calls the speech synthesis engine's API to convert the text data into voice data.

[1358] Step 11:

[1359] The server returns the generated audio data to the terminal as an HTTP response.

[1360] Step 12:

[1361] The device receives the voice data and provides audible guidance to the user through a super-directional speaker, saying, "The toilet is on the second floor, on your left."

[1362] Example 1

[1363] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1364] Conventional guidance systems in commercial facilities and public areas have faced challenges in protecting privacy when providing information to visitors using voice input, and there has also been a lack of technology to create an interface that is intuitive and easy for visitors to operate.

[1365] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1366] In this invention, the server includes a terminal including a highly directional microphone that receives a user's voice input, an information processing device including a voice recognition engine that converts the voice input into text data, an information processing device including a natural language processing engine that analyzes the text data and understands the user's intention, an information processing device including a response generation means that generates an appropriate answer based on the intention, an information processing device including a voice synthesis engine that converts the answer into voice data, a terminal including a highly directional speaker that outputs the voice data to the user, and a communication module that communicates with the information processing device via a wireless communication network. This makes it possible to provide information to visitors through an intuitive and easy-to-use interface while protecting their privacy.

[1367] "User" refers to a person who uses the system to input voice information and obtain guidance information.

[1368] A "highly directional microphone" is a microphone that can efficiently capture sound from a specific direction.

[1369] "Terminal" refers to a device that a user can operate directly or use for voice input.

[1370] A "voice recognition engine" is software or a system for converting voice data into text data.

[1371] "Information processing device" refers to a computing device that includes a speech recognition engine, a natural language processing engine, a response generation means, and a speech synthesis engine.

[1372] A "natural language processing engine" is software or a system that analyzes text data and understands the user's intent.

[1373] A "response generation means" is software or a system that has the function of generating an appropriate response based on the user's intentions.

[1374] A "speech synthesis engine" is software or a system for converting text data into speech data.

[1375] A "highly directional speaker" is a speaker that can effectively transmit sound in a specific direction.

[1376] "Wireless communication network" refers to the communications infrastructure for transmitting and receiving data between a server and a terminal.

[1377] The term "communication module" refers to a device for communicating with an information processing device through a wireless communication network.

[1378] This invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in a commercial facility or public area. The system includes a terminal including a highly directional microphone that receives voice input from a user, an information processing device including a voice recognition engine that converts the voice input into text data, an information processing device including a natural language processing engine that analyzes the text data and understands the user's intent, an information processing device including a response generation means that generates an appropriate response based on the intent, an information processing device including a voice synthesis engine that converts the response into voice data, a terminal including a highly directional speaker that outputs the voice data to the user, and a communication module that communicates with the information processing device via a wireless communication network.

[1379] Device and Technology Configuration

[1380] Terminal

[1381] Users input questions and instructions into the device by voice. The device is equipped with a highly directional microphone, which can capture voice input from a specific direction with high accuracy. The device then transmits the received voice data to an information processing device via a wireless communication network.

[1382] Information processing device

[1383] The information processing device includes a speech recognition engine, a natural language processing engine, a response generation means, and a speech synthesis engine. Received speech data is first converted into text data using the speech recognition engine. For example, Google Cloud Speech-to-Text is used to convert the speech data into text data. The text data is then sent to the natural language processing engine, which analyzes the user's intent. The Google Language API is used to analyze the user's question or instruction and generate an appropriate response. The response generation means creates an appropriate answer based on the analysis results, and the speech synthesis engine converts the answer into speech data. Text data is converted into speech data using Google Text-to-Speech.

[1384] Audio Output

[1385] The device outputs the received audio data to the user using a highly directional speaker, which transmits audio in a specific direction, preventing it from being heard by those around it and providing the user with audio guidance that protects their privacy.

[1386] Specific examples

[1387] Example 1: Library guide

[1388] User: "Excuse me, where is the restaurant?"

[1389] The terminal's highly directional microphone captures the voice and transmits the voice data to an information processing device.

[1390] The information processing device receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[1391] The information processing device analyzes the keyword "restaurant" using a natural language processing engine.

[1392] The information processing device retrieves optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on your right."

[1393] The information processing device converts the response message into voice data using a voice synthesis engine and transmits the voice data to the terminal.

[1394] The device's highly directional speaker provides voice guidance: "The restaurant is on the third floor, on your right."

[1395] Example 2: Providing event information

[1396] User: "What's happening today?"

[1397] The terminal's highly directional microphone captures the voice and transmits the voice data to an information processing device.

[1398] The information processing device receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[1399] The information processing device analyzes the keyword "today's events" using a natural language processing engine.

[1400] The information processing device acquires the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[1401] The information processing device converts the response message into voice data using a voice synthesis engine and transmits the voice data to the terminal.

[1402] The device's highly directional speaker provides a voice prompt saying, "There will be a concert starting at 3 o'clock today."

[1403] In this way, the invention allows visitors to easily obtain the information they need, while also alleviating the labor shortage in the operation of commercial facilities and public areas.

[1404] Prompt Sentence Examples

[1405] "Generate a program to design a system that provides natural conversational guidance to visitors in a commercial establishment."

[1406] "Please provide detailed instructions in natural language for building a system that recognizes input speech and returns an appropriate response."

[1407] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1408] Step 1: Receiving Audio Input

[1409] The user inputs questions and instructions into the terminal by voice.

[1410] The device's highly directional microphone captures the user's voice input.

[1411] The terminal processes the captured audio data to digitize it and transmits it to a server over a wireless communication network.

[1412] Input: User's spoken command (e.g., "Where is the restaurant?")

[1413] Output: Audio data sent to the server

[1414] Step 2: Convert audio data to text

[1415] The server receives the voice data sent from the terminal.

[1416] The server runs the voice data through a voice recognition engine, specifically a cloud-based voice recognition engine (e.g., a general-purpose voice recognition engine).

[1417] The server converts the voice data into character data (text information) using a voice recognition engine.

[1418] Input: Audio data sent from the device

[1419] Output: Text data (e.g., "Where is the restaurant?")

[1420] Step 3: Parsing character data

[1421] The server sends the generated character data to a natural language processing engine.

[1422] The server uses a natural language processing engine to analyze the text data and interpret the user's intent. Specifically, it uses a "text analysis engine."

[1423] Based on the analysis results, the server identifies the user's requests and questions.

[1424] Input: Text data (e.g., "Where is the restaurant?")

[1425] Output: Interpreted intent (e.g., "asking for restaurant locations")

[1426] Step 4: Generate an appropriate response

[1427] The server generates an appropriate response message based on the user's intent.

[1428] The server retrieves the necessary information from a database or external API. For example, it retrieves the "location of the restaurant" from the facility's floor map information.

[1429] The server uses a "response generation engine" to create a response message based on the information it has obtained.

[1430] Input: Interpreted intent (e.g., "asking for restaurant locations")

[1431] Output: Response message (e.g., "The restaurant is on the third floor, on the right.")

[1432] Step 5: Convert the response message to audio data

[1433] The server sends the generated response message to the speech synthesis engine.

[1434] The server uses a speech synthesis engine to generate voice data from text data. Specifically, it uses a "voice conversion engine."

[1435] The server transmits the generated voice data to the terminal.

[1436] Input: Response message (character data)

[1437] Output: Speech data (e.g., "The restaurant is on the third floor, on the right.")

[1438] Step 6: Outputting audio

[1439] The terminal receives the voice data sent from the server.

[1440] The terminal uses a highly directional speaker to play back the received audio data, allowing users to receive guidance information in a manner that protects their privacy without being seen by other visitors.

[1441] Input: Audio data sent from the server

[1442] Output: Spoken directions provided to the user (e.g., "The restaurant is on the third floor, on your right.")

[1443] Through these steps, the system can provide visitors with appropriate information while functioning efficiently and protecting their privacy.

[1444] (Application example 1)

[1445] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1446] In recent years, there has been a demand for methods to quickly and accurately provide visitors with the information they need in commercial facilities and public areas. However, previous guidance systems have had problems such as complex user interfaces and insufficient privacy protection. Furthermore, providing highly accurate information and guidance in real time has been difficult. In brick-and-mortar stores in particular, the inability to smoothly provide product information and location information has led to issues that can lead to lower customer satisfaction and reduced business efficiency.

[1447] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1448] In this invention, the server includes a device including a superdirectional microphone that receives a user's voice input, means including a voice recognition engine that converts the voice input into text data, means including a natural language processing engine that analyzes the text data and understands the user's intent, response generation means that generates a response message, means including a voice synthesis engine that converts the response into voice data, and means that provides the voice data to the user via a mobile information terminal with a built-in speaker or a mobile machine with a superdirectional speaker. This enables real-time, highly accurate, and privacy-protected guidance via voice input. Furthermore, use in commercial facilities and brick-and-mortar stores can improve customer convenience and streamline store operations.

[1449] 1. A "superdirectional microphone" is a microphone that has the ability to capture only sounds from a specific direction, and is a device that can accurately capture specific sounds by eliminating surrounding noise.

[1450] 2. "Speech Recognition Engine" means a software or hardware component that analyzes and converts voice input into text data.

[1451] 3. "Text data" means text data converted from speech by a speech recognition engine.

[1452] 4. A "natural language processing engine" is an algorithm or system that analyzes input text data and understands the user's intent and meaning.

[1453] 5. "Response generator" means a system or algorithm that creates an appropriate response message based on the user's intent as analyzed by the natural language processing engine.

[1454] 6. "Speech synthesis engine" means a software or hardware component that has the function of converting text data into speech data.

[1455] 7. An "ultra-directional speaker" is a speaker that has the ability to emit sound in a specific direction and is a device designed so that only specific users can hear the sound.

[1456] 8. "Personal digital assistant" means a portable device (e.g., smartphone, tablet) with a built-in speaker for providing audio data to the user.

[1457] 9. "Mobile Machine" means a robot or guide device that contains an ultradirectional speaker and is capable of moving to provide audio data to a user.

[1458] 10. "Mobile communication network" means a network that provides mobile communications for data communication between the entire system and the server.

[1459] 11. "SIM" means a subscriber identity module for use in a mobile communications network.

[1460] DETAILED DESCRIPTION OF THE INVENTION The present invention provides a system for providing visitors with information in a natural conversation while protecting their privacy in commercial facilities and public areas. An embodiment of the present invention will be described in detail below.

[1461] The system includes a device including a superdirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intent, response generation means that generates an appropriate response based on the intent, server means including a voice synthesis engine that converts the response into voice data, and a device including a superdirectional speaker that outputs the voice data to the user, or a mobile information terminal with a built-in speaker. These components also comprise a device including a SIM that communicates with the server means via a mobile communication network.

[1462] Program processing details (natural language explanation)

[1463] User Interface Design

[1464] The smartphones and robots used as terminals are equipped with a user interface that allows voice input and simple touch operations, making it easy for users to use the guidance service.

[1465] Building a voice recognition system

[1466] The device's built-in or super-directional microphone captures the user's voice input, which is then sent to a server, where it converts the speech into text using a cloud-based speech recognition engine such as Google Cloud Speech-to-Text.

[1467] Building a natural language processing system

[1468] The server then sends the converted text data to a natural language processing engine such as Google Language API to analyze the user's intent, allowing it to properly understand the user's question or instruction and generate a response.

[1469] Building a speech synthesis system

[1470] The server sends the generated response message to a speech synthesis engine such as Google Text-to-Speech to generate voice data, which is then sent to the device.

[1471] Audio output

[1472] Voice data synthesized on the server is played from the device's ultra-directional speaker or built-in speaker, providing guidance to the user while protecting their privacy.

[1473] Specific examples

[1474] Example 1: Checking product location

[1475] User: "Excuse me, where is the tea?"

[1476] The device's ultra-directional microphone or built-in microphone captures the sound and transmits the audio data to the server.

[1477] The server converts the voice data into text data and parses it as "Where is the tea?"

[1478] The server identifies the keyword "tea" and retrieves the most appropriate information from a product location database.

[1479] The server generates a response message saying, "Tea is in aisle 3 of the beverage section."

[1480] The server converts this response message into voice data and transmits it to the terminal.

[1481] The device's ultra-directional speaker or built-in speaker announces, "Tea is located in aisle 3 of the beverage section."

[1482] Examples of prompt statements

[1483] Example prompt sentence:

[1484] Please provide a sample audio of a user asking, "Where is the tea?"

[1485] In this way, the present invention allows visitors to easily obtain the information they need, and also improves the efficiency of operations in commercial facilities and public areas, thereby alleviating labor shortages.

[1486] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1487] Step 1:

[1488] This is a situation where a user performs voice input. The user asks a question to a device (e.g., a smartphone or a robot) such as "Where is the tea?" The input is the user's voice, and the output is an analog voice signal.

[1489] Step 2:

[1490] The device's superdirectional microphone or built-in microphone receives the user's voice. The received voice is converted into digital voice data and sent to the server. The input is an analog voice signal, and the output is digital voice data.

[1491] Step 3:

[1492] The server sends the received voice data to a speech recognition engine such as Google Cloud Speech-to-Text, where it is converted into text data. At this stage, the voice data is analyzed and output as text data. The input is digital voice data, and the output is text data.

[1493] Step 4:

[1494] The server sends the text data to a natural language processing engine such as Google Language API to analyze the user's intent. The server extracts the keyword "tea" from the text data "Where is the tea?" and uses this keyword to search the appropriate database. The input is text data, and the output is the search query and search results.

[1495] Step 5:

[1496] The server searches the database based on the analyzed keywords and generates a response message such as "Tea is in aisle 3 of the beverage section." The response generator creates an appropriate answer in text format. The input is the search results, and the output is the text response message.

[1497] Step 6:

[1498] The server sends the generated response message to a speech synthesis engine such as Google Text-to-Speech to generate audio data. The server converts the text data into audio data and sends it to the device. The input is the text response message, and the output is digital audio data.

[1499] Step 7:

[1500] The device's ultra-directional speaker or built-in speaker plays the audio data sent from the server. The user receives a voice prompt saying, "Tea is in aisle 3 in the beverage section." The input is digital audio data, and the output is the audio being played.

[1501] This will enable users to obtain necessary information in real time through voice guidance in commercial facilities and public areas.

[1502] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1503] The present invention combines a system for providing visitor guidance in commercial facilities and public areas through natural conversation while protecting privacy with an emotion engine that recognizes user emotions. Hereinafter, an embodiment of the present invention will be described in detail.

[1504] This system comprises a device including an ultradirectional microphone that receives a user's voice input, server means including a voice recognition engine that converts the voice input into text data, server means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, server means including a voice synthesis engine that converts the response into voice data, a device including an ultradirectional speaker that outputs the voice data to the user, and a device including a SIM that communicates with the server means via a mobile communication network.Furthermore, this invention incorporates an emotion engine that recognizes emotions from the user's voice input.

[1505] Program processing details (natural language explanation)

[1506] User Interface Design

[1507] The devices in this system are designed to be easy for users to operate, with a simple and intuitive UI that accepts touch panels and voice input.

[1508] Building a voice recognition system

[1509] The device's onboard super-directional microphone captures the user's voice input and transmits the voice data to a server, which then converts the voice data into text using a cloud-based speech recognition engine. For example, the server may use a cloud speech recognition service to convert the voice data into text.

[1510] Building a natural language processing system

[1511] The server then sends the converted text data to a natural language processing engine to analyze the user's intent. For example, it uses a natural language processing service to properly interpret the user's questions and instructions. The server then understands the user's request based on the analysis results and generates an appropriate response.

[1512] Building an Emotion Engine

[1513] The server sends the text data generated by speech recognition to an emotion engine, which analyzes the user's emotions. For example, in cooperation with a natural language processing engine, the emotion engine can recognize emotions from the user's tone of voice and vocabulary. For example, if the user is angry or in distress, it can respond accordingly.

[1514] Enhanced response generation

[1515] The server uses the emotion engine to generate a response that matches the user's emotion. For example, if the user is in trouble, the response generator will use a gentle tone. This allows for more personalized responses.

[1516] Building a speech synthesis system

[1517] The server sends the generated response message to a speech synthesis engine to generate voice data. For example, a speech synthesis service is used to convert text data into voice. The generated voice data is then sent from the server to the device.

[1518] Audio output

[1519] The device's built-in ultra-directional speaker plays back the audio data received from the server, and by emitting audio in a specific direction, it provides audio guidance to users while protecting their privacy.

[1520] Specific examples

[1521] Example 1: Library guide

[1522] User: "Excuse me, where is the restaurant?" (in a slightly confused voice)

[1523] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[1524] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[1525] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[1526] The server uses an emotion engine to recognize when the user is in trouble.

[1527] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[1528] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[1529] The device's ultra-directional speaker provides gentle voice guidance, saying, "The restaurant is on the third floor, on the right. We have staff who can guide you, so please ask if you need any help."

[1530] Example 2: Providing event information

[1531] User: "What's happening today?" (excited)

[1532] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[1533] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[1534] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[1535] The server uses an emotion engine to recognize when the user is excited.

[1536] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[1537] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[1538] The device's ultra-directional speaker provides an excited voice prompt: "There's a concert starting at 3 o'clock today. I'm really looking forward to it!"

[1539] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in commercial facilities and public areas. Furthermore, emotion recognition capabilities enable more personalized responses, further improving visitor satisfaction.

[1540] The processing flow will be explained below.

[1541] Step 1:

[1542] A user approaches the device and asks, "Where is the bathroom?"

[1543] Step 2:

[1544] The device's ultra-directional microphone captures the user's voice and stores the voice data in a buffer.

[1545] Step 3:

[1546] The terminal generates an HTTP POST request to send the audio data to the server, and attaches the audio data.

[1547] Step 4:

[1548] The server receives the voice data and sends it to the voice recognition engine, specifically by calling the voice recognition engine's API.

[1549] Step 5:

[1550] The server receives text data (e.g., "Where is the toilet?") from the speech recognition engine.

[1551] Step 6:

[1552] The server sends the text data to the emotion engine for emotion analysis. Specifically, it calls the emotion engine's API to analyze the text data.

[1553] Step 7:

[1554] The server receives the analysis result (e.g., the user is in trouble) from the emotion engine.

[1555] Step 8:

[1556] The server sends the text data to the natural language processing engine. Specifically, it calls the API of the natural language processing engine and instructs it to analyze the text data.

[1557] Step 9:

[1558] The server receives the analysis results (e.g., the keyword "toilet" and a request to obtain location information) from the natural language processing engine.

[1559] Step 10:

[1560] The server accesses the database and searches for location information of facilities related to the keyword "toilet." For example, it retrieves information that the toilet is on the second floor, on the left.

[1561] Step 11:

[1562] Based on the location information acquired by the server and the analysis results of the emotion engine, a response message (e.g., "The toilet is on the second floor, on the left. We have staff who can guide you, so please let us know if you need help") is generated.

[1563] Step 12:

[1564] The server sends the response message to the speech synthesis engine to generate voice data. Specifically, it calls the speech synthesis engine's API to convert the text data into voice data.

[1565] Step 13:

[1566] The server returns the generated audio data to the terminal as an HTTP response.

[1567] Step 14:

[1568] The device receives the voice data and provides audio guidance to the user through an ultra-directional speaker: "The toilet is on the second floor, on the left. We have staff who can guide you, so please let us know if you need any help."

[1569] Example 2

[1570] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1571] In recent years, there has been a demand for systems that can provide visitors with information through natural conversation while protecting their privacy in commercial facilities and public areas. However, conventional systems have the following problems.

[1572] 1. If the recognition accuracy of voice input is low, it is difficult to accurately understand the user's intention.

[1573] 2. It is difficult for general audio output devices to provide directional audio output while protecting privacy.

[1574] 3. The lack of functionality to recognize user emotions and respond appropriately can lead to low visitor satisfaction.

[1575] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1576] In this invention, the server includes means including an audio input device that receives a user's voice input, means including a voice recognition device that converts the voice input into text data, means including a natural language processing device that analyzes the text data and understands the user's intention, means including a response generation device that generates an appropriate answer based on the intention, means including an emotion authentication device that recognizes the user's emotion, means for enhancing the response generation device that generates an appropriate answer based on the emotion, means including a voice synthesis device that converts the answer into voice data, means including an audio output device that outputs the voice data to the user, and communication means for communicating with the processing device via a communication network. This makes it possible to understand the user's intention and emotion through highly accurate recognition of the voice input and natural conversation, and to provide appropriate voice guidance while protecting privacy.

[1577] An "acoustic input device" is a device that collects sound from a specific direction, and includes a directional microphone.

[1578] A "voice recognition device" is a device that converts voice data into text data, and includes systems that perform highly accurate voice recognition using cloud services.

[1579] A "natural language processing device" is a device that analyzes text data and understands the user's intent, and uses a natural language processing engine.

[1580] A "response generation device" is a device that generates an appropriate response based on the user's intentions, and outputs the generated response as text data.

[1581] An "emotion authentication device" is a device that analyzes emotions from a user's voice and text data, and recognizes the user's emotional state.

[1582] The "means for strengthening the response generation device" is a means for enabling the response generation device to provide a response according to the user's emotions based on the results of the emotion authentication device.

[1583] A "speech synthesizer" is a device that converts character data into voice data, and includes a system that generates synthetic voice.

[1584] An "audio output device" is a device that outputs sound in a specific direction, and includes a directional speaker.

[1585] "Communication means" refers to a means for data communication between a terminal and a server, and includes a SIM card or module for communication via a mobile communication network.

[1586] The present invention combines a system for providing visitor guidance in commercial facilities and public areas through natural conversation while protecting privacy with an emotion engine that recognizes user emotions. Hereinafter, an embodiment of the present invention will be described in detail.

[1587] The system includes the following hardware and software: a device including an ultradirectional microphone for receiving a user's voice input, server means including a voice recognition engine for converting the voice input into text data, server means including a natural language processing engine for analyzing the text data and understanding the user's intention, response generation means for generating an appropriate response based on the user's intention, an emotion authentication device for recognizing the user's emotion, means for enhancing the response generation device for generating an appropriate response based on the emotion, server means including a voice synthesis engine for converting the response into voice data, a device including an ultradirectional speaker for outputting the voice data to the user, and a device including a SIM for communicating with the server means via a mobile communication network.

[1588] First, the user speaks a question or instruction to the system. For example, the user might say, "Excuse me, where is the restaurant?" This voice input is received by the device's superdirectional microphone, and the device transmits the captured voice data via the SIM card and the mobile communication network to a server. The server converts the voice data into text data using a cloud-based speech recognition engine (e.g., Google Cloud Speech-to-Text), and sends this text data to a natural language processing engine (e.g., Google Natural Language API) to analyze the user's intent.

[1589] Next, the server sends the recognized text data and voice data to an emotion recognition device, which analyzes the user's emotions. This emotion recognition device (for example, Azure Cognitive Services' Emotion Analysis API) is used to recognize emotions from the user's tone of voice and phrasing. For example, if the user is distressed or excited, the emotion is identified. The server then activates a response generation device to generate an appropriate response based on the results of the natural language processing engine and emotion recognition device. The response generation device retrieves information from a database and creates a response message, such as "The restaurant is on the third floor, on the right." The response generation device is also enhanced to generate responses in a tone that corresponds to the emotion.

[1590] After the response message is generated, the server sends it to a speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) to generate voice data. The generated voice data is sent from the server to the device and played back by the device's super-directional speaker. This speaker emits voice in a specific direction, providing voice guidance to the user while protecting their privacy.

[1591] Example 1: Museum guide

[1592] User: "Excuse me, where is the restaurant?" (in a slightly confused voice)

[1593] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[1594] The server receives the voice data and generates text data such as "Where is the restaurant?" through a voice recognition engine.

[1595] The server analyzes the data using a natural language processing engine and detects the keyword "restaurant."

[1596] The server uses an emotion authentication device to recognize that the user is in trouble.

[1597] The server retrieves the optimal route information from the database and generates a response message saying, "The restaurant is on the third floor, on the right."

[1598] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[1599] The device's ultra-directional speaker provides gentle voice guidance, saying, "The restaurant is on the third floor, on the right. We have staff who can guide you, so please ask if you need any help."

[1600] Example 2: Providing event information

[1601] User: "What's happening today?" (excited)

[1602] The device's ultra-directional microphone captures the sound and transmits the audio data to the server.

[1603] The server receives the voice data and generates text data such as "What are the events today?" through a voice recognition engine.

[1604] The server analyzes the data using a natural language processing engine and detects the keyword "today's events."

[1605] The server uses an emotion authentication device to recognize that the user is excited.

[1606] The server retrieves the event information from the database and generates a response message saying, "There will be a concert starting at 3 o'clock today."

[1607] The server uses a speech synthesis engine to convert the response message into voice data and transmits it to the terminal.

[1608] The device's ultra-directional speaker provides an excited voice prompt: "There's a concert starting at 3 o'clock today. I'm really looking forward to it!"

[1609] In this way, the present invention allows visitors to easily obtain the information they need, and also alleviates labor shortages in commercial facilities and public areas. Furthermore, emotion recognition capabilities enable more personalized responses, further improving visitor satisfaction.

[1610] Example prompt sentence:

[1611] "How do we recognise when a visitor is in distress and design a system that can respond appropriately?"

[1612] "Please tell me an example of how emotion recognition is used in a guidance system for a commercial facility."

[1613] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1614] Step 1:

[1615] Accepting user voice input

[1616] The user speaks a question or command, for example, "Excuse me, where is the restaurant?"

[1617] Input: User's voice.

[1618] The device's ultra-directional microphone captures the audio.

[1619] Output: Audio data is generated on the device.

[1620] Step 2:

[1621] Sending audio data

[1622] The device transmits the received voice data to a server via the SIM and the mobile communication network.

[1623] Input: Audio data captured by a superdirectional microphone.

[1624] Output: The audio data sent to the server.

[1625] Step 3:

[1626] Speech Recognition Processing

[1627] The server inputs the received voice data into a voice recognition engine (cloud-based).

[1628] Input: The audio data sent to the server.

[1629] The speech recognition engine converts the voice data into text data, for example, generating the text data "Where is the restaurant?"

[1630] Output: Character data generated within the server.

[1631] Step 4:

[1632] Natural Language Processing

[1633] The server sends the generated character data to a natural language processing engine.

[1634] Input: The text data generated by the speech recognition engine.

[1635] A natural language processing engine analyzes text data to understand the user's intent, for example by extracting the keyword "restaurant."

[1636] Output: Data parsed for user intent.

[1637] Step 5:

[1638] Emotion Recognition Processing

[1639] The server inputs the character data and voice data into the emotion authentication device.

[1640] Input: The text data generated by the speech recognition engine and the original audio data.

[1641] Emotion recognition devices analyze emotions from voice tone and vocabulary, for example, recognizing when a user is distressed.

[1642] Output: Sentiment-analyzed data.

[1643] Step 6:

[1644] Proper response generation

[1645] The server generates an appropriate answer using a response generation means based on the results of the natural language processing engine and the emotion authentication device.

[1646] Input: User intent analyzed by natural language processing and user emotion analyzed by an emotion recognizer.

[1647] The response generation means retrieves information from the database and generates a response message such as "The restaurant is on the third floor, on the right."

[1648] Output: The generated response message.

[1649] Step 7:

[1650] Text-to-speech response messages

[1651] The server sends a response message to the speech synthesis engine.

[1652] Input: The response message generated by the response generator.

[1653] The speech synthesis engine converts text data into speech data, generating a voice file that says, for example, "The restaurant is on the third floor, on your right."

[1654] Output: The generated audio data.

[1655] Step 8:

[1656] Audio data output

[1657] The server transmits the generated voice data to the terminal.

[1658] Input: The speech data generated by the speech synthesis engine.

[1659] The device's ultra-directional speaker plays back the audio data.

[1660] As a specific example, the speaker provides voice guidance in a gentle tone, saying, "The restaurant is on the third floor on the right. We have staff who can guide you, so please ask if you need any help."

[1661] Output: A voice prompt of the processing results that the user hears.

[1662] (Application example 2)

[1663] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1664] In recent years, there has been a demand for automated visitor guidance systems in commercial facilities, public areas, and brick-and-mortar stores. However, existing guidance systems have difficulty protecting visitor privacy and are unable to provide personalized responses that take visitors' emotions into account. This makes it difficult for visitors to quickly and accurately obtain information that meets their needs, resulting in a decline in visitor experience and satisfaction. Given this background, there is a need for the development of a new guidance system that protects visitors' privacy while providing personalized responses that take their emotions into account.

[1665] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a superdirectional microphone that receives a user's voice input, means including a voice recognition engine that converts the voice input into text data, means including a natural language processing engine that analyzes the text data and understands the user's intention, response generation means that generates an appropriate response based on the intention, means including a voice synthesis engine that converts the response into voice data, means including a superdirectional speaker that outputs the voice data to the user, means including a SIM that communicates with the server means via a mobile communication network, server means that includes an emotion engine that recognizes emotions from the user's voice input, and means for adjusting the response content based on the emotion recognized by the emotion engine. This enables personalized guidance that corresponds to the emotions of the situation while protecting the visitor's privacy.

[1666] A "super-directional microphone" is a microphone that has the ability to pick up sound only from a specific direction, and is a device that reduces surrounding noise and records the user's voice clearly.

[1667] A "voice recognition engine" is a software system that analyzes voice data and converts it into text data, and is the basic technology for understanding the user's voice.

[1668] A "natural language processing engine" is a software system that analyzes text data to understand the user's intent, and interprets the user's request based on the text data generated by a voice recognition engine.

[1669] The "response generation means" is a system that generates appropriate answers based on the user's intent analyzed by the natural language processing engine, and is responsible for providing responses that meet the user's requests.

[1670] A "speech synthesis engine" is a software system that converts text data into speech data, and is a technology for outputting the generated response as speech.

[1671] An "ultra-directional speaker" is a speaker that has the ability to emit sound in a specific direction, and is a device that provides audio to users while protecting their privacy.

[1672] An "emotion engine" is a software system that analyzes voice and text data to recognize a user's emotions, and understands emotions from the user's tone of voice and choice of words.

[1673] A "SIM" is a module for connecting a device to a cloud server means via a mobile communication network, and is hardware that enables communication between the server and the device.

[1674] "Cloud-based" refers to a technology form that utilizes remote servers on the Internet to process and store data, and does not require the device itself to have high computing power or large amounts of memory.

[1675] This invention is a system for providing guidance to visitors in commercial facilities, public areas, and brick-and-mortar stores, and aims to provide guidance in a natural, conversational style while preserving user privacy. It also aims to improve visitor satisfaction by recognizing the user's emotions and providing personalized responses based on those emotions.

[1676] The system includes the following elements:

[1677] 1. Ultra-directional microphone: A device that has the ability to pick up sound only from a specific direction, clearly recording the user's voice.

[1678] 2. A speech recognition engine included in the server means: A software system that converts a user's voice input into text data.

[1679] 3. Natural language processing engine included in the server means: A software system for analyzing character data and understanding the user's intent.

[1680] 4. Response generation means: A system that generates appropriate responses based on the user's intent.

[1681] 5. A speech synthesis engine included in the server means: A software system that converts the generated answers into speech data.

[1682] 6. Ultra-directional speaker: A speaker that emits sound in a specific direction, a device that provides audio guidance while maintaining privacy.

[1683] 7. Device including SIM: A module that performs communication between the server means and the device via a mobile communication network.

[1684] 8. Emotion engine: A software system that analyzes voice and text data to recognize the user's emotions.

[1685] Let us explain the system's operation through an example:

[1686] Example 1: Product location guidance

[1687] The user asks in a confused voice, "Where is the chocolate?" The device's superdirectional microphone receives the voice and sends the voice data to the server. The server's speech recognition engine converts the voice data into text data, and the natural language processing engine detects the keyword "chocolate" and analyzes the user's intention. Next, the emotion engine recognizes that the user is confused, and the response generation means generates a gentle response saying, "The chocolate is on the first floor, on the left. Shall I show you?" The response generated by the speech synthesis engine is converted into voice data, and the superdirectional speaker provides audio guidance to the user.

[1688] Example 2: Providing sales information

[1689] The user asks in an excited voice, "Are there any sales going on right now?" The device's superdirectional microphone receives the voice and sends the voice data to the server. The server's speech recognition engine converts the voice data into text data, and the natural language processing engine detects the keyword "sale" and analyzes the user's intention. Next, the emotion engine recognizes that the user is excited, and the response generation means generates a response in an excited tone: "Yes, there's a 20% off sale in the clothing department on the first floor!" The speech synthesis engine converts the generated response into voice data, and the superdirectional speaker provides audio guidance to the user.

[1690] As these examples show, this system can improve visitor satisfaction by providing personalized guidance based on emotions while protecting visitor privacy.

[1691] Prompt Sentence Examples

[1692] Prompt when user uses smart glasses to ask for product location:

[1693] "A user asked in a confused voice, 'Where is the chocolate?'

[1694] The voice recognition engine converted the text data. Next, the natural language processing engine analyzed the product location, and the emotion recognition engine recognized the user's confused emotion. The response generation engine generated a response in a gentle tone, and the speech synthesis engine converted it into voice data. The super-directional speaker said in a gentle tone, "The chocolate is on the first floor, on the left. Shall I show you?"

[1695] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1696] Step 1: Receiving Audio Input

[1697] This is the role of the device. The device's built-in superdirectional microphone receives the user's voice input and records it as voice data. This input includes what the user says. For example, the user may ask, "Where is the chocolate?"

[1698] Step 2: Sending audio data

[1699] This is the role of the device. It transmits the received voice data to a cloud-based server via a mobile communication network. The input in this step is the voice data recorded in step 1, and the output is the voice data received by the server.

[1700] Step 3: Voice Recognition

[1701] This is the role of the server. A cloud-based speech recognition engine analyzes the voice data received by the server and converts the voice data into text data. The input is voice data and the output is text data. Specifically, the speech recognition engine analyzes the utterance "Where is the chocolate?" and converts it into text data "Where is the chocolate?"

[1702] Step 4: Natural Language Processing

[1703] This is the role of the server. The natural language processing engine analyzes the converted text data and understands the user's intent. The input is text data, and the output is the analyzed intent data. Specifically, the natural language processing engine detects the keyword "chocolate" and understands that the user is asking where the chocolate is.

[1704] Step 5: Emotion Recognition

[1705] This is the role of the server. The emotion engine analyzes the text data obtained by the natural language processing engine and recognizes the user's emotions. The input is text data, and the output is recognized emotional data. Specifically, the emotion engine recognizes that the user is in a difficult situation.

[1706] Step 6: Response Generation

[1707] This is the role of the server. Based on the recognized emotion data and analyzed intent data, the response generation means generates an appropriate answer. The input is the analyzed intent data and emotion data, and the output is a response message. Specifically, the response generation means generates a gentle response message saying, "The chocolate is on the first floor on the left. Shall I show you?"

[1708] Step 7: Text-to-Speech

[1709] This is the role of the server. The generated response message is converted into voice data by the speech synthesis engine. The input is the response message, and the output is voice data. Specifically, the speech synthesis engine generates voice data saying, "The chocolate is on the first floor on the left. Shall I show you?"

[1710] Step 8: Audio Output

[1711] This is the role of the terminal. The generated voice data is transmitted to the user by the terminal's superdirectional speaker. The input is voice data, and the output is the user receiving the voice. In concrete terms, the terminal's superdirectional speaker provides voice guidance to the user, saying, "The chocolate is on the first floor, on the left. Shall I show you?"

[1712] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1713] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1714] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1715] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1716] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1717] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1718] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1719] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1720] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1721] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1722] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1723] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1724] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1725] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1726] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1727] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1728] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1729] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1730] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1731] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1732] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1733] The following is further disclosed regarding the above embodiment.

[1734] (Claim 1)

[1735] a device including an ultradirectional microphone that receives a user's voice input;

[1736] a server means including a voice recognition engine for converting the voice input into character data;

[1737] a server means including a natural language processing engine that analyzes the character data and understands the user's intent;

[1738] a response generating means for generating an appropriate response based on the intention;

[1739] a server means including a voice synthesis engine for converting the answer into voice data;

[1740] a device including an ultra-directional speaker that outputs the audio data to a user;

[1741] A system including a device including a SIM that communicates with said server means over a mobile communications network.

[1742] (Claim 2)

[1743] The system of claim 1 , wherein the speech recognition engine is cloud-based.

[1744] (Claim 3)

[1745] 10. The system of claim 1, wherein the natural language processing engine is cloud-based and analyzes user intent in real time.

[1746] "Example 1"

[1747] (Claim 1)

[1748] a terminal including a highly directional microphone for receiving a user's voice input;

[1749] an information processing device including a voice recognition engine that converts the voice input into character data;

[1750] an information processing device including a natural language processing engine that analyzes the character data and understands the user's intent;

[1751] an information processing device including a response generation means for generating an appropriate response based on the intention;

[1752] an information processing device including a voice synthesis engine that converts the answer into voice data;

[1753] a terminal including a highly directional speaker that outputs the audio data to a user;

[1754] A system including a communication module that communicates with the information processing device via a wireless communication network.

[1755] (Claim 2)

[1756] The system of claim 1 , wherein the speech recognition engine is cloud-based.

[1757] (Claim 3)

[1758] 10. The system of claim 1, wherein the natural language processing engine is cloud-based and analyzes user intent in real time.

[1759] "Application Example 1"

[1760] (Claim 1)

[1761] a device including an ultradirectional microphone that receives a user's voice input;

[1762] a server means including a voice recognition engine for converting the voice input into character data;

[1763] a server means including a natural language processing engine that analyzes the character data and understands the user's intent;

[1764] a response generating means for generating an appropriate response based on the intention;

[1765] a server means including a voice synthesis engine for converting the answer into voice data;

[1766] a device including an ultra-directional speaker that outputs the audio data to a user;

[1767] a device including a SIM that communicates with the server means via a mobile communication network;

[1768] A system including a personal digital assistant or a mobile machine including a built-in speaker that provides a response message generated by said server means to a user, said system including a super-directional speaker.

[1769] (Claim 2)

[1770] The system of claim 1 , wherein the speech recognition engine is cloud-based.

[1771] (Claim 3)

[1772] 10. The system of claim 1, wherein the natural language processing engine is cloud-based and analyzes user intent in real time.

[1773] "Example 2: Combining Emotion Engines"

[1774] (Claim 1)

[1775] an apparatus including an audio input device for receiving a user's voice input;

[1776] a processing device including a voice recognition device that converts the voice input into character data;

[1777] a processing device including a natural language processing device that analyzes the character data and understands the user's intent;

[1778] a processing device including a response generation device that generates an appropriate response based on the intention;

[1779] a processing device including an emotion authentication device that recognizes the emotion of a user;

[1780] means for enhancing a response generator for generating appropriate responses based on said emotions;

[1781] a processing unit including a voice synthesizer that converts the response into voice data;

[1782] a device including an audio output device that outputs the audio data to a user;

[1783] A system including a device including a communication means for communicating with said processing device over a communications network.

[1784] (Claim 2)

[1785] The syste...

Claims

1. a device including an ultradirectional microphone that receives a user's voice input; a server means including a voice recognition engine for converting the voice input into character data; a server means including a natural language processing engine that analyzes the character data and understands the user's intent; a response generating means for generating an appropriate response based on the intention; a server means including a voice synthesis engine for converting the answer into voice data; a device including an ultra-directional speaker that outputs the audio data to a user; A system including a device including a SIM that communicates with said server means over a mobile communications network.

2. The system of claim 1 , wherein the speech recognition engine is cloud-based.

3. The system of claim 1 , wherein the natural language processing engine is cloud-based and analyzes user intent in real time.

Images