system

Abstract

Provide a system. A speech recognition unit converts a user's speech into text data; natural language processing means for analyzing user intent from text data; an operation guide generating means for generating an operation guide on a screen based on a user's intention; a user interface display means for displaying an operation guide on the screen in an overlay display; A user operation tracking means for tracking user operation behavior in real time; a correction guide generating means for generating a correction guide when a user's operation is not performed correctly; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Modern smartphones are multifunctional and offer numerous applications, but understanding and efficiently using all of these operations can be difficult for some users. For users who are unfamiliar with technology or the elderly, opening the desired application and performing specific operations can be a daunting task. For these users, there is a need for a system that uses voice input to guide operations and ensure the desired operations are carried out reliably. [Means for solving the problem]

[0005] The present invention provides a system that converts a user's speech into text data using a voice recognition means, analyzes the user's intention from the text data using a natural language processing means, generates operation procedures based on the analyzed intention using an operation guide generation means, and displays the operation guide in an overlay display on the screen using a user interface display means, tracks the user's operation behavior in real time using a user operation tracking means, and generates correction guides as needed using a correction guide generation means, thereby enabling the user to reliably achieve the target operation.

[0006] A "voice recognition means" is a device or software function that captures a user's voice and converts it into text data.

[0007] "Natural language processing means" is a technology that analyzes converted text data and understands the user's intentions and meaning from it.

[0008] The "operation guide generation means" is a function that generates operation procedures that the user should follow based on the results of natural language processing.

[0009] The "user interface display means" is a function that displays the generated operation guide on the user's screen in an overlay format.

[0010] The "user operation tracking means" is a function that monitors the operations that the user is performing on the screen in real time and tracks their progress.

[0011] The "correction guide generating means" is a function that generates a guide that instructs the user on how to perform an appropriate correction operation when the user's operation is not performed correctly. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] The present invention provides a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit. The program processing of this system will be explained below in natural language.

[0034] 1. Voice Recognition Method

[0035] A user speaks into their smartphone, "I want to watch cat videos on YouTube®." The device uses a built-in microphone to capture this speech and receives it as voice data. The voice recognition module then converts the voice data into text data. This process passes the user's instructions in text form to the system.

[0036] 2. Natural Language Processing Methods

[0037] The device then sends this text data to a natural language processing engine, which analyzes the text and extracts the user's intent, in this case "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[0038] 3. Operation guide generation method

[0039] Based on the analyzed intent, the device will generate a guide with relevant operation steps. For example, the first step is to open the YouTube app, so the device will locate the YouTube app icon on the home screen. Next, it will find the search bar and provide specific instructions for typing "cat."

[0040] 4. User Interface Display Method

[0041] The procedure generated by the operation guide generating means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this display continues until the user completes the tap operation.

[0042] 5. User Action Tracking Methods

[0043] The device tracks the user's actions in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar and monitoring the user's actions when typing "cat."

[0044] 6. Correction guide generation method

[0045] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly.

[0046] Specific examples

[0047] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[0048] 1. The user speaks.

[0049] 2. The device converts the voice into text and analyzes it using an NLP engine.

[0050] 3. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video).

[0051] 4. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[0052] 5. The user follows the instructions and taps the YouTube icon.

[0053] 6. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[0054] 7. User types "cat" and performs a search.

[0055] 8. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[0056] 9. User taps on the video to start playing.

[0057] In this way, the user can smoothly perform the desired operation by simply speaking. The present invention is a system that is particularly useful for users who are not familiar with technology and elderly people.

[0058] The processing flow will be explained below.

[0059] Step 1:

[0060] The user speaks into their smartphone, "I want to watch cat videos on YouTube."

[0061] Step 2:

[0062] The terminal uses a built-in microphone to capture the user's voice and receives it as voice data.

[0063] Step 3:

[0064] The terminal passes the received voice data to a voice recognition module and converts it into text data.

[0065] Step 4:

[0066] The device sends the converted text data to a natural language processing engine.

[0067] Step 5:

[0068] The device uses a natural language processing engine to analyze the text data and extract the user's intent, such as "open YouTube," "type 'cat' into the search bar," or "select a video from the search results."

[0069] Step 6:

[0070] The device prepares an operation procedure to generate an operation guide based on the analyzed intent. For example, first, it collects location information to identify the YouTube app icon.

[0071] Step 7:

[0072] The device will generate an operation guide and display a cursor pointing to the YouTube app icon and a guide saying "Press here" on the screen.

[0073] Step 8:

[0074] The user taps the YouTube icon.

[0075] Step 9:

[0076] The device detects the user's operation and confirms that the YouTube app has been launched correctly.

[0077] Step 10:

[0078] The device will identify the search bar within the YouTube app and display a cursor pointing to it with the prompt "Type 'cat' here."

[0079] Step 11:

[0080] A user types "cat" into the search bar and presses Enter.

[0081] Step 12:

[0082] The device detects whether the user's input was correct and verifies that the search results are displayed.

[0083] Step 13:

[0084] The device will locate the first video in the search results and display a cursor pointing to the video and a prompt saying "Play this."

[0085] Step 14:

[0086] The user taps on the first video and it begins playing.

[0087] Step 15:

[0088] The terminal confirms the completion of the user operation and ends the guide display.

[0089] Through the above steps, the system allows users to easily and reliably perform desired operations on their smartphones using voice utterances.

[0090] Example 1

[0091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0092] Conventional voice-operated systems have difficulty accurately recognizing user utterances and analyzing their intentions, making it difficult to effectively support operations, especially for users who are unfamiliar with technology or the elderly. Furthermore, there is a lack of means to provide appropriate correction guidance in the event of an incorrect operation, which means it takes time to correct the incorrect operation, reducing usability.

[0093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0094] In this invention, the server includes a voice recognition means, a natural language processing means, and an operation guide generation means, which make it possible to capture a user's voice instructions, convert them into text data, analyze the text data to extract the user's intention, and generate and display operation instructions based on the analyzed intention.

[0095] "Speech recognition means" is a technology that captures a user's speech as voice data and converts it into text data.

[0096] "Natural language processing means" is a technology that analyzes and extracts user intent from text data.

[0097] The "operation guide generation means" is a technology that generates an operation procedure to be presented to the user based on the analyzed intention.

[0098] The "user interface display means" is a technique for visually displaying the generated operation guide to the user.

[0099] "User operation tracking means" is a technology that monitors and tracks the operations performed by a user in real time.

[0100] The "correction guide generating means" is a technology that detects an incorrect operation made by a user and generates a guide that instructs the user on the correct operation.

[0101] "Text data" refers to character data converted by a speech recognition means.

[0102] "Intent analysis" refers to the process of interpreting user intent from text data using natural language processing tools.

[0103] "Operation instructions" refer to instructions that present the user with specific operations that should be performed based on the user's intentions.

[0104] An "incorrect operation" is when a user performs an operation that differs from the operation instructions presented by the system.

[0105] "Correction information" refers to specific instruction information for correcting an incorrect operation and performing the correct operation.

[0106] The present invention relates to a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit, and the system aims to capture a user's voice instructions, convert them into text data, analyze their intentions, and present the user with the necessary operation procedures.

[0107] 1. Voice Recognition Method

[0108] A user speaks into a smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and record it as audio data. The device's speech recognition module (e.g., Google® Cloud Speech-to-Text API) then converts the audio data into text data. This process passes the user's instructions in text form to the system.

[0109] 2. Natural Language Processing Methods

[0110] The text data obtained by the speech recognition module is sent to a natural language processing engine (e.g., OpenAI® GPT-3®). The NLP engine analyzes the text and extracts the user's intent. In this case, the intents identified are "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[0111] 3. Operation guide generation method

[0112] Based on the analyzed user intent, the operation guide generator generates relevant operation steps. For example, the first step is to open the YouTube app, so the device identifies the location of the YouTube app icon on the home screen. Next, it provides specific steps for finding the search bar and entering "cat."

[0113] 4. User Interface Display Method

[0114] The procedure generated by the operation guide generating means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this continues until the user completes the tap operation.

[0115] 5. User Action Tracking Methods

[0116] The device tracks user behavior in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar, and even monitoring whether the user is typing "cat" correctly.

[0117] 6. Correction guide generation method

[0118] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly.

[0119] Specific examples

[0120] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[0121] 1. The user says, "I want to watch cat videos on YouTube."

[0122] 2. The device converts the voice into text and analyzes it using an NLP engine.

[0123] 3. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video).

[0124] 4. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[0125] 5. The user follows the instructions and taps the YouTube icon.

[0126] 6. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[0127] 7. User types "cat" and performs a search.

[0128] 8. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[0129] 9. User taps on the video to start playing.

[0130] In this way, the user can smoothly perform the desired operation using only voice utterances. This system is particularly useful for users who are not familiar with technology or the elderly. As an example prompt sentence, the following sentence is input to the generative AI model:

[0131] "If a user says they want to watch cat videos on YouTube, explain how the system generates, displays, and supports the user with navigation."

[0132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0133] Step 1:

[0134] A user says, "I want to watch cat videos on YouTube."

[0135] Input: User utterance.

[0136] Output: The captured audio data.

[0137] How it works: A user verbally types into their smartphone, "I want to watch cat videos on YouTube."

[0138] Step 2:

[0139] The device captures the user's speech using a built-in microphone.

[0140] Input: User utterance.

[0141] Output: Audio data.

[0142] How it works: The smartphone's built-in microphone records the user's speech as audio data.

[0143] Step 3:

[0144] The device's voice recognition module converts the voice data into text data.

[0145] Input: Audio data.

[0146] Output: Character data.

[0147] What happens: A speech recognition module (e.g., Google Cloud Speech-to-Text API) analyzes the captured audio data and converts it into corresponding text data ("I want to watch cat videos on YouTube").

[0148] Step 4:

[0149] The device sends the text data to a natural language processing engine.

[0150] Input: Character data.

[0151] Output: User intent.

[0152] How it works: The converted text data is sent to a natural language processing engine (e.g., OpenAI GPT-3) to prepare it for analyzing the user's intent.

[0153] Step 5:

[0154] A natural language processing engine analyzes the text data and extracts the user's intent.

[0155] Input: Character data.

[0156] Output: User intent (e.g., "Open YouTube," "Type 'cat' into the search bar," "Select a video from the search results").

[0157] How it works: A natural language processing engine parses the text data it receives and identifies the user's request ("open YouTube," "type 'cat' in the search bar," "select a video from the search results").

[0158] Step 6:

[0159] The terminal generates operating procedures based on the analyzed intention.

[0160] Input: User intent.

[0161] Output: Operation instructions.

[0162] Action: The device constructs the necessary steps based on the user's intent (e.g., the steps to open the YouTube app).

[0163] Step 7:

[0164] Your device will locate the YouTube app icon on your home screen.

[0165] Input: Intention-based operating instructions.

[0166] Output: YouTube icon position.

[0167] What it does: Your device uses a digital map on your home screen to locate the YouTube icon.

[0168] Step 8:

[0169] The device overlays a guide on the user interface to guide you through the operation process.

[0170] Input: The location of the identified icon.

[0171] Output: On-screen cursor and guide message.

[0172] Behavior: The device displays a cursor over the YouTube icon and the message "Press here" on the screen until the user taps it.

[0173] Step 9:

[0174] The device tracks the user's operating behavior in real time.

[0175] Input: User taps.

[0176] Output: Confirmation that the operation was successful.

[0177] What it does: The device monitors and checks if the user has successfully launched the YouTube app.

[0178] Step 10:

[0179] Your device will locate the search bar and then guide you through typing "cat."

[0180] Input: Confirmation information for operation completion.

[0181] Output: Search bar position and guide message.

[0182] Behavior: The device displays a cursor pointing at the search bar and the message "Type 'cat' here" until the user types something.

[0183] Step 11:

[0184] A user types in "cat" and performs a search.

[0185] Input: Guide message.

[0186] Output: The search term entered.

[0187] What happens: A user types "cat" into the search bar and presses the search button.

[0188] Step 12:

[0189] The device will identify the appropriate video from the search results and display a guide pointing to it.

[0190] Input: The search term entered.

[0191] Output: Video position and guide message.

[0192] What it does: The device displays a cursor pointing to the appropriate video in the search results and the message "Play this" until the user makes a selection.

[0193] Step 13:

[0194] The user taps on the video to begin playing it.

[0195] Input: Guide message.

[0196] Output: Video begins playing.

[0197] How it works: The user is guided to tap on a video and it begins playing.

[0198] Through this series of steps, the user can smoothly perform the desired operation using only voice utterances.

[0199] (Application example 1)

[0200] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0201] With conventional store guidance systems, it was difficult for staff to grasp their physical location within the store and provide appropriate guidance according to customer requests. Furthermore, in order for staff to accurately guide customers in real time, they had to process a large amount of information in a short amount of time, which placed a heavy burden on staff. Furthermore, it was difficult to provide prompt corrective instructions when a customer was in the wrong place or made a mistake.

[0202] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0203] In this invention, the server includes a voice recognition means, a natural language processing means, an operation guide generation means, a user interface display means, a user operation tracking means, a correction guide generation means, a means for tracking position information in physical space, and a means for overlaying and displaying information in the user's field of view, thereby enabling store guidance in response to customer requests in real time.

[0204] A "voice recognition means" is a device or software module that converts voice data into text data.

[0205] A "natural language processing means" is a device or software module that analyzes user intent from text data.

[0206] The "operation guide generation means" is a device or software module that generates an operation procedure based on the analyzed intention.

[0207] The "user interface display means" is a device or software module that displays operation guides on a screen or display.

[0208] A "user operation tracking means" is a device or software module that tracks a user's operation behavior in real time.

[0209] The "correction guide generating means" is a device or software module that generates correction instructions when the user performs an incorrect operation.

[0210] A "physical location tracking device" is a device or software module that tracks a user's physical location using GPS or other location technology.

[0211] "Means for overlaying and displaying information in the user's field of view" refers to a device or software module that displays information by overlaying it in the user's field of view using smart glasses, a head-mounted display, or the like.

[0212] The present invention relates to a system that provides real-time store guidance in response to customer requests. The system combines the functions of voice recognition, natural language processing, operation guidance, user interface display, user operation tracking, correction guidance, location information tracking, and overlay display.

[0213] This system is configured as follows:

[0214] Voice recognition means

[0215] The server converts the voice received by the staff through devices such as smart glasses into text data using voice recognition software (e.g., Google Speech-to-Text API).

[0216] Natural language processing tools

[0217] The text data is analyzed by the server's natural language processing engine (e.g., Google Cloud Natural Language API or IBM Watson®) to extract the user's intent. For example, if a customer asks, "Where is the wine section?", the intent is extracted as "Find the location of the wine section."

[0218] Operation guide generation means

[0219] Based on the analyzed intent, the server generates a guide for the relevant operating procedures. Specifically, it refers to the store's map data to calculate the optimal route and generates data for displaying the guide.

[0220] User interface display means

[0221] The procedures generated by the operation guide generation means are overlaid on the display of the smart glasses worn by the staff member, using a user interface framework such as Unity.

[0222] User action tracking

[0223] The server tracks the physical location of staff in real time using GPS sensors and other location technologies, ensuring they are heading in the right direction.

[0224] Correction guide generation method

[0225] If the user is heading in the wrong direction or making a mistake, the server will instantly generate corrective instructions and display them on the smart glasses' display, such as "Turn left at the next corner."

[0226] Specific examples

[0227] As a concrete example, if a customer asks, "Where is the wine section?", the sequence of events is as follows:

[0228] 1. Staff wear smart glasses and capture customer questions using voice recognition.

[0229] 2. The server converts the speech to text.

[0230] 3. The NLP engine analyzes the question and extracts the intent: "Find a wine section."

[0231] 4. The store map data is referenced to generate a route guide, and the smart glasses display shows "Turn right from here."

[0232] 5. The server uses GPS data to track the staff's location and ensure they are heading in the right direction.

[0233] 6. If you are heading to the wrong location, the server will display a corrective instruction such as "Turn left at the next corner."

[0234] Prompt Sentence Examples

[0235] Enter the following prompts into the AI ​​model to enable it to generate the appropriate program:

[0236] Based on the user's voice question, your goal is to generate a program that uses an NLP engine to analyze the intent and provide appropriate product information and floor guides in real time. For example, if a customer asks, "Where is the wine section?", please explain the following steps:

[0237] 1. Voice Recognition

[0238] 2. NLP analysis

[0239] 3. Guide Generation

[0240] 4. UI display

[0241] 5. Operation Tracking

[0242] 6. Guide correction

[0243] Please also include specific examples of the hardware and software you will use.

[0244] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0245] Step 1:

[0246] Audio Capture and Conversion

[0247] Input: The user speaks a voice question into the smart glasses.

[0248] Specific operation: The staff activates the voice input function of the smart glasses and captures the question. The smart glasses use the built-in microphone to capture the voice data.

[0249] Data processing and calculation: A speech recognition module (e.g., Google Speech-to-Text API) converts the voice data into text data.

[0250] Output: Text data (e.g., "Where is the wine store?")

[0251] Step 2:

[0252] Text analysis and intent extraction

[0253] Input: Text data sent from smart glasses

[0254] Specific operation: The server receives the text data and sends it to a natural language processing engine (e.g., Google Cloud Natural Language API).

[0255] Data processing and calculation: A natural language processing engine analyzes text data and extracts user intent.

[0256] Output: Intent data (e.g., "Find a wine section")

[0257] Step 3:

[0258] Operation guide generation

[0259] Input: Intent data

[0260] Specific operation: Based on the intent data, the server references the location information within the store and calculates the appropriate guidance route.

[0261] Data processing and calculation: Taking information from the store's map database and applying algorithms to calculate the optimal route.

[0262] Output: Guide data (e.g. "Turn right from here")

[0263] Step 4:

[0264] User Interface Display

[0265] Input: Guide data

[0266] Specific operation: The server sends guide data to the smart glasses, which overlay it on the user's field of view. The smart glasses then display arrows and guide messages on the display.

[0267] Data processing and computation: Converting guide data into visual information using a user interface framework (e.g., Unity).

[0268] Output: Visual data (e.g., "Please turn right")

[0269] Step 5:

[0270] User Action Tracking

[0271] Input: Location data from smart glasses

[0272] What it does: The server tracks the physical location of staff in real time using GPS sensors and other location technologies.

[0273] Data processing and calculation: Analyze the acquired location data and check whether the staff is following the correct route.

[0274] Output: Tracking data (e.g. flags indicating whether you are moving in the right direction)

[0275] Step 6:

[0276] Correction guide generation

[0277] Input: Tracking data

[0278] Specific behavior: The server uses tracking data to generate corrective instructions if a staff member is taking the wrong route.

[0279] Data processing and calculation: Applying algorithms to recalculate guide data and generate corrective instructions.

[0280] Output: Corrective guidance data (e.g. "Turn left at the next corner")

[0281] This allows staff to provide appropriate store guidance in real time.

[0282] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0283] The present invention provides a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion engine. The program processing of this system will be explained below in natural language.

[0284] 1. Voice Recognition Method

[0285] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and receives it as audio data. The speech recognition module then converts the audio data into text data. This process passes the user's instructions in text form to the system.

[0286] 2. Natural Language Processing Methods

[0287] The device then sends this text data to a natural language processing engine, which analyzes the text and extracts the user's intent, in this case "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[0288] 3. Emotion Engine

[0289] In addition, the device is equipped with an emotion engine that recognizes the user's emotions from speech recognition. For example, if the user is anxious, the engine will detect "stress" from the tone and speed of the speech. This emotion information is used to generate and display subsequent operation guides.

[0290] 4. Operation guide generation method

[0291] The device generates a guide with relevant operation steps based on the analyzed intent and recognized emotion. For example, the first step is to open the YouTube app, so the device locates the YouTube app icon on the home screen. Next, it provides specific steps for finding the search bar and typing "cat." If the user is stressed, the device generates a more detailed guide than usual.

[0292] 5. User Interface Display Method

[0293] The procedure generated by the operation guide generation means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this display continues until the user completes the tap operation. The display method and level of detail of the guide are adjusted according to the user's emotional state.

[0294] 6. User Action Tracking Methods

[0295] The device tracks the user's actions in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar and monitoring the user's actions when typing "cat."

[0296] 7. Correction guide generation method

[0297] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly. If the user is feeling stressed, the means displays more detailed correction guides.

[0298] Specific examples

[0299] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[0300] 1. The user speaks.

[0301] 2. The device converts the voice into text and analyzes it using an NLP engine.

[0302] 3. The device analyzes the user's emotions using an emotion engine. For example, it determines that the user is anxious.

[0303] 4. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video). If the user is in a hurry, more detailed steps are generated.

[0304] 5. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[0305] 6. The user follows the instructions and taps the YouTube icon.

[0306] 7. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[0307] 8. User types "cat" and performs a search.

[0308] 9. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[0309] 10. User taps on the video to start playing.

[0310] In this way, users can smoothly perform desired operations using only voice utterances. Furthermore, by incorporating an emotion engine, it is possible to provide appropriate assistance according to the user's emotional state, resulting in a more user-friendly system.

[0311] The processing flow will be explained below.

[0312] Step 1:

[0313] The user speaks into their smartphone, "I want to watch cat videos on YouTube."

[0314] Step 2:

[0315] The terminal uses a built-in microphone to capture the user's voice and receives it as voice data.

[0316] Step 3:

[0317] The terminal passes the received voice data to a voice recognition module and converts it into text data.

[0318] Step 4:

[0319] The device sends the converted text data to a natural language processing engine.

[0320] Step 5:

[0321] The device uses a natural language processing engine to analyze the text data and extract the user's intent. The intents identified are "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[0322] Step 6:

[0323] The device sends the text data to an emotion engine, which analyzes the user's emotions. For example, it can determine that the user is anxious based on the tone and speed of the voice.

[0324] Step 7:

[0325] The device generates operation guidance based on the analyzed intention and the user's emotions. If the user is in a hurry, the operation guidance will include more detailed steps than usual.

[0326] Step 8:

[0327] Your device will locate the YouTube app icon on your home screen.

[0328] Step 9:

[0329] The device will display a cursor pointing to the YouTube app icon and a guide message saying "Press here."

[0330] Step 10:

[0331] The user taps the YouTube icon.

[0332] Step 11:

[0333] Make sure the YouTube app is launched correctly on your device.

[0334] Step 12:

[0335] The device identifies the search bar within the YouTube app.

[0336] Step 13:

[0337] The device displays a cursor pointing to the search bar and a guide message saying "Type 'cat' here."

[0338] Step 14:

[0339] A user types "cat" into the search bar and presses Enter.

[0340] Step 15:

[0341] The device confirms the user's input and verifies that search results are displayed.

[0342] Step 16:

[0343] The device locates the first video in the search results.

[0344] Step 17:

[0345] The device will display a cursor pointing to the video and a guide message saying "Play this."

[0346] Step 18:

[0347] The user taps on the first video and it begins playing.

[0348] Step 19:

[0349] The terminal confirms the completion of the user operation and ends the guide display.

[0350] Example 2

[0351] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0352] Conventional systems have difficulty accurately analyzing the user's intention when giving voice instructions, and the operation guidance provided does not take into account the user's emotional state. As a result, users often feel stressed or make mistakes while operating the device. Furthermore, if a user makes a mistake, no appropriate correction guidance is provided, forcing the user to manually correct the mistake, resulting in an insufficient operation experience.

[0353] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0354] In this invention, the server includes a voice recognition unit, a natural language processing unit, an emotion recognition unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit. This makes it possible to accurately analyze the user's voice instructions and provide detailed operation guides based on the user's emotional state. Furthermore, by quickly presenting correction guides when the user makes an operation error, the operation experience can be made smoother and stress-free.

[0355] A "voice recognition means" is a means for capturing a user's speech, saving it as voice data, and then converting it into text data.

[0356] "Natural language processing means" is a means for analyzing text data and extracting the user's intent.

[0357] An "emotion recognition means" is a means for analyzing a user's emotional state from the user's speech or text data.

[0358] The "operation guide generation means" is a means for generating operation procedures based on the user's intentions and emotional state, and providing the user with appropriate guidance.

[0359] The "user interface display means" is a means for overlaying and displaying the generated operation guide on the screen.

[0360] The "user operation tracking means" is a means for tracking the user's operation behavior in real time and monitoring whether the operation has been performed correctly.

[0361] The "correction guide generating means" is a means for providing appropriate correction information when the user makes an incorrect operation, and urging the user to perform the correct operation again.

[0362] The present invention provides a system for enabling a user to efficiently achieve a desired operation through voice instructions. A specific embodiment of the system will be described below.

[0363] Program processing

[0364] Voice recognition means

[0365] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and save it as audio data. The device then converts the speech data into text using the Google Cloud Speech-to-Text API. This process passes the user's instructions to the system in text format.

[0366] Natural language processing tools

[0367] The device sends the acquired text data to the Google Cloud Natural Language API, which analyzes the text and extracts the user's intent. In this case, it identifies the user's intent, such as "open YouTube," "enter 'cat' in the search bar," and "select a video from the search results."

[0368] emotion recognition means

[0369] The device uses IBM Watson Tone Analyzer to recognize the user's emotions from speech recognition and text data. For example, if the user is feeling anxious, the device can detect "stress" from the tone and speed of the voice. The recognized emotional information is reflected in the generation of the next operation guide.

[0370] Operation guide generation means

[0371] The device generates relevant instructions based on the analyzed intent and emotional state. Using ARKit or ARCore, the device identifies the YouTube app icon on the user's smartphone screen, then creates detailed instructions for finding the search bar and typing "cat." For example, if the user is feeling stressed, the device generates more detailed instructions than usual.

[0372] User interface display means

[0373] The generated operation guide is overlaid on the screen using Unity or Flutter (registered trademark). For example, a message saying "Press here" is displayed on top of the YouTube icon, and remains displayed until the user taps it. The display method and level of detail of the guide are adjusted depending on the emotion.

[0374] User action tracking

[0375] The device uses Firebase Analytics to track user behavior in real time, for example, to ensure that a user successfully opens the YouTube app and then types "cat" into the search bar.

[0376] Correction guide generation method

[0377] If the user makes an incorrect operation, the device generates a correction guide. For example, if the user accidentally taps the wrong icon, the device will provide a correction message saying, "You made a mistake. Please open YouTube." The device will then provide a guide pointing back to the YouTube icon. The correction guide will be adjusted to be more detailed depending on the user's emotional state.

[0378] Specific examples

[0379] As a concrete example, the following sequence of events occurs when a user says, "I want to watch cat videos on YouTube."

[0380] 1. The user speaks.

[0381] 2. The device converts the voice into text and analyzes it using a natural language processing engine.

[0382] 3. The device analyzes the user's emotions using an emotion recognition engine. For example, it determines that the user is anxious.

[0383] 4. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video). If the user is in a hurry, more detailed steps are generated.

[0384] 5. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[0385] 6. The user follows the instructions and taps the YouTube icon.

[0386] 7. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[0387] 8. User types "cat" and performs a search.

[0388] 9. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[0389] 10. User taps on the video to start playing.

[0390] Prompt Sentence Examples

[0391] An example of a prompt is as follows:

[0392] "I want to watch cat videos on YouTube. Convert the audio data into text, analyze the user's intent and emotions, and generate detailed operating instructions."

[0393] The above is a specific embodiment for carrying out the present invention. This system allows users to efficiently perform desired operations using only voice input, and by incorporating emotion recognition, it is possible to provide optimal support according to the user's condition.

[0394] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0395] Program processing flow

[0396] Step 1:

[0397] Audio capture and transcription

[0398] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and save it as audio data. The device then sends this audio data to the Google Cloud Speech-to-Text API, which converts it into text.

[0399] Input: Audio data

[0400] Output: Text data ("I want to watch cat videos on YouTube")

[0401] Step 2:

[0402] Intention analysis of text data

[0403] The device sends the acquired text data to the Google Cloud Natural Language API. This API analyzes the input text data and extracts the user's intent. For example, the analysis results may lead to operational steps such as "open the YouTube app," "enter 'cat' in the search bar," and "select a video from the search results."

[0404] Input: Text data

[0405] Output: User intent (list of action steps)

[0406] Step 3:

[0407] Emotion recognition

[0408] The device uses IBM Watson Tone Analyzer to analyze the user's emotions from voice and text data. For example, it can detect "stress" from the tone and speed of the user's voice. This emotional information is reflected in the generation of subsequent operation guides.

[0409] Input: Audio data, text data

[0410] Output: Emotion data (e.g., stress)

[0411] Step 4:

[0412] Generate operation guides

[0413] Based on the analyzed intent and emotional state, the device uses ARKit and ARCore to generate operation instructions, specifically, to identify the YouTube app icon on the user's smartphone screen, then create detailed instructions for finding the search bar and typing "cat."

[0414] Input: User intent, emotional data

[0415] Output: Operation Guide (detailed instructions)

[0416] Step 5:

[0417] View Guide

[0418] The device uses Unity or Flutter to overlay the generated operation guide on the screen. Specifically, a message saying "Press here" appears above the YouTube icon and remains displayed until the user taps it. The display method and level of detail of the guide are adjusted depending on the emotion.

[0419] Input: Operation Guide

[0420] Output: Guide display on the user interface

[0421] Step 6:

[0422] Tracking user actions

[0423] The device uses Firebase Analytics to track user behavior in real time, for example, whether a user successfully opens the YouTube app or types "cat" into the search bar.

[0424] Input: User operation data

[0425] Output: Operation tracking data (flag indicating whether the operation was successful)

[0426] Step 7:

[0427] Generate repair guides

[0428] If the user makes a mistake, the device generates a correction guide. For example, if the user accidentally taps a different icon, the device will notify the user, "You made a mistake. Please open YouTube," and provide a guide pointing back to the YouTube icon.

[0429] Input: Operation tracking data

[0430] Output: Fix guide

[0431] By performing these steps consecutively, users receive detailed and appropriate operation guidance through voice alone, enabling them to smoothly accomplish their desired operations. Furthermore, guidance is adjusted based on the user's emotional state, providing a more user-friendly experience.

[0432] (Application example 2)

[0433] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0434] In conventional systems, even when users issue voice instructions, it can be difficult for them to accurately understand and operate the system. Furthermore, if the user makes an incorrect operation, no appropriate guidance is provided, which can lead to operational delays and increased stress and frustration for the user. Furthermore, conventional systems do not take into account the user's emotional state when providing operation guidance, resulting in a lack of a user-friendly interface. This problem, which reduces the quality of the user experience, needs to be addressed.

[0435] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0436] In this invention, the server includes a voice recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion analysis unit. This allows the server to accurately understand the user's voice instructions, provide appropriate operation procedures, and provide customized support according to the user's emotional state. Furthermore, by providing correction guides in real time for user operation errors, the server allows the user to complete operations without stress.

[0437] "Speech recognition means" is a technology that captures a user's speech as voice data and converts it into text data.

[0438] "Natural language processing means" is a technology that analyzes the user's intentions from the character data converted by the voice recognition means.

[0439] The "operation guide generation means" is a technology that generates related operation procedures based on the analyzed user's intentions.

[0440] The "user interface display means" is a technique for visually displaying to the user the procedures generated by the operation guide generation means.

[0441] "User operation tracking means" is a technology that tracks user operation behavior in real time and confirms that the operation is being performed accurately.

[0442] The "correction guide generating means" is a technology that provides appropriate correction information when the user performs an incorrect operation.

[0443] "Emotion analysis means" is a technology that analyzes emotions from the content of a user's speech and determines their emotional state.

[0444] The present invention provides a system including a voice recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion analysis unit. The system aims to accurately understand a user's voice instructions, display appropriate operation guides according to the user's emotional state, and enable the user to smoothly perform the intended operations.

[0445] 1. Voice Recognition Method

[0446] First, the user speaks into the device. For example, if the user says, "I want to watch the latest movie," the device uses the built-in microphone to capture this speech and receives it as voice data. The voice recognition module then converts the voice data into text data. Through this process, the user's instructions are passed on to the system in text format. The speech_recognition library is used for this voice recognition.

[0447] 2. Natural Language Processing Methods

[0448] The device then sends this text data to a natural language processing engine, which uses Hugging Face's transformers library to analyze the user's intent, in this case "Show me the latest movies."

[0449] 3. Emotion analysis means

[0450] The device is also equipped with an emotion analysis unit that analyzes the user's emotions from speech recognition. The prompt text is used to input the emotion model, which determines whether the user is feeling anxious or stressed. This emotion information is then used to generate and display subsequent operation guides. The emotion analysis also utilizes the emotion analysis model from Hugging Face's transformers library.

[0451] 4. Operation guide generation method

[0452] The device generates a guide with relevant operation steps based on the analyzed intent and the recognized emotion. For example, the first step is "Display movie list." If the device recognizes that the user is impatient, the operation guide generates more detailed instructions than usual, allowing the user to proceed to the next step without hesitation.

[0453] 5. User Interface Display Method

[0454] The procedure generated by the operation guide generation means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the movie list icon, and this display continues until the user completes the tap operation. The display method and level of detail of the guide are adjusted according to the user's emotional state.

[0455] 6. User Action Tracking Methods

[0456] The device tracks the user's actions in real time, ensuring that the user correctly opens the movie list as intended, and then monitors other actions.

[0457] 7. Correction guide generation method

[0458] If the user does not perform the operation correctly, the correction guide generation means will provide appropriate correction information. For example, if the user accidentally taps a different icon, the means will notify the user, "You made a mistake. Please open the movie list." and display a guide pointing back to the movie list icon. If the user enters an incomplete search bar entry, the means will instruct the user, "Please enter 'movies' correctly." If the user is feeling stressed, the means will display more detailed correction guides.

[0459] Specific examples

[0460] The following sequence of events occurs when a user says, "I want to watch the latest movies."

[0461] 1. The user speaks.

[0462] 2. The device converts the voice into text and analyzes it using a natural language processing engine.

[0463] 3. The device analyzes the user's emotions using emotion analysis means. For example, it determines that the user is anxious.

[0464] 4. The device generates the necessary instructions (show movie list). If the user is impatient, more detailed instructions are generated.

[0465] 5. The device will display a cursor pointing at the movie list icon and a prompt saying "Press here."

[0466] 6. The user follows the instructions and taps the movie list icon.

[0467] 7. Your device will display a cursor pointing to the search bar and a prompt saying "Type 'Movies' here."

[0468] 8. User types "movies" and performs a search.

[0469] 9. The device will display a cursor pointing to the appropriate movie and a prompt saying "Play this."

[0470] 10. The user taps on a movie to start playing it.

[0471] An example prompt is:

[0472] User: "I want to watch the latest movies"

[0473] System: "Show the latest movies. Searching..."

[0474] In this way, users can smoothly perform desired operations using only voice utterances. Furthermore, by incorporating emotion analysis means, it is possible to provide appropriate assistance according to the user's emotional state, resulting in a more user-friendly system.

[0475] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0476] Step 1:

[0477] Start voice recognition

[0478] Input: User says "I want to watch the latest movies."

[0479] The server uses the built-in microphone to capture the user's voice, and converts this voice data into text data using the speech_recognition library.

[0480] Output: The converted text data, e.g. "I want to watch the latest movies."

[0481] Step 2:

[0482] Text data analysis

[0483] Input: Text data generated in step 1

[0484] The server uses Hugging Face's transformers library to send the text data to a natural language processing engine to analyze the user's intent, which is "Show me the latest movies."

[0485] Output: User intent ("Show me the latest movies").

[0486] Step 3:

[0487] Performing sentiment analysis

[0488] Input: Text data generated in step 1

[0489] The server then performs emotion analysis using the Hugging Face transformers library, again to determine the user's emotional state, such as whether they are anxious or stressed.

[0490] Output: User's emotional information (e.g., feeling anxious).

[0491] Step 4:

[0492] Generate operation guides

[0493] Input: User intent identified in step 2, user sentiment information obtained in step 3

[0494] The server generates relevant instructional steps based on the user's intention and emotional state. For example, if the user is impatient, it generates detailed instructions ("Tap the movie icon in the bottom right of the screen").

[0495] Output: Detailed operation guide.

[0496] Step 5:

[0497] Display in the user interface

[0498] Input: The operation guide generated in step 4

[0499] The terminal displays the generated operation guide as an overlay on the user interface, for example, by displaying a cursor and a message saying "Press here" on a movie list icon.

[0500] Output: A visual guide on the user interface.

[0501] Step 6:

[0502] User Action Tracking

[0503] Input: User action (user taps on the movie icon)

[0504] The device tracks the user's actions in real time, confirms whether the user performed the action correctly as intended, and takes the next step if necessary.

[0505] Output: User behavior data.

[0506] Step 7:

[0507] Providing a guide to remediation

[0508] Input: User behavior data obtained in step 6

[0509] If the user makes an incorrect operation, the server uses a correction guide generation means to provide appropriate correction information. For example, if the user accidentally taps a different icon, the server displays a message saying "You made a mistake. Please open the movie list" and instructs the user to perform the correct operation again.

[0510] Output: Correction guide and new operating instructions.

[0511] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0512] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0513] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0514] [Second embodiment]

[0515] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0516] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0517] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0518] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0519] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0520] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0521] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0522] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0523] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0524] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0525] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0526] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0527] The present invention provides a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit. The program processing of this system will be explained below in natural language.

[0528] 1. Voice Recognition Method

[0529] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and receives it as audio data. The speech recognition module then converts the audio data into text data. This process passes the user's instructions in text form to the system.

[0530] 2. Natural Language Processing Methods

[0531] The device then sends this text data to a natural language processing engine, which analyzes the text and extracts the user's intent, in this case "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[0532] 3. Operation guide generation method

[0533] Based on the analyzed intent, the device will generate a guide with relevant operation steps. For example, the first step is to open the YouTube app, so the device will locate the YouTube app icon on the home screen. Next, it will find the search bar and provide specific instructions for typing "cat."

[0534] 4. User Interface Display Method

[0535] The procedure generated by the operation guide generating means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this display continues until the user completes the tap operation.

[0536] 5. User Action Tracking Methods

[0537] The device tracks the user's actions in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar and monitoring the user's actions when typing "cat."

[0538] 6. Correction guide generation method

[0539] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly.

[0540] Specific examples

[0541] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[0542] 1. The user speaks.

[0543] 2. The device converts the voice into text and analyzes it using an NLP engine.

[0544] 3. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video).

[0545] 4. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[0546] 5. The user follows the instructions and taps the YouTube icon.

[0547] 6. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[0548] 7. User types "cat" and performs a search.

[0549] 8. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[0550] 9. User taps on the video to start playing.

[0551] In this way, the user can smoothly perform the desired operation by simply speaking. The present invention is a system that is particularly useful for users who are not familiar with technology and elderly people.

[0552] The processing flow will be explained below.

[0553] Step 1:

[0554] The user speaks into their smartphone, "I want to watch cat videos on YouTube."

[0555] Step 2:

[0556] The terminal uses a built-in microphone to capture the user's voice and receives it as voice data.

[0557] Step 3:

[0558] The terminal passes the received voice data to a voice recognition module and converts it into text data.

[0559] Step 4:

[0560] The device sends the converted text data to a natural language processing engine.

[0561] Step 5:

[0562] The device uses a natural language processing engine to analyze the text data and extract the user's intent, such as "open YouTube," "type 'cat' into the search bar," or "select a video from the search results."

[0563] Step 6:

[0564] The device prepares an operation procedure to generate an operation guide based on the analyzed intent. For example, first, it collects location information to identify the YouTube app icon.

[0565] Step 7:

[0566] The device will generate an operation guide and display a cursor pointing to the YouTube app icon and a guide saying "Press here" on the screen.

[0567] Step 8:

[0568] The user taps the YouTube icon.

[0569] Step 9:

[0570] The device detects the user's operation and confirms that the YouTube app has been launched correctly.

[0571] Step 10:

[0572] The device will identify the search bar within the YouTube app and display a cursor pointing to it with the prompt "Type 'cat' here."

[0573] Step 11:

[0574] A user types "cat" into the search bar and presses Enter.

[0575] Step 12:

[0576] The device detects whether the user's input was correct and verifies that the search results are displayed.

[0577] Step 13:

[0578] The device will locate the first video in the search results and display a cursor pointing to the video and a prompt saying "Play this."

[0579] Step 14:

[0580] The user taps on the first video and it begins playing.

[0581] Step 15:

[0582] The terminal confirms the completion of the user operation and ends the guide display.

[0583] Through the above steps, the system allows users to easily and reliably perform desired operations on their smartphones using voice utterances.

[0584] Example 1

[0585] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0586] Conventional voice-operated systems have difficulty accurately recognizing user utterances and analyzing their intentions, making it difficult to effectively support operations, especially for users who are unfamiliar with technology or the elderly. Furthermore, there is a lack of means to provide appropriate correction guidance in the event of an incorrect operation, which means it takes time to correct the incorrect operation, reducing usability.

[0587] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0588] In this invention, the server includes a voice recognition means, a natural language processing means, and an operation guide generation means, which make it possible to capture a user's voice instructions, convert them into text data, analyze the text data to extract the user's intention, and generate and display operation instructions based on the analyzed intention.

[0589] "Speech recognition means" is a technology that captures a user's speech as voice data and converts it into text data.

[0590] "Natural language processing means" is a technology that analyzes and extracts user intent from text data.

[0591] The "operation guide generation means" is a technology that generates an operation procedure to be presented to the user based on the analyzed intention.

[0592] The "user interface display means" is a technique for visually displaying the generated operation guide to the user.

[0593] "User operation tracking means" is a technology that monitors and tracks the operations performed by a user in real time.

[0594] The "correction guide generating means" is a technology that detects an incorrect operation made by a user and generates a guide that instructs the user on the correct operation.

[0595] "Text data" refers to character data converted by a speech recognition means.

[0596] "Intent analysis" refers to the process of interpreting user intent from text data using natural language processing tools.

[0597] "Operation instructions" refer to instructions that present the user with specific operations that should be performed based on the user's intentions.

[0598] An "incorrect operation" is when a user performs an operation that differs from the operation instructions presented by the system.

[0599] "Correction information" refers to specific instruction information for correcting an incorrect operation and performing the correct operation.

[0600] The present invention relates to a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit, and the system aims to capture a user's voice instructions, convert them into text data, analyze their intentions, and present the user with the necessary operation procedures.

[0601] 1. Voice Recognition Method

[0602] A user speaks into their smartphone, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and record it as audio data. The device's speech recognition module (e.g., Google Cloud Speech-to-Text API) then converts the audio data into text data. This process passes the user's instructions to the system in text format.

[0603] 2. Natural Language Processing Methods

[0604] The text data obtained by the speech recognition module is sent to a natural language processing engine (e.g., OpenAI GPT-3). The NLP engine analyzes the text and extracts the user's intent. In this case, the intent is identified as "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[0605] 3. Operation guide generation method

[0606] Based on the analyzed user intent, the operation guide generator generates relevant operation steps. For example, the first step is to open the YouTube app, so the device identifies the location of the YouTube app icon on the home screen. Next, it provides specific steps for finding the search bar and entering "cat."

[0607] 4. User Interface Display Method

[0608] The procedure generated by the operation guide generating means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this continues until the user completes the tap operation.

[0609] 5. User Action Tracking Methods

[0610] The device tracks user behavior in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar, and even monitoring whether the user is typing "cat" correctly.

[0611] 6. Correction guide generation method

[0612] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly.

[0613] Specific examples

[0614] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[0615] 1. The user says, "I want to watch cat videos on YouTube."

[0616] 2. The device converts the voice into text and analyzes it using an NLP engine.

[0617] 3. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video).

[0618] 4. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[0619] 5. The user follows the instructions and taps the YouTube icon.

[0620] 6. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[0621] 7. User types "cat" and performs a search.

[0622] 8. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[0623] 9. User taps on the video to start playing.

[0624] In this way, the user can smoothly perform the desired operation using only voice utterances. This system is particularly useful for users who are not familiar with technology or the elderly. As an example prompt sentence, the following sentence is input to the generative AI model:

[0625] "If a user says they want to watch cat videos on YouTube, explain how the system generates, displays, and supports the user with navigation."

[0626] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0627] Step 1:

[0628] A user says, "I want to watch cat videos on YouTube."

[0629] Input: User utterance.

[0630] Output: The captured audio data.

[0631] How it works: A user verbally types into their smartphone, "I want to watch cat videos on YouTube."

[0632] Step 2:

[0633] The device captures the user's speech using a built-in microphone.

[0634] Input: User utterance.

[0635] Output: Audio data.

[0636] How it works: The smartphone's built-in microphone records the user's speech as audio data.

[0637] Step 3:

[0638] The device's voice recognition module converts the voice data into text data.

[0639] Input: Audio data.

[0640] Output: Character data.

[0641] What happens: A speech recognition module (e.g., Google Cloud Speech-to-Text API) analyzes the captured audio data and converts it into corresponding text data ("I want to watch cat videos on YouTube").

[0642] Step 4:

[0643] The device sends the text data to a natural language processing engine.

[0644] Input: Character data.

[0645] Output: User intent.

[0646] How it works: The converted text data is sent to a natural language processing engine (e.g., OpenAI GPT-3) to prepare it for analyzing the user's intent.

[0647] Step 5:

[0648] A natural language processing engine analyzes the text data and extracts the user's intent.

[0649] Input: Character data.

[0650] Output: User intent (e.g., "Open YouTube," "Type 'cat' into the search bar," "Select a video from the search results").

[0651] How it works: A natural language processing engine parses the text data it receives and identifies the user's request ("open YouTube," "type 'cat' in the search bar," "select a video from the search results").

[0652] Step 6:

[0653] The terminal generates operating procedures based on the analyzed intention.

[0654] Input: User intent.

[0655] Output: Operation instructions.

[0656] Action: The device constructs the necessary steps based on the user's intent (e.g., the steps to open the YouTube app).

[0657] Step 7:

[0658] Your device will locate the YouTube app icon on your home screen.

[0659] Input: Intention-based operating instructions.

[0660] Output: YouTube icon position.

[0661] What it does: Your device uses a digital map on your home screen to locate the YouTube icon.

[0662] Step 8:

[0663] The device overlays a guide on the user interface to guide you through the operation process.

[0664] Input: The location of the identified icon.

[0665] Output: On-screen cursor and guide message.

[0666] Behavior: The device displays a cursor over the YouTube icon and the message "Press here" on the screen until the user taps it.

[0667] Step 9:

[0668] The device tracks the user's operating behavior in real time.

[0669] Input: User taps.

[0670] Output: Confirmation that the operation was successful.

[0671] What it does: The device monitors and checks if the user has successfully launched the YouTube app.

[0672] Step 10:

[0673] Your device will locate the search bar and then guide you through typing "cat."

[0674] Input: Confirmation information for operation completion.

[0675] Output: Search bar position and guide message.

[0676] Behavior: The device displays a cursor pointing at the search bar and the message "Type 'cat' here" until the user types something.

[0677] Step 11:

[0678] A user types in "cat" and performs a search.

[0679] Input: Guide message.

[0680] Output: The search term entered.

[0681] What happens: A user types "cat" into the search bar and presses the search button.

[0682] Step 12:

[0683] The device will identify the appropriate video from the search results and display a guide pointing to it.

[0684] Input: The search term entered.

[0685] Output: Video position and guide message.

[0686] What it does: The device displays a cursor pointing to the appropriate video in the search results and the message "Play this" until the user makes a selection.

[0687] Step 13:

[0688] The user taps on the video to begin playing it.

[0689] Input: Guide message.

[0690] Output: Video begins playing.

[0691] How it works: The user is guided to tap on a video and it begins playing.

[0692] Through this series of steps, the user can smoothly perform the desired operation using only voice utterances.

[0693] (Application example 1)

[0694] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0695] With conventional store guidance systems, it was difficult for staff to grasp their physical location within the store and provide appropriate guidance according to customer requests. Furthermore, in order for staff to accurately guide customers in real time, they had to process a large amount of information in a short amount of time, which placed a heavy burden on staff. Furthermore, it was difficult to provide prompt corrective instructions when a customer was in the wrong place or made a mistake.

[0696] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0697] In this invention, the server includes a voice recognition means, a natural language processing means, an operation guide generation means, a user interface display means, a user operation tracking means, a correction guide generation means, a means for tracking position information in physical space, and a means for overlaying and displaying information in the user's field of view, thereby enabling store guidance in response to customer requests in real time.

[0698] A "voice recognition means" is a device or software module that converts voice data into text data.

[0699] A "natural language processing means" is a device or software module that analyzes user intent from text data.

[0700] The "operation guide generation means" is a device or software module that generates an operation procedure based on the analyzed intention.

[0701] The "user interface display means" is a device or software module that displays operation guides on a screen or display.

[0702] A "user operation tracking means" is a device or software module that tracks a user's operation behavior in real time.

[0703] The "correction guide generating means" is a device or software module that generates correction instructions when the user performs an incorrect operation.

[0704] A "physical location tracking device" is a device or software module that tracks a user's physical location using GPS or other location technology.

[0705] "Means for overlaying and displaying information in the user's field of view" refers to a device or software module that displays information by overlaying it in the user's field of view using smart glasses, a head-mounted display, or the like.

[0706] The present invention relates to a system that provides real-time store guidance in response to customer requests. The system combines the functions of voice recognition, natural language processing, operation guidance, user interface display, user operation tracking, correction guidance, location information tracking, and overlay display.

[0707] This system is configured as follows:

[0708] Voice recognition means

[0709] The server converts the voice received by the staff through devices such as smart glasses into text data using voice recognition software (e.g., Google Speech-to-Text API).

[0710] Natural language processing tools

[0711] The text data is analyzed by a server-side natural language processing engine (e.g., Google Cloud Natural Language API or IBM Watson) to extract the user's intent. For example, if a customer asks, "Where is the wine section?", the intent is extracted as "Find the location of the wine section."

[0712] Operation guide generation means

[0713] Based on the analyzed intent, the server generates a guide for the relevant operating procedures. Specifically, it refers to the store's map data to calculate the optimal route and generates data for displaying the guide.

[0714] User interface display means

[0715] The procedures generated by the operation guide generation means are overlaid on the display of the smart glasses worn by the staff member, using a user interface framework such as Unity.

[0716] User action tracking

[0717] The server tracks the physical location of staff in real time using GPS sensors and other location technologies, ensuring they are heading in the right direction.

[0718] Correction guide generation method

[0719] If the user is heading in the wrong direction or making a mistake, the server will instantly generate corrective instructions and display them on the smart glasses' display, such as "Turn left at the next corner."

[0720] Specific examples

[0721] As a concrete example, if a customer asks, "Where is the wine section?", the sequence of events is as follows:

[0722] 1. Staff wear smart glasses and capture customer questions using voice recognition.

[0723] 2. The server converts the speech to text.

[0724] 3. The NLP engine analyzes the question and extracts the intent: "Find a wine section."

[0725] 4. The store map data is referenced to generate a route guide, and the smart glasses display shows "Turn right from here."

[0726] 5. The server uses GPS data to track the staff's location and ensure they are heading in the right direction.

[0727] 6. If you are heading to the wrong location, the server will display a corrective instruction such as "Turn left at the next corner."

[0728] Prompt Sentence Examples

[0729] Enter the following prompts into the AI ​​model to enable it to generate the appropriate program:

[0730] Based on the user's voice question, your goal is to generate a program that uses an NLP engine to analyze the intent and provide appropriate product information and floor guides in real time. For example, if a customer asks, "Where is the wine section?", please explain the following steps:

[0731] 1. Voice Recognition

[0732] 2. NLP analysis

[0733] 3. Guide Generation

[0734] 4. UI display

[0735] 5. Operation Tracking

[0736] 6. Guide correction

[0737] Please also include specific examples of the hardware and software you will use.

[0738] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0739] Step 1:

[0740] Audio Capture and Conversion

[0741] Input: The user speaks a voice question into the smart glasses.

[0742] Specific operation: The staff activates the voice input function of the smart glasses and captures the question. The smart glasses use the built-in microphone to capture the voice data.

[0743] Data processing and calculation: A speech recognition module (e.g., Google Speech-to-Text API) converts the voice data into text data.

[0744] Output: Text data (e.g., "Where is the wine store?")

[0745] Step 2:

[0746] Text analysis and intent extraction

[0747] Input: Text data sent from smart glasses

[0748] Specific operation: The server receives the text data and sends it to a natural language processing engine (e.g., Google Cloud Natural Language API).

[0749] Data processing and calculation: A natural language processing engine analyzes text data and extracts user intent.

[0750] Output: Intent data (e.g., "Find a wine section")

[0751] Step 3:

[0752] Operation guide generation

[0753] Input: Intent data

[0754] Specific operation: Based on the intent data, the server references the location information within the store and calculates the appropriate guidance route.

[0755] Data processing and calculation: Taking information from the store's map database and applying algorithms to calculate the optimal route.

[0756] Output: Guide data (e.g. "Turn right from here")

[0757] Step 4:

[0758] User Interface Display

[0759] Input: Guide data

[0760] Specific operation: The server sends guide data to the smart glasses, which overlay it on the user's field of view. The smart glasses then display arrows and guide messages on the display.

[0761] Data processing and computation: Converting guide data into visual information using a user interface framework (e.g., Unity).

[0762] Output: Visual data (e.g., "Please turn right")

[0763] Step 5:

[0764] User Action Tracking

[0765] Input: Location data from smart glasses

[0766] What it does: The server tracks the physical location of staff in real time using GPS sensors and other location technologies.

[0767] Data processing and calculation: Analyze the acquired location data and check whether the staff is following the correct route.

[0768] Output: Tracking data (e.g. flags indicating whether you are moving in the right direction)

[0769] Step 6:

[0770] Correction guide generation

[0771] Input: Tracking data

[0772] Specific behavior: The server uses tracking data to generate corrective instructions if a staff member is taking the wrong route.

[0773] Data processing and calculation: Applying algorithms to recalculate guide data and generate corrective instructions.

[0774] Output: Corrective guidance data (e.g. "Turn left at the next corner")

[0775] This allows staff to provide appropriate store guidance in real time.

[0776] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0777] The present invention provides a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion engine. The program processing of this system will be explained below in natural language.

[0778] 1. Voice Recognition Method

[0779] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and receives it as audio data. The speech recognition module then converts the audio data into text data. This process passes the user's instructions in text form to the system.

[0780] 2. Natural Language Processing Methods

[0781] The device then sends this text data to a natural language processing engine, which analyzes the text and extracts the user's intent, in this case "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[0782] 3. Emotion Engine

[0783] In addition, the device is equipped with an emotion engine that recognizes the user's emotions from speech recognition. For example, if the user is anxious, the engine will detect "stress" from the tone and speed of the speech. This emotion information is used to generate and display subsequent operation guides.

[0784] 4. Operation guide generation method

[0785] The device generates a guide with relevant operation steps based on the analyzed intent and recognized emotion. For example, the first step is to open the YouTube app, so the device locates the YouTube app icon on the home screen. Next, it provides specific steps for finding the search bar and typing "cat." If the user is stressed, the device generates a more detailed guide than usual.

[0786] 5. User Interface Display Method

[0787] The procedure generated by the operation guide generation means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this display continues until the user completes the tap operation. The display method and level of detail of the guide are adjusted according to the user's emotional state.

[0788] 6. User Action Tracking Methods

[0789] The device tracks the user's actions in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar and monitoring the user's actions when typing "cat."

[0790] 7. Correction guide generation method

[0791] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly. If the user is feeling stressed, the means displays more detailed correction guides.

[0792] Specific examples

[0793] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[0794] 1. The user speaks.

[0795] 2. The device converts the voice into text and analyzes it using an NLP engine.

[0796] 3. The device analyzes the user's emotions using an emotion engine. For example, it determines that the user is anxious.

[0797] 4. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video). If the user is in a hurry, more detailed steps are generated.

[0798] 5. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[0799] 6. The user follows the instructions and taps the YouTube icon.

[0800] 7. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[0801] 8. User types "cat" and performs a search.

[0802] 9. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[0803] 10. User taps on the video to start playing.

[0804] In this way, users can smoothly perform desired operations using only voice utterances. Furthermore, by incorporating an emotion engine, it is possible to provide appropriate assistance according to the user's emotional state, resulting in a more user-friendly system.

[0805] The processing flow will be explained below.

[0806] Step 1:

[0807] The user speaks into their smartphone, "I want to watch cat videos on YouTube."

[0808] Step 2:

[0809] The terminal uses a built-in microphone to capture the user's voice and receives it as voice data.

[0810] Step 3:

[0811] The terminal passes the received voice data to a voice recognition module and converts it into text data.

[0812] Step 4:

[0813] The device sends the converted text data to a natural language processing engine.

[0814] Step 5:

[0815] The device uses a natural language processing engine to analyze the text data and extract the user's intent. The intents identified are "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[0816] Step 6:

[0817] The device sends the text data to an emotion engine, which analyzes the user's emotions. For example, it can determine that the user is anxious based on the tone and speed of the voice.

[0818] Step 7:

[0819] The device generates operation guidance based on the analyzed intention and the user's emotions. If the user is in a hurry, the operation guidance will include more detailed steps than usual.

[0820] Step 8:

[0821] Your device will locate the YouTube app icon on your home screen.

[0822] Step 9:

[0823] The device will display a cursor pointing to the YouTube app icon and a guide message saying "Press here."

[0824] Step 10:

[0825] The user taps the YouTube icon.

[0826] Step 11:

[0827] Make sure the YouTube app is launched correctly on your device.

[0828] Step 12:

[0829] The device identifies the search bar within the YouTube app.

[0830] Step 13:

[0831] The device displays a cursor pointing to the search bar and a guide message saying "Type 'cat' here."

[0832] Step 14:

[0833] A user types "cat" into the search bar and presses Enter.

[0834] Step 15:

[0835] The device confirms the user's input and verifies that search results are displayed.

[0836] Step 16:

[0837] The device locates the first video in the search results.

[0838] Step 17:

[0839] The device will display a cursor pointing to the video and a guide message saying "Play this."

[0840] Step 18:

[0841] The user taps on the first video and it begins playing.

[0842] Step 19:

[0843] The terminal confirms the completion of the user operation and ends the guide display.

[0844] Example 2

[0845] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0846] Conventional systems have difficulty accurately analyzing the user's intention when giving voice instructions, and the operation guidance provided does not take into account the user's emotional state. As a result, users often feel stressed or make mistakes while operating the device. Furthermore, if a user makes a mistake, no appropriate correction guidance is provided, forcing the user to manually correct the mistake, resulting in an insufficient operation experience.

[0847] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0848] In this invention, the server includes a voice recognition unit, a natural language processing unit, an emotion recognition unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit. This makes it possible to accurately analyze the user's voice instructions and provide detailed operation guides based on the user's emotional state. Furthermore, by quickly presenting correction guides when the user makes an operation error, the operation experience can be made smoother and stress-free.

[0849] A "voice recognition means" is a means for capturing a user's speech, saving it as voice data, and then converting it into text data.

[0850] "Natural language processing means" is a means for analyzing text data and extracting the user's intent.

[0851] An "emotion recognition means" is a means for analyzing a user's emotional state from the user's speech or text data.

[0852] The "operation guide generation means" is a means for generating operation procedures based on the user's intentions and emotional state, and providing the user with appropriate guidance.

[0853] The "user interface display means" is a means for overlaying and displaying the generated operation guide on the screen.

[0854] The "user operation tracking means" is a means for tracking the user's operation behavior in real time and monitoring whether the operation has been performed correctly.

[0855] The "correction guide generating means" is a means for providing appropriate correction information when the user makes an incorrect operation, and urging the user to perform the correct operation again.

[0856] The present invention provides a system for enabling a user to efficiently achieve a desired operation through voice instructions. A specific embodiment of the system will be described below.

[0857] Program processing

[0858] Voice recognition means

[0859] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and save it as audio data. The device then converts the speech data into text using the Google Cloud Speech-to-Text API. This process passes the user's instructions to the system in text format.

[0860] Natural language processing tools

[0861] The device sends the acquired text data to the Google Cloud Natural Language API, which analyzes the text and extracts the user's intent. In this case, it identifies the user's intent, such as "open YouTube," "enter 'cat' in the search bar," and "select a video from the search results."

[0862] emotion recognition means

[0863] The device uses IBM Watson Tone Analyzer to recognize the user's emotions from speech recognition and text data. For example, if the user is feeling anxious, the device can detect "stress" from the tone and speed of the voice. The recognized emotional information is reflected in the generation of the next operation guide.

[0864] Operation guide generation means

[0865] The device generates relevant instructions based on the analyzed intent and emotional state. Using ARKit or ARCore, the device identifies the YouTube app icon on the user's smartphone screen, then creates detailed instructions for finding the search bar and typing "cat." For example, if the user is feeling stressed, the device generates more detailed instructions than usual.

[0866] User interface display means

[0867] The generated operation guide is overlaid on the screen using Unity or Flutter. For example, a message saying "Press here" is displayed on top of the YouTube icon, and remains displayed until the user taps it. The display method and level of detail of the guide are adjusted depending on the emotion.

[0868] User action tracking

[0869] The device uses Firebase Analytics to track user behavior in real time, for example, to ensure that a user successfully opens the YouTube app and then types "cat" into the search bar.

[0870] Correction guide generation method

[0871] If the user makes an incorrect operation, the device generates a correction guide. For example, if the user accidentally taps the wrong icon, the device will provide a correction message saying, "You made a mistake. Please open YouTube." The device will then provide a guide pointing back to the YouTube icon. The correction guide will be adjusted to be more detailed depending on the user's emotional state.

[0872] Specific examples

[0873] As a concrete example, the following sequence of events occurs when a user says, "I want to watch cat videos on YouTube."

[0874] 1. The user speaks.

[0875] 2. The device converts the voice into text and analyzes it using a natural language processing engine.

[0876] 3. The device analyzes the user's emotions using an emotion recognition engine. For example, it determines that the user is anxious.

[0877] 4. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video). If the user is in a hurry, more detailed steps are generated.

[0878] 5. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[0879] 6. The user follows the instructions and taps the YouTube icon.

[0880] 7. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[0881] 8. User types "cat" and performs a search.

[0882] 9. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[0883] 10. User taps on the video to start playing.

[0884] Prompt Sentence Examples

[0885] An example of a prompt is as follows:

[0886] "I want to watch cat videos on YouTube. Convert the audio data into text, analyze the user's intent and emotions, and generate detailed operating instructions."

[0887] The above is a specific embodiment for carrying out the present invention. This system allows users to efficiently perform desired operations using only voice input, and by incorporating emotion recognition, it is possible to provide optimal support according to the user's condition.

[0888] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0889] Program processing flow

[0890] Step 1:

[0891] Audio capture and transcription

[0892] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and save it as audio data. The device then sends this audio data to the Google Cloud Speech-to-Text API, which converts it into text.

[0893] Input: Audio data

[0894] Output: Text data ("I want to watch cat videos on YouTube")

[0895] Step 2:

[0896] Intention analysis of text data

[0897] The device sends the acquired text data to the Google Cloud Natural Language API. This API analyzes the input text data and extracts the user's intent. For example, the analysis results may lead to operational steps such as "open the YouTube app," "enter 'cat' in the search bar," and "select a video from the search results."

[0898] Input: Text data

[0899] Output: User intent (list of action steps)

[0900] Step 3:

[0901] Emotion recognition

[0902] The device uses IBM Watson Tone Analyzer to analyze the user's emotions from voice and text data. For example, it can detect "stress" from the tone and speed of the user's voice. This emotional information is reflected in the generation of subsequent operation guides.

[0903] Input: Audio data, text data

[0904] Output: Emotion data (e.g., stress)

[0905] Step 4:

[0906] Generate operation guides

[0907] Based on the analyzed intent and emotional state, the device uses ARKit and ARCore to generate operation instructions, specifically, to identify the YouTube app icon on the user's smartphone screen, then create detailed instructions for finding the search bar and typing "cat."

[0908] Input: User intent, emotional data

[0909] Output: Operation Guide (detailed instructions)

[0910] Step 5:

[0911] View Guide

[0912] The device uses Unity or Flutter to overlay the generated operation guide on the screen. Specifically, a message saying "Press here" appears above the YouTube icon and remains displayed until the user taps it. The display method and level of detail of the guide are adjusted depending on the emotion.

[0913] Input: Operation Guide

[0914] Output: Guide display on the user interface

[0915] Step 6:

[0916] Tracking user actions

[0917] The device uses Firebase Analytics to track user behavior in real time, for example, whether a user successfully opens the YouTube app or types "cat" into the search bar.

[0918] Input: User operation data

[0919] Output: Operation tracking data (flag indicating whether the operation was successful)

[0920] Step 7:

[0921] Generate repair guides

[0922] If the user makes a mistake, the device generates a correction guide. For example, if the user accidentally taps a different icon, the device will notify the user, "You made a mistake. Please open YouTube," and provide a guide pointing back to the YouTube icon.

[0923] Input: Operation tracking data

[0924] Output: Fix guide

[0925] By performing these steps consecutively, users receive detailed and appropriate operation guidance through voice alone, enabling them to smoothly accomplish their desired operations. Furthermore, guidance is adjusted based on the user's emotional state, providing a more user-friendly experience.

[0926] (Application example 2)

[0927] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0928] In conventional systems, even when users issue voice instructions, it can be difficult for them to accurately understand and operate the system. Furthermore, if the user makes an incorrect operation, no appropriate guidance is provided, which can lead to operational delays and increased stress and frustration for the user. Furthermore, conventional systems do not take into account the user's emotional state when providing operation guidance, resulting in a lack of a user-friendly interface. This problem, which reduces the quality of the user experience, needs to be addressed.

[0929] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0930] In this invention, the server includes a voice recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion analysis unit. This allows the server to accurately understand the user's voice instructions, provide appropriate operation procedures, and provide customized support according to the user's emotional state. Furthermore, by providing correction guides in real time for user operation errors, the server allows the user to complete operations without stress.

[0931] "Speech recognition means" is a technology that captures a user's speech as voice data and converts it into text data.

[0932] "Natural language processing means" is a technology that analyzes the user's intentions from the character data converted by the voice recognition means.

[0933] The "operation guide generation means" is a technology that generates related operation procedures based on the analyzed user's intentions.

[0934] The "user interface display means" is a technique for visually displaying to the user the procedures generated by the operation guide generation means.

[0935] "User operation tracking means" is a technology that tracks user operation behavior in real time and confirms that the operation is being performed accurately.

[0936] The "correction guide generating means" is a technology that provides appropriate correction information when the user performs an incorrect operation.

[0937] "Emotion analysis means" is a technology that analyzes emotions from the content of a user's speech and determines their emotional state.

[0938] The present invention provides a system including a voice recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion analysis unit. The system aims to accurately understand a user's voice instructions, display appropriate operation guides according to the user's emotional state, and enable the user to smoothly perform the intended operations.

[0939] 1. Voice Recognition Method

[0940] First, the user speaks into the device. For example, if the user says, "I want to watch the latest movie," the device uses the built-in microphone to capture this speech and receives it as voice data. The voice recognition module then converts the voice data into text data. Through this process, the user's instructions are passed on to the system in text format. The speech_recognition library is used for this voice recognition.

[0941] 2. Natural Language Processing Methods

[0942] The device then sends this text data to a natural language processing engine, which uses Hugging Face's transformers library to analyze the user's intent, in this case "Show me the latest movies."

[0943] 3. Emotion analysis means

[0944] The device is also equipped with an emotion analysis unit that analyzes the user's emotions from speech recognition. The prompt text is used to input the emotion model, which determines whether the user is feeling anxious or stressed. This emotion information is then used to generate and display subsequent operation guides. The emotion analysis also utilizes the emotion analysis model from Hugging Face's transformers library.

[0945] 4. Operation guide generation method

[0946] The device generates a guide with relevant operation steps based on the analyzed intent and the recognized emotion. For example, the first step is "Display movie list." If the device recognizes that the user is impatient, the operation guide generates more detailed instructions than usual, allowing the user to proceed to the next step without hesitation.

[0947] 5. User Interface Display Method

[0948] The procedure generated by the operation guide generation means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the movie list icon, and this display continues until the user completes the tap operation. The display method and level of detail of the guide are adjusted according to the user's emotional state.

[0949] 6. User Action Tracking Methods

[0950] The device tracks the user's actions in real time, ensuring that the user correctly opens the movie list as intended, and then monitors other actions.

[0951] 7. Correction guide generation method

[0952] If the user does not perform the operation correctly, the correction guide generation means will provide appropriate correction information. For example, if the user accidentally taps a different icon, the means will notify the user, "You made a mistake. Please open the movie list." and display a guide pointing back to the movie list icon. If the user enters an incomplete search bar entry, the means will instruct the user, "Please enter 'movies' correctly." If the user is feeling stressed, the means will display more detailed correction guides.

[0953] Specific examples

[0954] The following sequence of events occurs when a user says, "I want to watch the latest movies."

[0955] 1. The user speaks.

[0956] 2. The device converts the voice into text and analyzes it using a natural language processing engine.

[0957] 3. The device analyzes the user's emotions using emotion analysis means. For example, it determines that the user is anxious.

[0958] 4. The device generates the necessary instructions (show movie list). If the user is impatient, more detailed instructions are generated.

[0959] 5. The device will display a cursor pointing at the movie list icon and a prompt saying "Press here."

[0960] 6. The user follows the instructions and taps the movie list icon.

[0961] 7. Your device will display a cursor pointing to the search bar and a prompt saying "Type 'Movies' here."

[0962] 8. User types "movies" and performs a search.

[0963] 9. The device will display a cursor pointing to the appropriate movie and a prompt saying "Play this."

[0964] 10. The user taps on a movie to start playing it.

[0965] An example prompt is:

[0966] User: "I want to watch the latest movies"

[0967] System: "Show the latest movies. Searching..."

[0968] In this way, users can smoothly perform desired operations using only voice utterances. Furthermore, by incorporating emotion analysis means, it is possible to provide appropriate assistance according to the user's emotional state, resulting in a more user-friendly system.

[0969] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0970] Step 1:

[0971] Start voice recognition

[0972] Input: User says "I want to watch the latest movies."

[0973] The server uses the built-in microphone to capture the user's voice, and converts this voice data into text data using the speech_recognition library.

[0974] Output: The converted text data, e.g. "I want to watch the latest movies."

[0975] Step 2:

[0976] Text data analysis

[0977] Input: Text data generated in step 1

[0978] The server uses Hugging Face's transformers library to send the text data to a natural language processing engine to analyze the user's intent, which is "Show me the latest movies."

[0979] Output: User intent ("Show me the latest movies").

[0980] Step 3:

[0981] Performing sentiment analysis

[0982] Input: Text data generated in step 1

[0983] The server then performs emotion analysis using the Hugging Face transformers library, again to determine the user's emotional state, such as whether they are anxious or stressed.

[0984] Output: User's emotional information (e.g., feeling anxious).

[0985] Step 4:

[0986] Generate operation guides

[0987] Input: User intent identified in step 2, user sentiment information obtained in step 3

[0988] The server generates relevant instructional steps based on the user's intention and emotional state. For example, if the user is impatient, it generates detailed instructions ("Tap the movie icon in the bottom right of the screen").

[0989] Output: Detailed operation guide.

[0990] Step 5:

[0991] Display in the user interface

[0992] Input: The operation guide generated in step 4

[0993] The terminal displays the generated operation guide as an overlay on the user interface, for example, by displaying a cursor and a message saying "Press here" on a movie list icon.

[0994] Output: A visual guide on the user interface.

[0995] Step 6:

[0996] User Action Tracking

[0997] Input: User action (user taps on the movie icon)

[0998] The device tracks the user's actions in real time, confirms whether the user performed the action correctly as intended, and takes the next step if necessary.

[0999] Output: User behavior data.

[1000] Step 7:

[1001] Providing a guide to remediation

[1002] Input: User behavior data obtained in step 6

[1003] If the user makes an incorrect operation, the server uses a correction guide generation means to provide appropriate correction information. For example, if the user accidentally taps a different icon, the server displays a message saying "You made a mistake. Please open the movie list" and instructs the user to perform the correct operation again.

[1004] Output: Correction guide and new operating instructions.

[1005] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1006] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1007] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1008] [Third embodiment]

[1009] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1010] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1011] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1012] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1013] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1014] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1015] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1016] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1017] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1018] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1019] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1020] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1021] The present invention provides a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit. The program processing of this system will be explained below in natural language.

[1022] 1. Voice Recognition Method

[1023] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and receives it as audio data. The speech recognition module then converts the audio data into text data. This process passes the user's instructions in text form to the system.

[1024] 2. Natural Language Processing Methods

[1025] The device then sends this text data to a natural language processing engine, which analyzes the text and extracts the user's intent, in this case "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[1026] 3. Operation guide generation method

[1027] Based on the analyzed intent, the device will generate a guide with relevant operation steps. For example, the first step is to open the YouTube app, so the device will locate the YouTube app icon on the home screen. Next, it will find the search bar and provide specific instructions for typing "cat."

[1028] 4. User Interface Display Method

[1029] The procedure generated by the operation guide generating means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this display continues until the user completes the tap operation.

[1030] 5. User Action Tracking Methods

[1031] The device tracks user behavior in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar and monitoring the user's actions when typing "cat."

[1032] 6. Correction guide generation method

[1033] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly.

[1034] Specific examples

[1035] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[1036] 1. The user speaks.

[1037] 2. The device converts the voice into text and analyzes it using an NLP engine.

[1038] 3. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video).

[1039] 4. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[1040] 5. The user follows the instructions and taps the YouTube icon.

[1041] 6. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[1042] 7. User types "cat" and performs a search.

[1043] 8. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[1044] 9. User taps on the video to start playing.

[1045] In this way, the user can smoothly perform the desired operation by simply speaking. The present invention is a system that is particularly useful for users who are not familiar with technology and elderly people.

[1046] The processing flow will be explained below.

[1047] Step 1:

[1048] The user speaks into their smartphone, "I want to watch cat videos on YouTube."

[1049] Step 2:

[1050] The terminal uses a built-in microphone to capture the user's voice and receives it as voice data.

[1051] Step 3:

[1052] The terminal passes the received voice data to a voice recognition module and converts it into text data.

[1053] Step 4:

[1054] The device sends the converted text data to a natural language processing engine.

[1055] Step 5:

[1056] The device uses a natural language processing engine to analyze the text data and extract the user's intent, such as "open YouTube," "type 'cat' into the search bar," or "select a video from the search results."

[1057] Step 6:

[1058] The device prepares an operation procedure to generate an operation guide based on the analyzed intent. For example, first, it collects location information to identify the YouTube app icon.

[1059] Step 7:

[1060] The device will generate an operation guide and display a cursor pointing to the YouTube app icon and a guide saying "Press here" on the screen.

[1061] Step 8:

[1062] The user taps the YouTube icon.

[1063] Step 9:

[1064] The device detects the user's operation and confirms that the YouTube app has been launched correctly.

[1065] Step 10:

[1066] The device will identify the search bar within the YouTube app and display a cursor pointing to it with the prompt "Type 'cat' here."

[1067] Step 11:

[1068] A user types "cat" into the search bar and presses Enter.

[1069] Step 12:

[1070] The device detects whether the user's input was correct and verifies that the search results are displayed.

[1071] Step 13:

[1072] The device will locate the first video in the search results and display a cursor pointing to the video and a prompt saying "Play this."

[1073] Step 14:

[1074] The user taps on the first video and it begins playing.

[1075] Step 15:

[1076] The terminal confirms the completion of the user operation and ends the guide display.

[1077] Through the above steps, the system allows users to easily and reliably perform desired operations on their smartphones using voice utterances.

[1078] Example 1

[1079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1080] Conventional voice-operated systems have difficulty accurately recognizing user utterances and analyzing their intentions, making it difficult to effectively support operations, especially for users who are unfamiliar with technology or the elderly. Furthermore, there is a lack of means to provide appropriate correction guidance in the event of an incorrect operation, which means it takes time to correct the incorrect operation, reducing usability.

[1081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1082] In this invention, the server includes a voice recognition means, a natural language processing means, and an operation guide generation means, which make it possible to capture a user's voice instructions, convert them into text data, analyze the text data to extract the user's intention, and generate and display operation instructions based on the analyzed intention.

[1083] "Speech recognition means" is a technology that captures a user's speech as voice data and converts it into text data.

[1084] "Natural language processing means" is a technology that analyzes and extracts user intent from text data.

[1085] The "operation guide generation means" is a technology that generates an operation procedure to be presented to the user based on the analyzed intention.

[1086] The "user interface display means" is a technique for visually displaying the generated operation guide to the user.

[1087] "User operation tracking means" is a technology that monitors and tracks the operations performed by a user in real time.

[1088] The "correction guide generating means" is a technology that detects an incorrect operation made by a user and generates a guide that instructs the user on the correct operation.

[1089] "Text data" refers to character data converted by a speech recognition means.

[1090] "Intent analysis" refers to the process of interpreting user intent from text data using natural language processing tools.

[1091] "Operation instructions" refer to instructions that present the user with specific operations that should be performed based on the user's intentions.

[1092] An "incorrect operation" is when a user performs an operation that differs from the operation instructions presented by the system.

[1093] "Correction information" refers to specific instruction information for correcting an incorrect operation and performing the correct operation.

[1094] The present invention relates to a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit, and the system aims to capture a user's voice instructions, convert them into text data, analyze their intentions, and present the user with the necessary operation procedures.

[1095] 1. Voice Recognition Method

[1096] A user speaks into their smartphone, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and record it as audio data. The device's speech recognition module (e.g., Google Cloud Speech-to-Text API) then converts the audio data into text data. This process passes the user's instructions to the system in text format.

[1097] 2. Natural Language Processing Methods

[1098] The text data obtained by the speech recognition module is sent to a natural language processing engine (e.g., OpenAI GPT-3). The NLP engine analyzes the text and extracts the user's intent. In this case, the intent is identified as "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[1099] 3. Operation guide generation method

[1100] Based on the analyzed user intent, the operation guide generator generates relevant operation steps. For example, the first step is to open the YouTube app, so the device identifies the location of the YouTube app icon on the home screen. Next, it provides specific steps for finding the search bar and entering "cat."

[1101] 4. User Interface Display Method

[1102] The procedure generated by the operation guide generating means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this continues until the user completes the tap operation.

[1103] 5. User Action Tracking Methods

[1104] The device tracks user behavior in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar, and even monitoring whether the user is typing "cat" correctly.

[1105] 6. Correction guide generation method

[1106] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly.

[1107] Specific examples

[1108] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[1109] 1. The user says, "I want to watch cat videos on YouTube."

[1110] 2. The device converts the voice into text and analyzes it using an NLP engine.

[1111] 3. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video).

[1112] 4. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[1113] 5. The user follows the instructions and taps the YouTube icon.

[1114] 6. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[1115] 7. User types "cat" and performs a search.

[1116] 8. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[1117] 9. User taps on the video to start playing.

[1118] In this way, the user can smoothly perform the desired operation using only voice utterances. This system is particularly useful for users who are not familiar with technology or the elderly. As an example prompt sentence, the following sentence is input to the generative AI model:

[1119] "If a user says they want to watch cat videos on YouTube, explain how the system generates, displays, and supports the user with navigation."

[1120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1121] Step 1:

[1122] A user says, "I want to watch cat videos on YouTube."

[1123] Input: User utterance.

[1124] Output: The captured audio data.

[1125] How it works: A user verbally types into their smartphone, "I want to watch cat videos on YouTube."

[1126] Step 2:

[1127] The device captures the user's speech using a built-in microphone.

[1128] Input: User utterance.

[1129] Output: Audio data.

[1130] How it works: The smartphone's built-in microphone records the user's speech as audio data.

[1131] Step 3:

[1132] The device's voice recognition module converts the voice data into text data.

[1133] Input: Audio data.

[1134] Output: Character data.

[1135] What happens: A speech recognition module (e.g., Google Cloud Speech-to-Text API) analyzes the captured audio data and converts it into corresponding text data ("I want to watch cat videos on YouTube").

[1136] Step 4:

[1137] The device sends the text data to a natural language processing engine.

[1138] Input: Character data.

[1139] Output: User intent.

[1140] How it works: The converted text data is sent to a natural language processing engine (e.g., OpenAI GPT-3) to prepare it for analyzing the user's intent.

[1141] Step 5:

[1142] A natural language processing engine analyzes the text data and extracts the user's intent.

[1143] Input: Character data.

[1144] Output: User intent (e.g., "Open YouTube," "Type 'cat' into the search bar," "Select a video from the search results").

[1145] How it works: A natural language processing engine parses the text data it receives and identifies the user's request ("open YouTube," "type 'cat' in the search bar," "select a video from the search results").

[1146] Step 6:

[1147] The terminal generates operating procedures based on the analyzed intention.

[1148] Input: User intent.

[1149] Output: Operation instructions.

[1150] Action: The device constructs the necessary steps based on the user's intent (e.g., the steps to open the YouTube app).

[1151] Step 7:

[1152] Your device will locate the YouTube app icon on your home screen.

[1153] Input: Intention-based operating instructions.

[1154] Output: YouTube icon position.

[1155] What it does: Your device uses a digital map on your home screen to locate the YouTube icon.

[1156] Step 8:

[1157] The device overlays a guide on the user interface to guide you through the operation process.

[1158] Input: The location of the identified icon.

[1159] Output: On-screen cursor and guide message.

[1160] Behavior: The device displays a cursor over the YouTube icon and the message "Press here" on the screen until the user taps it.

[1161] Step 9:

[1162] The device tracks the user's operating behavior in real time.

[1163] Input: User taps.

[1164] Output: Confirmation that the operation was successful.

[1165] What it does: The device monitors and checks if the user has successfully launched the YouTube app.

[1166] Step 10:

[1167] Your device will locate the search bar and then guide you through typing "cat."

[1168] Input: Confirmation information for operation completion.

[1169] Output: Search bar position and guide message.

[1170] Behavior: The device displays a cursor pointing at the search bar and the message "Type 'cat' here" until the user types something.

[1171] Step 11:

[1172] A user types in "cat" and performs a search.

[1173] Input: Guide message.

[1174] Output: The search term entered.

[1175] What happens: A user types "cat" into the search bar and presses the search button.

[1176] Step 12:

[1177] The device will identify the appropriate video from the search results and display a guide pointing to it.

[1178] Input: The search term entered.

[1179] Output: Video position and guide message.

[1180] What it does: The device displays a cursor pointing to the appropriate video in the search results and the message "Play this" until the user makes a selection.

[1181] Step 13:

[1182] The user taps on the video to begin playing it.

[1183] Input: Guide message.

[1184] Output: Video begins playing.

[1185] How it works: The user is guided to tap on a video and it begins playing.

[1186] Through this series of steps, the user can smoothly perform the desired operation using only voice utterances.

[1187] (Application example 1)

[1188] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1189] With conventional store guidance systems, it was difficult for staff to grasp their physical location within the store and provide appropriate guidance according to customer requests. Furthermore, in order for staff to accurately guide customers in real time, they had to process a large amount of information in a short amount of time, which placed a heavy burden on staff. Furthermore, it was difficult to provide prompt corrective instructions when a customer was in the wrong place or made a mistake.

[1190] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1191] In this invention, the server includes a voice recognition means, a natural language processing means, an operation guide generation means, a user interface display means, a user operation tracking means, a correction guide generation means, a means for tracking position information in physical space, and a means for overlaying and displaying information in the user's field of view, thereby enabling store guidance in response to customer requests in real time.

[1192] A "voice recognition means" is a device or software module that converts voice data into text data.

[1193] A "natural language processing means" is a device or software module that analyzes user intent from text data.

[1194] The "operation guide generation means" is a device or software module that generates an operation procedure based on the analyzed intention.

[1195] The "user interface display means" is a device or software module that displays operation guides on a screen or display.

[1196] A "user operation tracking means" is a device or software module that tracks a user's operation behavior in real time.

[1197] The "correction guide generating means" is a device or software module that generates correction instructions when the user performs an incorrect operation.

[1198] A "physical location tracking device" is a device or software module that tracks a user's physical location using GPS or other location technology.

[1199] "Means for overlaying and displaying information in the user's field of view" refers to a device or software module that displays information by overlaying it in the user's field of view using smart glasses, a head-mounted display, or the like.

[1200] The present invention relates to a system that provides real-time store guidance in response to customer requests. The system combines the functions of voice recognition, natural language processing, operation guidance, user interface display, user operation tracking, correction guidance, location information tracking, and overlay display.

[1201] This system is configured as follows:

[1202] Voice recognition means

[1203] The server converts the voice received by the staff through devices such as smart glasses into text data using voice recognition software (e.g., Google Speech-to-Text API).

[1204] Natural language processing tools

[1205] The text data is analyzed by a server-side natural language processing engine (e.g., Google Cloud Natural Language API or IBM Watson) to extract the user's intent. For example, if a customer asks, "Where is the wine section?", the intent is extracted as "Find the location of the wine section."

[1206] Operation guide generation means

[1207] Based on the analyzed intent, the server generates a guide for the relevant operating procedures. Specifically, it refers to the store's map data to calculate the optimal route and generates data for displaying the guide.

[1208] User interface display means

[1209] The procedures generated by the operation guide generation means are overlaid on the display of the smart glasses worn by the staff member, using a user interface framework such as Unity.

[1210] User action tracking

[1211] The server tracks the physical location of staff in real time using GPS sensors and other location technologies, ensuring they are heading in the right direction.

[1212] Correction guide generation method

[1213] If the user is heading in the wrong direction or making a mistake, the server will instantly generate corrective instructions and display them on the smart glasses' display, such as "Turn left at the next corner."

[1214] Specific examples

[1215] As a concrete example, if a customer asks, "Where is the wine section?", the sequence of events is as follows:

[1216] 1. Staff wear smart glasses and capture customer questions using voice recognition.

[1217] 2. The server converts the speech to text.

[1218] 3. The NLP engine analyzes the question and extracts the intent: "Find a wine section."

[1219] 4. The store map data is referenced to generate a route guide, and the smart glasses display shows "Turn right from here."

[1220] 5. The server uses GPS data to track the staff's location and ensure they are heading in the right direction.

[1221] 6. If you are heading to the wrong location, the server will display a corrective instruction such as "Turn left at the next corner."

[1222] Prompt Sentence Examples

[1223] Enter the following prompts into the AI ​​model to enable it to generate the appropriate program:

[1224] Based on the user's voice question, your goal is to generate a program that uses an NLP engine to analyze the intent and provide appropriate product information and floor guides in real time. For example, if a customer asks, "Where is the wine section?", please explain the following steps:

[1225] 1. Voice Recognition

[1226] 2. NLP analysis

[1227] 3. Guide Generation

[1228] 4. UI display

[1229] 5. Operation Tracking

[1230] 6. Guide correction

[1231] Please also include specific examples of the hardware and software you will use.

[1232] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1233] Step 1:

[1234] Audio Capture and Conversion

[1235] Input: The user speaks a voice question into the smart glasses.

[1236] Specific operation: The staff activates the voice input function of the smart glasses and captures the question. The smart glasses use the built-in microphone to capture the voice data.

[1237] Data processing and calculation: A speech recognition module (e.g., Google Speech-to-Text API) converts the voice data into text data.

[1238] Output: Text data (e.g., "Where is the wine store?")

[1239] Step 2:

[1240] Text analysis and intent extraction

[1241] Input: Text data sent from smart glasses

[1242] Specific operation: The server receives the text data and sends it to a natural language processing engine (e.g., Google Cloud Natural Language API).

[1243] Data processing and calculation: A natural language processing engine analyzes text data and extracts user intent.

[1244] Output: Intent data (e.g., "Find a wine section")

[1245] Step 3:

[1246] Operation guide generation

[1247] Input: Intent data

[1248] Specific operation: Based on the intent data, the server references the location information within the store and calculates the appropriate guidance route.

[1249] Data processing and calculation: Taking information from the store's map database and applying algorithms to calculate the optimal route.

[1250] Output: Guide data (e.g. "Turn right from here")

[1251] Step 4:

[1252] User Interface Display

[1253] Input: Guide data

[1254] Specific operation: The server sends guide data to the smart glasses, which overlay it on the user's field of view. The smart glasses then display arrows and guide messages on the display.

[1255] Data processing and computation: Converting guide data into visual information using a user interface framework (e.g., Unity).

[1256] Output: Visual data (e.g., "Please turn right")

[1257] Step 5:

[1258] User Action Tracking

[1259] Input: Location data from smart glasses

[1260] What it does: The server tracks the physical location of staff in real time using GPS sensors and other location technologies.

[1261] Data processing and calculation: Analyze the acquired location data and check whether the staff is following the correct route.

[1262] Output: Tracking data (e.g. flags indicating whether you are moving in the right direction)

[1263] Step 6:

[1264] Correction guide generation

[1265] Input: Tracking data

[1266] Specific behavior: The server uses tracking data to generate corrective instructions if a staff member is taking the wrong route.

[1267] Data processing and calculation: Applying algorithms to recalculate guide data and generate corrective instructions.

[1268] Output: Corrective guidance data (e.g. "Turn left at the next corner")

[1269] This allows staff to provide appropriate store guidance in real time.

[1270] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1271] The present invention provides a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion engine. The program processing of this system will be explained below in natural language.

[1272] 1. Voice Recognition Method

[1273] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and receives it as audio data. The speech recognition module then converts the audio data into text data. This process passes the user's instructions in text form to the system.

[1274] 2. Natural Language Processing Methods

[1275] The device then sends this text data to a natural language processing engine, which analyzes the text and extracts the user's intent, in this case "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[1276] 3. Emotion Engine

[1277] In addition, the device is equipped with an emotion engine that recognizes the user's emotions from speech recognition. For example, if the user is anxious, the engine will detect "stress" from the tone and speed of the speech. This emotion information is used to generate and display subsequent operation guides.

[1278] 4. Operation guide generation method

[1279] The device generates a guide with relevant operation steps based on the analyzed intent and recognized emotion. For example, the first step is to open the YouTube app, so the device locates the YouTube app icon on the home screen. Next, it provides specific steps for finding the search bar and typing "cat." If the user is stressed, the device generates a more detailed guide than usual.

[1280] 5. User Interface Display Method

[1281] The procedure generated by the operation guide generation means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this display continues until the user completes the tap operation. The display method and level of detail of the guide are adjusted according to the user's emotional state.

[1282] 6. User Action Tracking Methods

[1283] The device tracks the user's actions in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar and monitoring the user's actions when typing "cat."

[1284] 7. Correction guide generation method

[1285] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly. If the user is feeling stressed, the means displays more detailed correction guides.

[1286] Specific examples

[1287] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[1288] 1. The user speaks.

[1289] 2. The device converts the voice into text and analyzes it using an NLP engine.

[1290] 3. The device analyzes the user's emotions using an emotion engine. For example, it determines that the user is anxious.

[1291] 4. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video). If the user is in a hurry, more detailed steps are generated.

[1292] 5. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[1293] 6. The user follows the instructions and taps the YouTube icon.

[1294] 7. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[1295] 8. User types "cat" and performs a search.

[1296] 9. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[1297] 10. User taps on the video to start playing.

[1298] In this way, users can smoothly perform desired operations using only voice utterances. Furthermore, by incorporating an emotion engine, it is possible to provide appropriate assistance according to the user's emotional state, resulting in a more user-friendly system.

[1299] The processing flow will be explained below.

[1300] Step 1:

[1301] The user speaks into their smartphone, "I want to watch cat videos on YouTube."

[1302] Step 2:

[1303] The terminal uses a built-in microphone to capture the user's voice and receives it as voice data.

[1304] Step 3:

[1305] The terminal passes the received voice data to a voice recognition module and converts it into text data.

[1306] Step 4:

[1307] The device sends the converted text data to a natural language processing engine.

[1308] Step 5:

[1309] The device uses a natural language processing engine to analyze the text data and extract the user's intent. The intents identified are "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[1310] Step 6:

[1311] The device sends the text data to an emotion engine, which analyzes the user's emotions. For example, it can determine that the user is anxious based on the tone and speed of the voice.

[1312] Step 7:

[1313] The device generates operation guidance based on the analyzed intention and the user's emotions. If the user is in a hurry, the operation guidance will include more detailed steps than usual.

[1314] Step 8:

[1315] Your device will locate the YouTube app icon on your home screen.

[1316] Step 9:

[1317] The device will display a cursor pointing to the YouTube app icon and a guide message saying "Press here."

[1318] Step 10:

[1319] The user taps the YouTube icon.

[1320] Step 11:

[1321] Make sure the YouTube app is launched correctly on your device.

[1322] Step 12:

[1323] The device identifies the search bar within the YouTube app.

[1324] Step 13:

[1325] The device displays a cursor pointing to the search bar and a guide message saying "Type 'cat' here."

[1326] Step 14:

[1327] A user types "cat" into the search bar and presses Enter.

[1328] Step 15:

[1329] The device confirms the user's input and verifies that search results are displayed.

[1330] Step 16:

[1331] The device locates the first video in the search results.

[1332] Step 17:

[1333] The device will display a cursor pointing to the video and a guide message saying "Play this."

[1334] Step 18:

[1335] The user taps on the first video and it begins playing.

[1336] Step 19:

[1337] The terminal confirms the completion of the user operation and ends the guide display.

[1338] Example 2

[1339] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1340] Conventional systems have difficulty accurately analyzing the user's intention when giving voice instructions, and the operation guidance provided does not take into account the user's emotional state. As a result, users often feel stressed or make mistakes while operating the device. Furthermore, if a user makes a mistake, no appropriate correction guidance is provided, forcing the user to manually correct the mistake, resulting in an insufficient operation experience.

[1341] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1342] In this invention, the server includes a voice recognition unit, a natural language processing unit, an emotion recognition unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit. This makes it possible to accurately analyze the user's voice instructions and provide detailed operation guides based on the user's emotional state. Furthermore, by quickly presenting correction guides when the user makes an operation error, the operation experience can be made smoother and stress-free.

[1343] A "voice recognition means" is a means for capturing a user's speech, saving it as voice data, and then converting it into text data.

[1344] "Natural language processing means" is a means for analyzing text data and extracting the user's intent.

[1345] An "emotion recognition means" is a means for analyzing a user's emotional state from the user's speech or text data.

[1346] The "operation guide generation means" is a means for generating operation procedures based on the user's intentions and emotional state, and providing the user with appropriate guidance.

[1347] The "user interface display means" is a means for overlaying and displaying the generated operation guide on the screen.

[1348] The "user operation tracking means" is a means for tracking the user's operation behavior in real time and monitoring whether the operation has been performed correctly.

[1349] The "correction guide generating means" is a means for providing appropriate correction information when the user makes an incorrect operation, and urging the user to perform the correct operation again.

[1350] The present invention provides a system for enabling a user to efficiently achieve a desired operation through voice instructions. A specific embodiment of the system will be described below.

[1351] Program processing

[1352] Voice recognition means

[1353] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and save it as audio data. The device then converts the speech data into text using the Google Cloud Speech-to-Text API. This process passes the user's instructions to the system in text format.

[1354] Natural language processing tools

[1355] The device sends the acquired text data to the Google Cloud Natural Language API, which analyzes the text and extracts the user's intent. In this case, it identifies the user's intent, such as "open YouTube," "enter 'cat' in the search bar," and "select a video from the search results."

[1356] emotion recognition means

[1357] The device uses IBM Watson Tone Analyzer to recognize the user's emotions from speech recognition and text data. For example, if the user is feeling anxious, the device can detect "stress" from the tone and speed of the voice. The recognized emotional information is reflected in the generation of the next operation guide.

[1358] Operation guide generation means

[1359] The device generates relevant instructions based on the analyzed intent and emotional state. Using ARKit or ARCore, the device identifies the YouTube app icon on the user's smartphone screen, then creates detailed instructions for finding the search bar and typing "cat." For example, if the user is feeling stressed, the device generates more detailed instructions than usual.

[1360] User interface display means

[1361] The generated operation guide is overlaid on the screen using Unity or Flutter. For example, a message saying "Press here" is displayed on top of the YouTube icon, and remains displayed until the user taps it. The display method and level of detail of the guide are adjusted depending on the emotion.

[1362] User action tracking

[1363] The device uses Firebase Analytics to track user behavior in real time, for example, to ensure that a user successfully opens the YouTube app and then types "cat" into the search bar.

[1364] Correction guide generation method

[1365] If the user makes an incorrect operation, the device generates a correction guide. For example, if the user accidentally taps the wrong icon, the device will provide a correction message saying, "You made a mistake. Please open YouTube." The device will then provide a guide pointing back to the YouTube icon. The correction guide will be adjusted to be more detailed depending on the user's emotional state.

[1366] Specific examples

[1367] As a concrete example, the following sequence of events occurs when a user says, "I want to watch cat videos on YouTube."

[1368] 1. The user speaks.

[1369] 2. The device converts the voice into text and analyzes it using a natural language processing engine.

[1370] 3. The device analyzes the user's emotions using an emotion recognition engine. For example, it determines that the user is anxious.

[1371] 4. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video). If the user is in a hurry, more detailed steps are generated.

[1372] 5. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[1373] 6. The user follows the instructions and taps the YouTube icon.

[1374] 7. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[1375] 8. User types "cat" and performs a search.

[1376] 9. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[1377] 10. User taps on the video to start playing.

[1378] Prompt Sentence Examples

[1379] An example of a prompt is as follows:

[1380] "I want to watch cat videos on YouTube. Convert the audio data into text, analyze the user's intent and emotions, and generate detailed operating instructions."

[1381] The above is a specific embodiment for carrying out the present invention. This system allows users to efficiently perform desired operations using only voice input, and by incorporating emotion recognition, it is possible to provide optimal support according to the user's condition.

[1382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1383] Program processing flow

[1384] Step 1:

[1385] Audio capture and transcription

[1386] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and save it as audio data. The device then sends this audio data to the Google Cloud Speech-to-Text API, which converts it into text.

[1387] Input: Audio data

[1388] Output: Text data ("I want to watch cat videos on YouTube")

[1389] Step 2:

[1390] Intention analysis of text data

[1391] The device sends the acquired text data to the Google Cloud Natural Language API. This API analyzes the input text data and extracts the user's intent. For example, the analysis results may lead to operational steps such as "open the YouTube app," "enter 'cat' in the search bar," and "select a video from the search results."

[1392] Input: Text data

[1393] Output: User intent (list of action steps)

[1394] Step 3:

[1395] Emotion recognition

[1396] The device uses IBM Watson Tone Analyzer to analyze the user's emotions from voice and text data. For example, it can detect "stress" from the tone and speed of the user's voice. This emotional information is reflected in the generation of subsequent operation guides.

[1397] Input: Audio data, text data

[1398] Output: Emotion data (e.g., stress)

[1399] Step 4:

[1400] Generate operation guides

[1401] Based on the analyzed intent and emotional state, the device uses ARKit and ARCore to generate operation instructions, specifically, to identify the YouTube app icon on the user's smartphone screen, then create detailed instructions for finding the search bar and typing "cat."

[1402] Input: User intent, emotional data

[1403] Output: Operation Guide (detailed instructions)

[1404] Step 5:

[1405] View Guide

[1406] The device uses Unity or Flutter to overlay the generated operation guide on the screen. Specifically, a message saying "Press here" appears above the YouTube icon and remains displayed until the user taps it. The display method and level of detail of the guide are adjusted depending on the emotion.

[1407] Input: Operation Guide

[1408] Output: Guide display on the user interface

[1409] Step 6:

[1410] Tracking user actions

[1411] The device uses Firebase Analytics to track user behavior in real time, for example, whether a user successfully opens the YouTube app or types "cat" into the search bar.

[1412] Input: User operation data

[1413] Output: Operation tracking data (flag indicating whether the operation was successful)

[1414] Step 7:

[1415] Generate repair guides

[1416] If the user makes a mistake, the device generates a correction guide. For example, if the user accidentally taps a different icon, the device will notify the user, "You made a mistake. Please open YouTube," and provide a guide pointing back to the YouTube icon.

[1417] Input: Operation tracking data

[1418] Output: Fix guide

[1419] By performing these steps consecutively, users receive detailed and appropriate operation guidance through voice alone, enabling them to smoothly accomplish their desired operations. Furthermore, guidance is adjusted based on the user's emotional state, providing a more user-friendly experience.

[1420] (Application example 2)

[1421] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1422] In conventional systems, even when users issue voice instructions, it can be difficult for them to accurately understand and operate the system. Furthermore, if the user makes an incorrect operation, no appropriate guidance is provided, which can lead to operational delays and increased stress and frustration for the user. Furthermore, conventional systems do not take into account the user's emotional state when providing operation guidance, resulting in a lack of a user-friendly interface. This problem, which reduces the quality of the user experience, needs to be addressed.

[1423] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1424] In this invention, the server includes a voice recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion analysis unit. This allows the server to accurately understand the user's voice instructions, provide appropriate operation procedures, and provide customized support according to the user's emotional state. Furthermore, by providing correction guides in real time for user operation errors, the server allows the user to complete operations without stress.

[1425] "Speech recognition means" is a technology that captures a user's speech as voice data and converts it into text data.

[1426] "Natural language processing means" is a technology that analyzes the user's intentions from the character data converted by the voice recognition means.

[1427] The "operation guide generation means" is a technology that generates related operation procedures based on the analyzed user's intentions.

[1428] The "user interface display means" is a technique for visually displaying to the user the procedures generated by the operation guide generation means.

[1429] "User operation tracking means" is a technology that tracks user operation behavior in real time and confirms that the operation is being performed accurately.

[1430] The "correction guide generating means" is a technology that provides appropriate correction information when the user performs an incorrect operation.

[1431] "Emotion analysis means" is a technology that analyzes emotions from the content of a user's speech and determines their emotional state.

[1432] The present invention provides a system including a voice recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion analysis unit. The system aims to accurately understand a user's voice instructions, display appropriate operation guides according to the user's emotional state, and enable the user to smoothly perform the intended operations.

[1433] 1. Voice Recognition Method

[1434] First, the user speaks into the device. For example, if the user says, "I want to watch the latest movie," the device uses the built-in microphone to capture this speech and receives it as voice data. The voice recognition module then converts the voice data into text data. Through this process, the user's instructions are passed on to the system in text format. The speech_recognition library is used for this voice recognition.

[1435] 2. Natural Language Processing Methods

[1436] The device then sends this text data to a natural language processing engine, which uses Hugging Face's transformers library to analyze the user's intent, in this case "Show me the latest movies."

[1437] 3. Emotion analysis means

[1438] The device is also equipped with an emotion analysis unit that analyzes the user's emotions from speech recognition. The prompt text is used to input the emotion model, which determines whether the user is feeling anxious or stressed. This emotion information is then used to generate and display subsequent operation guides. The emotion analysis also utilizes the emotion analysis model from Hugging Face's transformers library.

[1439] 4. Operation guide generation method

[1440] The device generates a guide with relevant operation steps based on the analyzed intent and the recognized emotion. For example, the first step is "Display movie list." If the device recognizes that the user is impatient, the operation guide generates more detailed instructions than usual, allowing the user to proceed to the next step without hesitation.

[1441] 5. User Interface Display Method

[1442] The procedure generated by the operation guide generation means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the movie list icon, and this display continues until the user completes the tap operation. The display method and level of detail of the guide are adjusted according to the user's emotional state.

[1443] 6. User Action Tracking Methods

[1444] The device tracks the user's actions in real time, ensuring that the user correctly opens the movie list as intended, and then monitors other actions.

[1445] 7. Correction guide generation method

[1446] If the user does not perform the operation correctly, the correction guide generation means will provide appropriate correction information. For example, if the user accidentally taps a different icon, the means will notify the user, "You made a mistake. Please open the movie list." and display a guide pointing back to the movie list icon. If the user enters an incomplete search bar entry, the means will instruct the user, "Please enter 'movies' correctly." If the user is feeling stressed, the means will display more detailed correction guides.

[1447] Specific examples

[1448] The following sequence of events occurs when a user says, "I want to watch the latest movies."

[1449] 1. The user speaks.

[1450] 2. The device converts the voice into text and analyzes it using a natural language processing engine.

[1451] 3. The device analyzes the user's emotions using emotion analysis means. For example, it determines that the user is anxious.

[1452] 4. The device generates the necessary instructions (show movie list). If the user is impatient, more detailed instructions are generated.

[1453] 5. The device will display a cursor pointing at the movie list icon and a prompt saying "Press here."

[1454] 6. The user follows the instructions and taps the movie list icon.

[1455] 7. Your device will display a cursor pointing to the search bar and a prompt saying "Type 'Movies' here."

[1456] 8. User types "movies" and performs a search.

[1457] 9. The device will display a cursor pointing to the appropriate movie and a prompt saying "Play this."

[1458] 10. The user taps on a movie to start playing it.

[1459] An example prompt is:

[1460] User: "I want to watch the latest movies"

[1461] System: "Show the latest movies. Searching..."

[1462] In this way, users can smoothly perform desired operations using only voice utterances. Furthermore, by incorporating emotion analysis means, it is possible to provide appropriate assistance according to the user's emotional state, resulting in a more user-friendly system.

[1463] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1464] Step 1:

[1465] Start voice recognition

[1466] Input: User says "I want to watch the latest movies."

[1467] The server uses the built-in microphone to capture the user's voice, and converts this voice data into text data using the speech_recognition library.

[1468] Output: The converted text data, e.g. "I want to watch the latest movies."

[1469] Step 2:

[1470] Text data analysis

[1471] Input: Text data generated in step 1

[1472] The server uses Hugging Face's transformers library to send the text data to a natural language processing engine to analyze the user's intent, which is "Show me the latest movies."

[1473] Output: User intent ("Show me the latest movies").

[1474] Step 3:

[1475] Performing sentiment analysis

[1476] Input: Text data generated in step 1

[1477] The server then performs emotion analysis using the Hugging Face transformers library, again to determine the user's emotional state, such as whether they are anxious or stressed.

[1478] Output: User's emotional information (e.g., feeling anxious).

[1479] Step 4:

[1480] Generate operation guides

[1481] Input: User intent identified in step 2, user sentiment information obtained in step 3

[1482] The server generates relevant instructional steps based on the user's intention and emotional state. For example, if the user is impatient, it generates detailed instructions ("Tap the movie icon in the bottom right of the screen").

[1483] Output: Detailed operation guide.

[1484] Step 5:

[1485] Display in the user interface

[1486] Input: The operation guide generated in step 4

[1487] The terminal displays the generated operation guide as an overlay on the user interface, for example, by displaying a cursor and a message saying "Press here" on a movie list icon.

[1488] Output: A visual guide on the user interface.

[1489] Step 6:

[1490] User Action Tracking

[1491] Input: User action (user taps on the movie icon)

[1492] The device tracks the user's actions in real time, confirms whether the user performed the action correctly as intended, and takes the next step if necessary.

[1493] Output: User behavior data.

[1494] Step 7:

[1495] Providing a guide to remediation

[1496] Input: User behavior data obtained in step 6

[1497] If the user makes an incorrect operation, the server uses a correction guide generation means to provide appropriate correction information. For example, if the user accidentally taps a different icon, the server displays a message saying "You made a mistake. Please open the movie list" and instructs the user to perform the correct operation again.

[1498] Output: Correction guide and new operating instructions.

[1499] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1500] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1501] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1502] [Fourth embodiment]

[1503] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1504] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1505] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1506] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1507] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1508] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1509] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1510] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1511] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1512] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1513] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1514] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1515] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1516] The present invention provides a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit. The program processing of this system will be explained below in natural language.

[1517] 1. Voice Recognition Method

[1518] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and receives it as audio data. The speech recognition module then converts the audio data into text data. This process passes the user's instructions in text form to the system.

[1519] 2. Natural Language Processing Methods

[1520] The device then sends this text data to a natural language processing engine, which analyzes the text and extracts the user's intent, in this case "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[1521] 3. Operation guide generation method

[1522] Based on the analyzed intent, the device will generate a guide with relevant operation steps. For example, the first step is to open the YouTube app, so the device will locate the YouTube app icon on the home screen. Next, it will find the search bar and provide specific instructions for typing "cat."

[1523] 4. User Interface Display Method

[1524] The procedure generated by the operation guide generating means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this display continues until the user completes the tap operation.

[1525] 5. User Action Tracking Methods

[1526] The device tracks the user's actions in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar and monitoring the user's actions when typing "cat."

[1527] 6. Correction guide generation method

[1528] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly.

[1529] Specific examples

[1530] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[1531] 1. The user speaks.

[1532] 2. The device converts the voice into text and analyzes it using an NLP engine.

[1533] 3. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video).

[1534] 4. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[1535] 5. The user follows the instructions and taps the YouTube icon.

[1536] 6. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[1537] 7. User types "cat" and performs a search.

[1538] 8. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[1539] 9. User taps on the video to start playing.

[1540] In this way, the user can smoothly perform the desired operation by simply speaking. The present invention is a system that is particularly useful for users who are not familiar with technology and elderly people.

[1541] The processing flow will be explained below.

[1542] Step 1:

[1543] The user speaks into their smartphone, "I want to watch cat videos on YouTube."

[1544] Step 2:

[1545] The terminal uses a built-in microphone to capture the user's voice and receives it as voice data.

[1546] Step 3:

[1547] The terminal passes the received voice data to a voice recognition module and converts it into text data.

[1548] Step 4:

[1549] The device sends the converted text data to a natural language processing engine.

[1550] Step 5:

[1551] The device uses a natural language processing engine to analyze the text data and extract the user's intent, such as "open YouTube," "type 'cat' into the search bar," or "select a video from the search results."

[1552] Step 6:

[1553] The device prepares an operation procedure to generate an operation guide based on the analyzed intent. For example, first, it collects location information to identify the YouTube app icon.

[1554] Step 7:

[1555] The device will generate an operation guide and display a cursor pointing to the YouTube app icon and a guide saying "Press here" on the screen.

[1556] Step 8:

[1557] The user taps the YouTube icon.

[1558] Step 9:

[1559] The device detects the user's operation and confirms that the YouTube app has been launched correctly.

[1560] Step 10:

[1561] The device will identify the search bar within the YouTube app and display a cursor pointing to it with the prompt "Type 'cat' here."

[1562] Step 11:

[1563] A user types "cat" into the search bar and presses Enter.

[1564] Step 12:

[1565] The device detects whether the user's input was correct and verifies that the search results are displayed.

[1566] Step 13:

[1567] The device will locate the first video in the search results and display a cursor pointing to the video and a prompt saying "Play this."

[1568] Step 14:

[1569] The user taps on the first video and it begins playing.

[1570] Step 15:

[1571] The terminal confirms the completion of the user operation and ends the guide display.

[1572] Through the above steps, the system allows users to easily and reliably perform desired operations on their smartphones using voice utterances.

[1573] Example 1

[1574] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1575] Conventional voice-operated systems have difficulty accurately recognizing user utterances and analyzing their intentions, making it difficult to effectively support operations, especially for users who are unfamiliar with technology or the elderly. Furthermore, there is a lack of means to provide appropriate correction guidance in the event of an incorrect operation, which means it takes time to correct the incorrect operation, reducing usability.

[1576] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1577] In this invention, the server includes a voice recognition means, a natural language processing means, and an operation guide generation means, which make it possible to capture a user's voice instructions, convert them into text data, analyze the text data to extract the user's intention, and generate and display operation instructions based on the analyzed intention.

[1578] "Speech recognition means" is a technology that captures a user's speech as voice data and converts it into text data.

[1579] "Natural language processing means" is a technology that analyzes and extracts user intent from text data.

[1580] The "operation guide generation means" is a technology that generates an operation procedure to be presented to the user based on the analyzed intention.

[1581] The "user interface display means" is a technique for visually displaying the generated operation guide to the user.

[1582] "User operation tracking means" is a technology that monitors and tracks the operations performed by a user in real time.

[1583] The "correction guide generating means" is a technology that detects an incorrect operation made by a user and generates a guide that instructs the user on the correct operation.

[1584] "Text data" refers to character data converted by a speech recognition means.

[1585] "Intent analysis" refers to the process of interpreting user intent from text data using natural language processing tools.

[1586] "Operation instructions" refer to instructions that present the user with specific operations that should be performed based on the user's intentions.

[1587] An "incorrect operation" is when a user performs an operation that differs from the operation instructions presented by the system.

[1588] "Correction information" refers to specific instruction information for correcting an incorrect operation and performing the correct operation.

[1589] The present invention relates to a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit, and the system aims to capture a user's voice instructions, convert them into text data, analyze their intentions, and present the user with the necessary operation procedures.

[1590] 1. Voice Recognition Method

[1591] A user speaks into their smartphone, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and record it as audio data. The device's speech recognition module (e.g., Google Cloud Speech-to-Text API) then converts the audio data into text data. This process passes the user's instructions to the system in text format.

[1592] 2. Natural Language Processing Methods

[1593] The text data obtained by the speech recognition module is sent to a natural language processing engine (e.g., OpenAI GPT-3). The NLP engine analyzes the text and extracts the user's intent. In this case, the intent is identified as "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[1594] 3. Operation guide generation method

[1595] Based on the analyzed user intent, the operation guide generator generates relevant operation steps. For example, the first step is to open the YouTube app, so the device identifies the location of the YouTube app icon on the home screen. Next, it provides specific steps for finding the search bar and entering "cat."

[1596] 4. User Interface Display Method

[1597] The procedure generated by the operation guide generating means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this continues until the user completes the tap operation.

[1598] 5. User Action Tracking Methods

[1599] The device tracks user behavior in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar, and even monitoring whether the user is typing "cat" correctly.

[1600] 6. Correction guide generation method

[1601] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly.

[1602] Specific examples

[1603] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[1604] 1. The user says, "I want to watch cat videos on YouTube."

[1605] 2. The device converts the voice into text and analyzes it using an NLP engine.

[1606] 3. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video).

[1607] 4. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[1608] 5. The user follows the instructions and taps the YouTube icon.

[1609] 6. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[1610] 7. User types "cat" and performs a search.

[1611] 8. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[1612] 9. User taps on the video to start playing.

[1613] In this way, the user can smoothly perform the desired operation using only voice utterances. This system is particularly useful for users who are not familiar with technology or the elderly. As an example prompt sentence, the following sentence is input to the generative AI model:

[1614] "If a user says they want to watch cat videos on YouTube, explain how the system generates, displays, and supports the user with navigation."

[1615] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1616] Step 1:

[1617] A user says, "I want to watch cat videos on YouTube."

[1618] Input: User utterance.

[1619] Output: The captured audio data.

[1620] How it works: A user verbally types into their smartphone, "I want to watch cat videos on YouTube."

[1621] Step 2:

[1622] The device captures the user's speech using a built-in microphone.

[1623] Input: User utterance.

[1624] Output: Audio data.

[1625] How it works: The smartphone's built-in microphone records the user's speech as audio data.

[1626] Step 3:

[1627] The device's voice recognition module converts the voice data into text data.

[1628] Input: Audio data.

[1629] Output: Character data.

[1630] What happens: A speech recognition module (e.g., Google Cloud Speech-to-Text API) analyzes the captured audio data and converts it into corresponding text data ("I want to watch cat videos on YouTube").

[1631] Step 4:

[1632] The device sends the text data to a natural language processing engine.

[1633] Input: Character data.

[1634] Output: User intent.

[1635] How it works: The converted text data is sent to a natural language processing engine (e.g., OpenAI GPT-3) to prepare it for analyzing the user's intent.

[1636] Step 5:

[1637] A natural language processing engine analyzes the text data and extracts the user's intent.

[1638] Input: Character data.

[1639] Output: User intent (e.g., "Open YouTube," "Type 'cat' into the search bar," "Select a video from the search results").

[1640] How it works: A natural language processing engine parses the text data it receives and identifies the user's request ("open YouTube," "type 'cat' in the search bar," "select a video from the search results").

[1641] Step 6:

[1642] The terminal generates operating procedures based on the analyzed intention.

[1643] Input: User intent.

[1644] Output: Operation instructions.

[1645] Action: The device constructs the necessary steps based on the user's intent (e.g., the steps to open the YouTube app).

[1646] Step 7:

[1647] Your device will locate the YouTube app icon on your home screen.

[1648] Input: Intention-based operating instructions.

[1649] Output: YouTube icon position.

[1650] What it does: Your device uses a digital map on your home screen to locate the YouTube icon.

[1651] Step 8:

[1652] The device overlays a guide on the user interface to guide you through the operation process.

[1653] Input: The location of the identified icon.

[1654] Output: On-screen cursor and guide message.

[1655] Behavior: The device displays a cursor over the YouTube icon and the message "Press here" on the screen until the user taps it.

[1656] Step 9:

[1657] The device tracks the user's operating behavior in real time.

[1658] Input: User taps.

[1659] Output: Confirmation that the operation was successful.

[1660] What it does: The device monitors and checks if the user has successfully launched the YouTube app.

[1661] Step 10:

[1662] Your device will locate the search bar and then guide you through typing "cat."

[1663] Input: Confirmation information for operation completion.

[1664] Output: Search bar position and guide message.

[1665] Behavior: The device displays a cursor pointing at the search bar and the message "Type 'cat' here" until the user types something.

[1666] Step 11:

[1667] A user types in "cat" and performs a search.

[1668] Input: Guide message.

[1669] Output: The search term entered.

[1670] What happens: A user types "cat" into the search bar and presses the search button.

[1671] Step 12:

[1672] The device will identify the appropriate video from the search results and display a guide pointing to it.

[1673] Input: The search term entered.

[1674] Output: Video position and guide message.

[1675] What it does: The device displays a cursor pointing to the appropriate video in the search results and the message "Play this" until the user makes a selection.

[1676] Step 13:

[1677] The user taps on the video to begin playing it.

[1678] Input: Guide message.

[1679] Output: Video begins playing.

[1680] How it works: The user is guided to tap on a video and it begins playing.

[1681] Through this series of steps, the user can smoothly perform the desired operation using only voice utterances.

[1682] (Application example 1)

[1683] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1684] With conventional store guidance systems, it was difficult for staff to grasp their physical location within the store and provide appropriate guidance according to customer requests. Furthermore, in order for staff to accurately guide customers in real time, they had to process a large amount of information in a short amount of time, which placed a heavy burden on staff. Furthermore, it was difficult to provide prompt corrective instructions when a customer was in the wrong place or made a mistake.

[1685] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1686] In this invention, the server includes a voice recognition means, a natural language processing means, an operation guide generation means, a user interface display means, a user operation tracking means, a correction guide generation means, a means for tracking position information in physical space, and a means for overlaying and displaying information in the user's field of view, thereby enabling store guidance in response to customer requests in real time.

[1687] A "voice recognition means" is a device or software module that converts voice data into text data.

[1688] A "natural language processing means" is a device or software module that analyzes user intent from text data.

[1689] The "operation guide generation means" is a device or software module that generates an operation procedure based on the analyzed intention.

[1690] The "user interface display means" is a device or software module that displays operation guides on a screen or display.

[1691] A "user operation tracking means" is a device or software module that tracks a user's operation behavior in real time.

[1692] The "correction guide generating means" is a device or software module that generates correction instructions when the user performs an incorrect operation.

[1693] A "physical location tracking device" is a device or software module that tracks a user's physical location using GPS or other location technology.

[1694] "Means for overlaying and displaying information in the user's field of view" refers to a device or software module that displays information by overlaying it in the user's field of view using smart glasses, a head-mounted display, or the like.

[1695] The present invention relates to a system that provides real-time store guidance in response to customer requests. The system combines the functions of voice recognition, natural language processing, operation guidance, user interface display, user operation tracking, correction guidance, location information tracking, and overlay display.

[1696] This system is configured as follows:

[1697] Voice recognition means

[1698] The server converts the voice received by the staff through devices such as smart glasses into text data using voice recognition software (e.g., Google Speech-to-Text API).

[1699] Natural language processing tools

[1700] The text data is analyzed by a server-side natural language processing engine (e.g., Google Cloud Natural Language API or IBM Watson) to extract the user's intent. For example, if a customer asks, "Where is the wine section?", the intent is extracted as "Find the location of the wine section."

[1701] Operation guide generation means

[1702] Based on the analyzed intent, the server generates a guide for the relevant operating procedures. Specifically, it refers to the store's map data to calculate the optimal route and generates data for displaying the guide.

[1703] User interface display means

[1704] The procedures generated by the operation guide generation means are overlaid on the display of the smart glasses worn by the staff member, using a user interface framework such as Unity.

[1705] User action tracking

[1706] The server tracks the physical location of staff in real time using GPS sensors and other location technologies, ensuring they are heading in the right direction.

[1707] Correction guide generation method

[1708] If the user is heading in the wrong direction or making a mistake, the server will instantly generate corrective instructions and display them on the smart glasses' display, such as "Turn left at the next corner."

[1709] Specific examples

[1710] As a concrete example, if a customer asks, "Where is the wine section?", the sequence of events is as follows:

[1711] 1. Staff wear smart glasses and capture customer questions using voice recognition.

[1712] 2. The server converts the speech to text.

[1713] 3. The NLP engine analyzes the question and extracts the intent: "Find a wine section."

[1714] 4. The store map data is referenced to generate a route guide, and the smart glasses display shows "Turn right from here."

[1715] 5. The server uses GPS data to track the staff's location and ensure they are heading in the right direction.

[1716] 6. If you are heading to the wrong location, the server will display a corrective instruction such as "Turn left at the next corner."

[1717] Prompt Sentence Examples

[1718] Enter the following prompts into the AI ​​model to enable it to generate the appropriate program:

[1719] Based on the user's voice question, your goal is to generate a program that uses an NLP engine to analyze the intent and provide appropriate product information and floor guides in real time. For example, if a customer asks, "Where is the wine section?", please explain the following steps:

[1720] 1. Voice Recognition

[1721] 2. NLP analysis

[1722] 3. Guide Generation

[1723] 4. UI display

[1724] 5. Operation Tracking

[1725] 6. Guide correction

[1726] Please also include specific examples of the hardware and software you will use.

[1727] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1728] Step 1:

[1729] Audio Capture and Conversion

[1730] Input: The user speaks a voice question into the smart glasses.

[1731] Specific operation: The staff activates the voice input function of the smart glasses and captures the question. The smart glasses use the built-in microphone to capture the voice data.

[1732] Data processing and calculation: A speech recognition module (e.g., Google Speech-to-Text API) converts the voice data into text data.

[1733] Output: Text data (e.g., "Where is the wine store?")

[1734] Step 2:

[1735] Text analysis and intent extraction

[1736] Input: Text data sent from smart glasses

[1737] Specific operation: The server receives the text data and sends it to a natural language processing engine (e.g., Google Cloud Natural Language API).

[1738] Data processing and calculation: A natural language processing engine analyzes text data and extracts user intent.

[1739] Output: Intent data (e.g., "Find a wine section")

[1740] Step 3:

[1741] Operation guide generation

[1742] Input: Intent data

[1743] Specific operation: Based on the intent data, the server references the location information within the store and calculates the appropriate guidance route.

[1744] Data processing and calculation: Taking information from the store's map database and applying algorithms to calculate the optimal route.

[1745] Output: Guide data (e.g. "Turn right from here")

[1746] Step 4:

[1747] User Interface Display

[1748] Input: Guide data

[1749] Specific operation: The server sends guide data to the smart glasses, which overlay it on the user's field of view. The smart glasses then display arrows and guide messages on the display.

[1750] Data processing and computation: Converting guide data into visual information using a user interface framework (e.g., Unity).

[1751] Output: Visual data (e.g., "Please turn right")

[1752] Step 5:

[1753] User Action Tracking

[1754] Input: Location data from smart glasses

[1755] What it does: The server tracks the physical location of staff in real time using GPS sensors and other location technologies.

[1756] Data processing and calculation: Analyze the acquired location data and check whether the staff is following the correct route.

[1757] Output: Tracking data (e.g. flags indicating whether you are moving in the right direction)

[1758] Step 6:

[1759] Correction guide generation

[1760] Input: Tracking data

[1761] Specific behavior: The server uses tracking data to generate corrective instructions if a staff member is taking the wrong route.

[1762] Data processing and calculation: Applying algorithms to recalculate guide data and generate corrective instructions.

[1763] Output: Corrective guidance data (e.g. "Turn left at the next corner")

[1764] This allows staff to provide appropriate store guidance in real time.

[1765] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1766] The present invention provides a system including a speech recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion engine. The program processing of this system will be explained below in natural language.

[1767] 1. Voice Recognition Method

[1768] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and receives it as audio data. The speech recognition module then converts the audio data into text data. This process passes the user's instructions in text form to the system.

[1769] 2. Natural Language Processing Methods

[1770] The device then sends this text data to a natural language processing engine, which analyzes the text and extracts the user's intent, in this case "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[1771] 3. Emotion Engine

[1772] In addition, the device is equipped with an emotion engine that recognizes the user's emotions from speech recognition. For example, if the user is anxious, the engine will detect "stress" from the tone and speed of the speech. This emotion information is used to generate and display subsequent operation guides.

[1773] 4. Operation guide generation method

[1774] The device generates a guide with relevant operation steps based on the analyzed intent and recognized emotion. For example, the first step is to open the YouTube app, so the device locates the YouTube app icon on the home screen. Next, it provides specific steps for finding the search bar and typing "cat." If the user is stressed, the device generates a more detailed guide than usual.

[1775] 5. User Interface Display Method

[1776] The procedure generated by the operation guide generation means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the YouTube icon, and this display continues until the user completes the tap operation. The display method and level of detail of the guide are adjusted according to the user's emotional state.

[1777] 6. User Action Tracking Methods

[1778] The device tracks the user's actions in real time, for example, checking to see if the user has correctly opened the YouTube app, then identifying the search bar and monitoring the user's actions when typing "cat."

[1779] 7. Correction guide generation method

[1780] If the user does not perform the operation correctly, the correction guide generation means provides appropriate correction information. For example, if the user accidentally taps a different icon, the means notifies the user by saying, "You made a mistake. Please open YouTube," and displays a guide pointing back to the YouTube icon. If the user enters an incomplete search bar entry, the means instructs the user to enter "cat" correctly. If the user is feeling stressed, the means displays more detailed correction guides.

[1781] Specific examples

[1782] Specifically, the sequence of events that occurs when a user says, "I want to watch cat videos on YouTube" is shown below.

[1783] 1. The user speaks.

[1784] 2. The device converts the voice into text and analyzes it using an NLP engine.

[1785] 3. The device analyzes the user's emotions using an emotion engine. For example, it determines that the user is anxious.

[1786] 4. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video). If the user is in a hurry, more detailed steps are generated.

[1787] 5. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[1788] 6. The user follows the instructions and taps the YouTube icon.

[1789] 7. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[1790] 8. User types "cat" and performs a search.

[1791] 9. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[1792] 10. User taps on the video to start playing.

[1793] In this way, users can smoothly perform desired operations using only voice utterances. Furthermore, by incorporating an emotion engine, it is possible to provide appropriate assistance according to the user's emotional state, resulting in a more user-friendly system.

[1794] The processing flow will be explained below.

[1795] Step 1:

[1796] The user speaks into their smartphone, "I want to watch cat videos on YouTube."

[1797] Step 2:

[1798] The terminal uses a built-in microphone to capture the user's voice and receives it as voice data.

[1799] Step 3:

[1800] The terminal passes the received voice data to a voice recognition module and converts it into text data.

[1801] Step 4:

[1802] The device sends the converted text data to a natural language processing engine.

[1803] Step 5:

[1804] The device uses a natural language processing engine to analyze the text data and extract the user's intent. The intents identified are "open YouTube," "type 'cat' into the search bar," and "select a video from the search results."

[1805] Step 6:

[1806] The device sends the text data to an emotion engine, which analyzes the user's emotions. For example, it can determine that the user is anxious based on the tone and speed of the voice.

[1807] Step 7:

[1808] The device generates operation guidance based on the analyzed intention and the user's emotions. If the user is in a hurry, the operation guidance will include more detailed steps than usual.

[1809] Step 8:

[1810] Your device will locate the YouTube app icon on your home screen.

[1811] Step 9:

[1812] The device will display a cursor pointing to the YouTube app icon and a guide message saying "Press here."

[1813] Step 10:

[1814] The user taps the YouTube icon.

[1815] Step 11:

[1816] Make sure the YouTube app is launched correctly on your device.

[1817] Step 12:

[1818] The device identifies the search bar within the YouTube app.

[1819] Step 13:

[1820] The device displays a cursor pointing to the search bar and a guide message saying "Type 'cat' here."

[1821] Step 14:

[1822] A user types "cat" into the search bar and presses Enter.

[1823] Step 15:

[1824] The device confirms the user's input and verifies that search results are displayed.

[1825] Step 16:

[1826] The device locates the first video in the search results.

[1827] Step 17:

[1828] The device will display a cursor pointing to the video and a guide message saying "Play this."

[1829] Step 18:

[1830] The user taps on the first video and it begins playing.

[1831] Step 19:

[1832] The terminal confirms the completion of the user operation and ends the guide display.

[1833] Example 2

[1834] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1835] Conventional systems have difficulty accurately analyzing the user's intention when giving voice instructions, and the operation guidance provided does not take into account the user's emotional state. As a result, users often feel stressed or make mistakes while operating the device. Furthermore, if a user makes a mistake, no appropriate correction guidance is provided, forcing the user to manually correct the mistake, resulting in an insufficient operation experience.

[1836] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1837] In this invention, the server includes a voice recognition unit, a natural language processing unit, an emotion recognition unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, and a correction guide generation unit. This makes it possible to accurately analyze the user's voice instructions and provide detailed operation guides based on the user's emotional state. Furthermore, by quickly presenting correction guides when the user makes an operation error, the operation experience can be made smoother and stress-free.

[1838] A "voice recognition means" is a means for capturing a user's speech, saving it as voice data, and then converting it into text data.

[1839] "Natural language processing means" is a means for analyzing text data and extracting the user's intent.

[1840] An "emotion recognition means" is a means for analyzing a user's emotional state from the user's speech or text data.

[1841] The "operation guide generation means" is a means for generating operation procedures based on the user's intentions and emotional state, and providing the user with appropriate guidance.

[1842] The "user interface display means" is a means for overlaying and displaying the generated operation guide on the screen.

[1843] The "user operation tracking means" is a means for tracking the user's operation behavior in real time and monitoring whether the operation has been performed correctly.

[1844] The "correction guide generating means" is a means for providing appropriate correction information when the user makes an incorrect operation, and urging the user to perform the correct operation again.

[1845] The present invention provides a system for enabling a user to efficiently achieve a desired operation through voice instructions. A specific embodiment of the system will be described below.

[1846] Program processing

[1847] Voice recognition means

[1848] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and save it as audio data. The device then converts the speech data into text using the Google Cloud Speech-to-Text API. This process passes the user's instructions to the system in text format.

[1849] Natural language processing tools

[1850] The device sends the acquired text data to the Google Cloud Natural Language API, which analyzes the text and extracts the user's intent. In this case, it identifies the user's intent, such as "open YouTube," "enter 'cat' in the search bar," and "select a video from the search results."

[1851] emotion recognition means

[1852] The device uses IBM Watson Tone Analyzer to recognize the user's emotions from speech recognition and text data. For example, if the user is feeling anxious, the device can detect "stress" from the tone and speed of the voice. The recognized emotional information is reflected in the generation of the next operation guide.

[1853] Operation guide generation means

[1854] The device generates relevant instructions based on the analyzed intent and emotional state. Using ARKit or ARCore, the device identifies the YouTube app icon on the user's smartphone screen, then creates detailed instructions for finding the search bar and typing "cat." For example, if the user is feeling stressed, the device generates more detailed instructions than usual.

[1855] User interface display means

[1856] The generated operation guide is overlaid on the screen using Unity or Flutter. For example, a message saying "Press here" is displayed on top of the YouTube icon, and remains displayed until the user taps it. The display method and level of detail of the guide are adjusted depending on the emotion.

[1857] User action tracking

[1858] The device uses Firebase Analytics to track user behavior in real time, for example, to ensure that a user successfully opens the YouTube app and then types "cat" into the search bar.

[1859] Correction guide generation method

[1860] If the user makes an incorrect operation, the device generates a correction guide. For example, if the user accidentally taps the wrong icon, the device will provide a correction message saying, "You made a mistake. Please open YouTube." The device will then provide a guide pointing back to the YouTube icon. The correction guide will be adjusted to be more detailed depending on the user's emotional state.

[1861] Specific examples

[1862] As a concrete example, the following sequence of events occurs when a user says, "I want to watch cat videos on YouTube."

[1863] 1. The user speaks.

[1864] 2. The device converts the voice into text and analyzes it using a natural language processing engine.

[1865] 3. The device analyzes the user's emotions using an emotion recognition engine. For example, it determines that the user is anxious.

[1866] 4. The device generates the necessary steps (open the YouTube app, type 'cat' in the search bar, select a video). If the user is in a hurry, more detailed steps are generated.

[1867] 5. The device will display a cursor pointing at the YouTube icon on the home screen and a prompt saying "Press here."

[1868] 6. The user follows the instructions and taps the YouTube icon.

[1869] 7. The device will display a cursor pointing to the search bar and the prompt "Type 'cat' here."

[1870] 8. User types "cat" and performs a search.

[1871] 9. The device will display a cursor pointing to the appropriate video and a prompt saying "Play this."

[1872] 10. User taps on the video to start playing.

[1873] Prompt Sentence Examples

[1874] An example of a prompt is as follows:

[1875] "I want to watch cat videos on YouTube. Convert the audio data into text, analyze the user's intent and emotions, and generate detailed operating instructions."

[1876] The above is a specific embodiment for carrying out the present invention. This system allows users to efficiently perform desired operations using only voice input, and by incorporating emotion recognition, it is possible to provide optimal support according to the user's condition.

[1877] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1878] Program processing flow

[1879] Step 1:

[1880] Audio capture and transcription

[1881] A user speaks into their smartphone, saying, "I want to watch cat videos on YouTube." The device uses its built-in microphone to capture this speech and save it as audio data. The device then sends this audio data to the Google Cloud Speech-to-Text API, which converts it into text.

[1882] Input: Audio data

[1883] Output: Text data ("I want to watch cat videos on YouTube")

[1884] Step 2:

[1885] Intention analysis of text data

[1886] The device sends the acquired text data to the Google Cloud Natural Language API. This API analyzes the input text data and extracts the user's intent. For example, the analysis results may lead to operational steps such as "open the YouTube app," "enter 'cat' in the search bar," and "select a video from the search results."

[1887] Input: Text data

[1888] Output: User intent (list of action steps)

[1889] Step 3:

[1890] Emotion recognition

[1891] The device uses IBM Watson Tone Analyzer to analyze the user's emotions from voice and text data. For example, it can detect "stress" from the tone and speed of the user's voice. This emotional information is reflected in the generation of subsequent operation guides.

[1892] Input: Audio data, text data

[1893] Output: Emotion data (e.g., stress)

[1894] Step 4:

[1895] Generate operation guides

[1896] Based on the analyzed intent and emotional state, the device uses ARKit and ARCore to generate operation instructions, specifically, to identify the YouTube app icon on the user's smartphone screen, then create detailed instructions for finding the search bar and typing "cat."

[1897] Input: User intent, emotional data

[1898] Output: Operation Guide (detailed instructions)

[1899] Step 5:

[1900] View Guide

[1901] The device uses Unity or Flutter to overlay the generated operation guide on the screen. Specifically, a message saying "Press here" appears above the YouTube icon and remains displayed until the user taps it. The display method and level of detail of the guide are adjusted depending on the emotion.

[1902] Input: Operation Guide

[1903] Output: Guide display on the user interface

[1904] Step 6:

[1905] Tracking user actions

[1906] The device uses Firebase Analytics to track user behavior in real time, for example, whether a user successfully opens the YouTube app or types "cat" into the search bar.

[1907] Input: User operation data

[1908] Output: Operation tracking data (flag indicating whether the operation was successful)

[1909] Step 7:

[1910] Generate repair guides

[1911] If the user makes a mistake, the device generates a correction guide. For example, if the user accidentally taps a different icon, the device will notify the user, "You made a mistake. Please open YouTube," and provide a guide pointing back to the YouTube icon.

[1912] Input: Operation tracking data

[1913] Output: Fix guide

[1914] By performing these steps consecutively, users receive detailed and appropriate operation guidance through voice alone, enabling them to smoothly accomplish their desired operations. Furthermore, guidance is adjusted based on the user's emotional state, providing a more user-friendly experience.

[1915] (Application example 2)

[1916] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1917] In conventional systems, even when users issue voice instructions, it can be difficult for them to accurately understand and operate the system. Furthermore, if the user makes an incorrect operation, no appropriate guidance is provided, which can lead to operational delays and increased stress and frustration for the user. Furthermore, conventional systems do not take into account the user's emotional state when providing operation guidance, resulting in a lack of a user-friendly interface. This problem, which reduces the quality of the user experience, needs to be addressed.

[1918] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1919] In this invention, the server includes a voice recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion analysis unit. This allows the server to accurately understand the user's voice instructions, provide appropriate operation procedures, and provide customized support according to the user's emotional state. Furthermore, by providing correction guides in real time for user operation errors, the server allows the user to complete operations without stress.

[1920] "Speech recognition means" is a technology that captures a user's speech as voice data and converts it into text data.

[1921] "Natural language processing means" is a technology that analyzes the user's intentions from the character data converted by the voice recognition means.

[1922] The "operation guide generation means" is a technology that generates related operation procedures based on the analyzed user's intentions.

[1923] The "user interface display means" is a technique for visually displaying to the user the procedures generated by the operation guide generation means.

[1924] "User operation tracking means" is a technology that tracks user operation behavior in real time and confirms that the operation is being performed accurately.

[1925] The "correction guide generating means" is a technology that provides appropriate correction information when the user performs an incorrect operation.

[1926] "Emotion analysis means" is a technology that analyzes emotions from the content of a user's speech and determines their emotional state.

[1927] The present invention provides a system including a voice recognition unit, a natural language processing unit, an operation guide generation unit, a user interface display unit, a user operation tracking unit, a correction guide generation unit, and an emotion analysis unit. The system aims to accurately understand a user's voice instructions, display appropriate operation guides according to the user's emotional state, and enable the user to smoothly perform the intended operations.

[1928] 1. Voice Recognition Method

[1929] First, the user speaks into the device. For example, if the user says, "I want to watch the latest movie," the device uses the built-in microphone to capture this speech and receives it as voice data. The voice recognition module then converts the voice data into text data. Through this process, the user's instructions are passed on to the system in text format. The speech_recognition library is used for this voice recognition.

[1930] 2. Natural Language Processing Methods

[1931] The device then sends this text data to a natural language processing engine, which uses Hugging Face's transformers library to analyze the user's intent, in this case "Show me the latest movies."

[1932] 3. Emotion analysis means

[1933] The device is also equipped with an emotion analysis unit that analyzes the user's emotions from speech recognition. The prompt text is used to input the emotion model, which determines whether the user is feeling anxious or stressed. This emotion information is then used to generate and display subsequent operation guides. The emotion analysis also utilizes the emotion analysis model from Hugging Face's transformers library.

[1934] 4. Operation guide generation method

[1935] The device generates a guide with relevant operation steps based on the analyzed intent and the recognized emotion. For example, the first step is "Display movie list." If the device recognizes that the user is impatient, the operation guide generates more detailed instructions than usual, allowing the user to proceed to the next step without hesitation.

[1936] 5. User Interface Display Method

[1937] The procedure generated by the operation guide generation means is overlaid on the screen via the user interface display means. Specifically, a cursor and a message saying "Press here" are displayed on the movie list icon, and this display continues until the user completes the tap operation. The display method and level of detail of the guide are adjusted according to the user's emotional state.

[1938] 6. User Action Tracking Methods

[1939] The device tracks the user's actions in real time, ensuring that the user correctly opens the movie list as intended, and then monitors other actions.

[1940] 7. Correction guide generation method

[1941] If the user does not perform the operation correctly, the correction guide generation means will provide appropriate correction information. For example, if the user accidentally taps a different icon, the means will notify the user, "You made a mistake. Please open the movie list." and display a guide pointing back to the movie list icon. If the user enters an incomplete search bar entry, the means will instruct the user, "Please enter 'movies' correctly." If the user is feeling stressed, the means will display more detailed correction guides.

[1942] Specific examples

[1943] The following sequence of events occurs when a user says, "I want to watch the latest movies."

[1944] 1. The user speaks.

[1945] 2. The device converts the voice into text and analyzes it using a natural language processing engine.

[1946] 3. The device analyzes the user's emotions using emotion analysis means. For example, it determines that the user is anxious.

[1947] 4. The device generates the necessary instructions (show movie list). If the user is impatient, more detailed instructions are generated.

[1948] 5. The device will display a cursor pointing at the movie list icon and a prompt saying "Press here."

[1949] 6. The user follows the instructions and taps the movie list icon.

[1950] 7. Your device will display a cursor pointing to the search bar and a prompt saying "Type 'Movies' here."

[1951] 8. User types "movies" and performs a search.

[1952] 9. The device will display a cursor pointing to the appropriate movie and a prompt saying "Play this."

[1953] 10. The user taps on a movie to start playing it.

[1954] An example prompt is:

[1955] User: "I want to watch the latest movies"

[1956] System: "Show the latest movies. Searching..."

[1957] In this way, users can smoothly perform desired operations using only voice utterances. Furthermore, by incorporating emotion analysis means, it is possible to provide appropriate assistance according to the user's emotional state, resulting in a more user-friendly system.

[1958] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1959] Step 1:

[1960] Start voice recognition

[1961] Input: User says "I want to watch the latest movies."

[1962] The server uses the built-in microphone to capture the user's voice, and converts this voice data into text data using the speech_recognition library.

[1963] Output: The converted text data, e.g. "I want to watch the latest movies."

[1964] Step 2:

[1965] Text data analysis

[1966] Input: Text data generated in step 1

[1967] The server uses Hugging Face's transformers library to send the text data to a natural language processing engine to analyze the user's intent, which is "Show me the latest movies."

[1968] Output: User intent ("Show me the latest movies").

[1969] Step 3:

[1970] Performing sentiment analysis

[1971] Input: Text data generated in step 1

[1972] The server then performs emotion analysis using the Hugging Face transformers library, again to determine the user's emotional state, such as whether they are anxious or stressed.

[1973] Output: User's emotional information (e.g., feeling anxious).

[1974] Step 4:

[1975] Generate operation guides

[1976] Input: User intent identified in step 2, user sentiment information obtained in step 3

[1977] The server generates relevant instructional steps based on the user's intention and emotional state. For example, if the user is impatient, it generates detailed instructions ("Tap the movie icon in the bottom right of the screen").

[1978] Output: Detailed operation guide.

[1979] Step 5:

[1980] Display in the user interface

[1981] Input: The operation guide generated in step 4

[1982] The terminal displays the generated operation guide as an overlay on the user interface, for example, by displaying a cursor and a message saying "Press here" on a movie list icon.

[1983] Output: A visual guide on the user interface.

[1984] Step 6:

[1985] User Action Tracking

[1986] Input: User action (user taps on the movie icon)

[1987] The device tracks the user's actions in real time, confirms whether the user performed the action correctly as intended, and takes the next step if necessary.

[1988] Output: User behavior data.

[1989] Step 7:

[1990] Providing a guide to remediation

[1991] Input: User behavior data obtained in step 6

[1992] If the user makes an incorrect operation, the server uses a correction guide generation means to provide appropriate correction information. For example, if the user accidentally taps a different icon, the server displays a message saying "You made a mistake. Please open the movie list" and instructs the user to perform the correct operation again.

[1993] Output: Correction guide and new operating instructions.

[1994] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1995] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1996] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1997] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1998] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1999] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2000] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2001] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2002] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2003] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2004] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2005] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2006] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2007] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2008] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2009] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2010] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2011] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2012] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2013] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2014] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2015] The following is further disclosed regarding the above embodiment.

[2016] (Claim 1)

[2017] a speech recognition means;

[2018] natural language processing means;

[2019] An operation guide generating means;

[2020] a user interface display means;

[2021] A means for tracking user actions;

[2022] A correction guide generating means;

[2023] A system including:

[2024] (Claim 2)

[2025] 2. The system according to claim 1, wherein the speech recognition means converts the user's speech into text data.

[2026] (Claim 3)

[2027] 2. The system according to claim 1, wherein the system analyzes the user's intention from the text data using natural language processing means.

[2028] (Claim 4)

[2029] 2. The system according to claim 1, wherein the operation guide generating means generates an operation guide on the screen based on the user's intention.

[2030] (Claim 5)

[2031] 2. The system according to claim 1, wherein the user interface display means displays an operation guide on the screen in an overlay display.

[2032] (Claim 6)

[2033] 2. The system according to claim 1, wherein the user operation tracking means tracks the user's operation behavior in real time.

[2034] (Claim 7)

[2035] 2. The system according to claim 1, wherein the correction guide generating means generates a correction guide when the user's operation is not performed correctly.

[2036] "Example 1"

[2037] (Claim 1)

[2038] a speech recognition means;

[2039] natural language processing means;

[2040] An operation guide generating means;

[2041] a user interface display means;

[2042] A means for tracking user actions;

[2043] A correction guide generating means;

[2044] a means for capturing and converting user voice instructions into text data;

[2045] A means of analyzing text data to extract user intent;

[2046] means for generating and displaying an operation instruction based on the analyzed intention;

[2047] a means for tracking user operations in real time in accordance with the operation instructions;

[2048] A means for displaying a correction guide when an incorrect operation is detected;

[2049] A system including:

[2050] (Claim 2)

[2051] 2. The system according to claim 1, wherein the system converts the user's speech into text data using a speech recognition means, and displays relevant operating procedures as a guide on the screen based on the analyzed intention.

[2052] (Claim 3)

[2053] 2. The system according to claim 1, wherein natural language processing means analyzes the user's intention from text data and generates guidance for specific operations in real time.

[2054] "Application Example 1"

[2055] (Claim 1)

[2056] a speech recognition means;

[2057] natural language processing means;

[2058] An operation guide generating means;

[2059] a user interface display means;

[2060] A means for tracking user actions;

[2061] A correction guide generating means;

[2062] a means for tracking location information within a physical space;

[2063] a means for overlaying information onto the user's field of view;

[2064] A system including:

[2065] (Claim 2)

[2066] 2. The system according to claim 1, wherein the speech recognition means converts the user's speech into text data.

[2067] (Claim 3)

[2068] 2. The system according to claim 1, wherein the system analyzes the user's intention from the text data using natural language processing means.

[2069] "Example 2: Combining Emotion Engines"

[2070] (Claim 1)

[2071] a speech recognition means;

[2072] natural language processing means;

[2073] An emotion recognition means;

[2074] An operation guide generating means;

[2075] a user interface display means;

[2076] A means for tracking user actions;

[2077] A correction guide generating means;

[2078] A system including:

[2079] (Claim 2)

[2080] 2. The system according to claim 1, wherein the speech recognition means converts the user's speech into text data.

[2081] (Claim 3)

[2082] 2. The system according to claim 1, wherein the system analyzes the user's intention from the text data using natural language processing means.

[2083] (Claim 4)

[2084] 2. The system according to claim 1, wherein the emotion recognition means analyzes the emotional state of the user and adjusts the operation guide generation means based on the analysis.

[2085] (Claim 5)

[2086] 2. The system according to claim 1, wherein the operation guide generating means generates an operation procedure according to the user's intention and emotional state.

[2087] (Claim 6)

[2088] 2. The system according to claim 1, wherein the user interface display means displays the generated operation guide on the screen as an overlay.

[2089] (Claim 7)

[2090] 2. The system according to claim 1, wherein the user operation tracking means tracks the user's operation behavior in real time.

[2091] (Claim 8)

[2092] 2. The system according to claim 1, wherein the correction guide generating means provides appropriate correction information when the user's operation is incorrect.

[2093] "Application example 2 when combining emotion engines"

[2094] (Claim 1)

[2095] a speech recognition means;

[2096] natural language processing means;

[2097] An operation guide generating means;

[2098] a user interface display means;

[2099] A means for tracking user actions;

[2100] A correction guide generating means;

[2101] A sentiment analysis means;

[2102] A system including:

[2103] (Claim 2)

[2104] 2. The system according to claim 1, wherein the speech recognition means converts the user's speech into text data.

[2105] (Claim 3)

[2106] 2. The system according to claim 1, wherein the system analyzes the user's intention from the text data using natural language processing means.

[2107] (Claim 4)

[2108] 2. The system according to claim 1, wherein the emotion analysis means analyzes emotions from the content of the user's speech.

[2109] (Claim 5)

[2110] 2. The system according to claim 1, wherein the operation guide generation means generates an operation procedure according to the emotional state of the user based on the emotion analysis result.

[2111] (Claim 6)

[2112] 2. The system according to claim 1, wherein the user interface display means displays an appropriate operation guide based on the emotion analysis result.

[2113] (Claim 7)

[2114] 2. The system according to claim 1, wherein the user operation tracking means tracks the user's operation behavior in real time, and when an incorrect operation is detected, the correction guide generating means provides appropriate correction information. [Explanation of symbols]

[2115] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

[Claim 1] a speech recognition means for converting a user's speech into text data; natural language processing means for analyzing user intent from text data; an operation guide generating means for generating an operation guide on a screen based on a user's intention; a user interface display means for displaying an operation guide on the screen in an overlay display; A user operation tracking means for tracking user operation behavior in real time; a correction guide generating means for generating a correction guide when a user's operation is not performed correctly; A system including:

Images