system

Abstract

Provide a system. A method for capturing audio input, comprising: means for converting the voice input into text data; means for analyzing the text data and identifying the user's intention; means for selecting and launching an appropriate application based on the user's intent; means for acquiring the execution result of the application and notifying the user by voice; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional smartphone applications are complicated to operate for elderly people and users unfamiliar with technology, making it difficult to select and launch the desired application and check the results. In particular, using transit information and map applications requires numerous operations, making it difficult for elderly people to use them smoothly. To solve this problem, there was a demand for a system that allows applications to be operated using only voice and that notifies necessary information by voice. [Means for solving the problem]

[0005] The present invention provides a system including means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying a user's intent, means for selecting and launching an appropriate application based on the user's intent, and means for obtaining the execution results of the application and notifying the user by voice. In particular, by automating the selection and launch of transit guide and map applications and notifying the results by voice, a simple and convenient user experience can be provided even for elderly users. This system enables elderly users and users who are unfamiliar with technology to use smartphones efficiently and effectively.

[0006] "Voice input" refers to a user speaking into a smartphone or other device, which then records the speech as a digital signal.

[0007] "Capturing means" refers to a device or function that captures audio input as digital data and passes that data on to further processing steps.

[0008] "Text data" refers to character string information converted from voice input, and has the format of a natural language.

[0009] "Analysis" refers to the process of analyzing text data to understand its meaning and identify the user's intent.

[0010] "User intent" refers to the goal or request that the user wants to achieve through voice input.

[0011] "Means for selecting and launching an appropriate application" refers to the process of identifying the most appropriate application based on the user's intent and automatically launching that application.

[0012] "Application" refers to software designed to perform a specific function (e.g., transit directions or map display).

[0013] "Execution results" refers to the information or results output by the launched application.

[0014] "Means for notifying by voice" refers to a process of converting the execution result into voice and audibly conveying it to the user.

[0015] "System" refers to the overall configuration that integrates these means and operates, and refers to technology designed to allow users to operate it using only voice. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system that captures voice input, selects and launches an appropriate application according to the user's intention, and notifies the user of the result by voice. Specific embodiments for implementing this system are described below.

[0038] 1. Receiving voice input

[0039] The user speaks to their smartphone, saying, "I want to go from here to where I want to go." For example, they might say, "I want to go from Shibuya to Shinjuku." The device captures the voice data through a microphone. This voice data is temporarily stored on the device.

[0040] 2. Voice Recognition

[0041] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice input of "I want to go from Shibuya to Shinjuku" is converted into text "I want to go from Shibuya to Shinjuku." This text data is temporarily saved as it will be used in the next step.

[0042] 3. Natural Language Processing (NLP)

[0043] The device analyzes the text data using a natural language processing (NLP) engine. The NLP engine identifies the user's intent and extracts the necessary parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" may be extracted. The analysis results are also temporarily saved.

[0044] 4. Select and launch the app

[0045] The device selects an appropriate application based on the analysis results and prepares to launch it. For example, if the analysis determines that the user has the intent to "move," the device selects a transit guide application. The device prepares to pass the necessary parameters (such as departure and arrival points) to the application.

[0046] 5. Result acquisition and voice notification

[0047] The server accesses the specified API endpoint and obtains the information requested by the user. For example, a user requests "route from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. The obtained information is then returned from the server to the device.

[0048] The device converts the received result into voice data using a speech synthesis engine. For example, a notification such as "The next train leaves at 10:30 and costs 300 yen" is generated. This voice data is played back from the device's speaker, providing the user with an audible notification.

[0049] Specific examples

[0050] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[0051] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[0052] 2. The device captures the voice and converts it into text using a speech recognition engine: "I want to go from Shibuya to Shinjuku."

[0053] 3. The device analyzes the text using an NLP engine and extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[0054] 4. The device selects the transit information application and prepares to launch it.

[0055] 5. The server retrieves route information from Shibuya to Shinjuku from the API and returns details (next train time, fare, etc.) to the terminal.

[0056] 6. The terminal converts the result of the operation into voice and announces, "The next train leaves at 10:30 and costs 300 yen."

[0057] 7. The device launches the transit information application and displays detailed information.

[0058] In this way, the present invention provides a system that enables even elderly people and users who are unfamiliar with technology to easily operate a desired application and obtain necessary information.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] The user speaks into the smartphone to input the necessary instructions, for example, "I want to go from Shibuya to Shinjuku." This voice data is then captured by the smartphone's microphone.

[0062] Step 2:

[0063] The device sends the captured voice data to its internal voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku."

[0064] Step 3:

[0065] The device sends the converted text data to a natural language processing (NLP) engine, which analyzes the text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted.

[0066] Step 4:

[0067] The device checks the analysis results and determines which application is appropriate. In this case, since the user's intention is related to "travel," the transit guide application is selected.

[0068] Step 5:

[0069] The terminal prepares to launch the selected transit information application by passing necessary parameters (e.g., departure point, arrival point).

[0070] Step 6:

[0071] The server processes API requests for transit information. For example, a request like "https: / / api.transit.com / getInfo?saddr=Shibuya&daddr=Shinjuku" is sent to retrieve information such as "The next train departs at 10:30 and costs 300 yen."

[0072] Step 7:

[0073] The server returns the acquired information to the terminal in JSON format.

[0074] Step 8:

[0075] The terminal analyzes the information it receives and extracts the necessary information (e.g., the time of the next train, the fare).

[0076] Step 9:

[0077] The device sends the extracted information to a speech synthesis engine, which converts it into voice data. For example, the generated voice data would say, "The next train leaves at 10:30 and costs 300 yen."

[0078] Step 10:

[0079] The terminal plays the generated voice data and notifies the user, so that the user receives the necessary information by voice.

[0080] In this way, this system combines voice recognition and natural language processing to select and launch the appropriate application based on the user's voice input, and then returns the results in voice form, implementing a series of processes.

[0081] Example 1

[0082] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0083] In modern society, when elderly people and users unfamiliar with technology use smartphones to obtain information, the operation is often complicated, making it difficult to obtain information. In particular, when obtaining information using voice input, there is a demand for an efficient system that can smoothly and intuitively perform the entire process from voice recognition to application selection and notification of results.

[0084] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0085] In this invention, the server includes: means for capturing voice input; means for converting the voice input into text data; means for analyzing the text data and identifying the user's intent; means for selecting and launching appropriate software based on the user's intent; means for acquiring the execution results of the software and notifying the user by voice; means for capturing the voice input with a microphone and converting it into text data using a voice recognition engine; means for analyzing the text data with a natural language processing engine and extracting necessary parameters; means for sending a request to the server based on the parameters and acquiring necessary information from a specified API endpoint; and means for converting the acquired information into voice data using a voice synthesis engine and notifying the user. This enables even elderly people and users who are not familiar with technology to easily acquire desired information using only voice input.

[0086] "Voice input" refers to the act of a user giving instructions by voice, which is captured as digital voice data via a microphone.

[0087] "Text data" refers to data that has been converted into text form after voice input has been analyzed by a voice recognition engine.

[0088] "Analysis" refers to the process of using a natural language processing engine to identify the user's intent and required information from text data.

[0089] "User intent" refers to the action or information the user is seeking, as determined from the voice input.

[0090] "Software" refers to application programs that are selected and launched based on the user's intent.

[0091] A "microphone" is a hardware device for capturing audio input.

[0092] A "speech recognition engine" is a software or hardware mechanism for converting voice data captured by a microphone into text data.

[0093] A "natural language processing engine" is a software mechanism that analyzes text data and extracts the user's intent and necessary parameters.

[0094] "Parameters" refer to necessary information extracted from the analyzed text data, specifically, the departure and arrival points.

[0095] A "server" is a program or device with a specific function that retrieves information from a specified API based on a user request.

[0096] "API endpoint" refers to the interface of an external service that a server accesses to obtain information.

[0097] A "speech synthesis engine" is software for converting text data into voice data.

[0098] "Voice notification" refers to the act of notifying a user of information using voice data generated by a voice synthesis engine.

[0099] The present invention relates to a system that allows a user to obtain information by voice input and notifies the user of the information by voice. Specific means for implementing this system will be described below.

[0100] overview

[0101] The system of this invention captures and analyzes voice input, selects and launches appropriate software according to the user's intention, and notifies the user of the results by voice, making it easy for even elderly users and those unfamiliar with technology to obtain information.

[0102] Hardware and software configuration

[0103] 1. Receiving voice input

[0104] The user issues specific instructions to the smartphone. For example, the user says, "I want to go from Shibuya to Shinjuku." The device's microphone captures this voice input and generates voice data. This voice data is stored in the device's temporary memory.

[0105] 2. Voice Recognition

[0106] The device sends the voice data to a voice recognition engine (for example, a commercial voice recognition API). The voice recognition engine converts the captured voice data into text data. The converted text data is in the format of, for example, "I want to go from Shibuya to Shinjuku." This text data is also temporarily stored in the device.

[0107] 3. Natural Language Processing (NLP)

[0108] The device sends the text data to a natural language processing (NLP) engine (e.g., a commercial NLP API). The NLP engine analyzes the text data and extracts important parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" is extracted. This parameter information is also temporarily stored on the device.

[0109] 4. Select and launch the app

[0110] The device selects the appropriate software based on the analysis results. For example, if the user has the intent to "move," the device selects transit information software (e.g., a commercial transit information app). The device prepares to launch the selected application and collects the necessary parameters (start and end locations).

[0111] 5. Result acquisition and voice notification

[0112] The device sends the analysis results (parameters) to the server, and the server accesses the specified API endpoint (for example, a commercial API endpoint) to obtain the required information. For example, the server requests "route information from Shibuya to Shinjuku" and obtains the next train time, fare, etc. The obtained information is returned from the server to the device.

[0113] The device converts the acquired information into speech using a speech synthesis engine (for example, a commercial speech synthesis API). The speech data will be in the form of "The next train leaves at 10:30 and costs 300 yen." This speech data is played back to the user through the device's speaker.

[0114] Specific examples

[0115] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[0116] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[0117] 2. The device's microphone captures the audio data and stores it in temporary memory.

[0118] 3. The device sends the voice data to a speech recognition engine and converts it into text.

[0119] 4. The device sends the text data to a natural language processing engine, which extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[0120] 5. The device selects the transit information software and prepares to launch it.

[0121] 6. The terminal sends the extracted parameters to the server.

[0122] 7. The server requests the API endpoint and obtains the route information.

[0123] 8. The server returns the acquired information (next train time, fare) to the terminal.

[0124] 9. The information received by the device is converted into voice using a speech synthesis engine, and the user is notified that "The next train leaves at 10:30 and costs 300 yen."

[0125] 10. The device will launch the transit information software and display detailed information.

[0126] Prompt Sentence Examples

[0127] "If I say I want to go from Shibuya to Shinjuku, please explain the specific process."

[0128] In this way, the present invention enables even elderly people and users who are unfamiliar with technology to quickly and easily obtain desired information using only voice input.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Step 1: Receiving voice input

[0131] The user issues specific instructions to the smartphone. For example, the user might say, "I want to go from Shibuya to Shinjuku." The device's microphone captures this voice input and generates digital audio data, which is then stored in the device's temporary memory.

[0132] Input: User voice input

[0133] Output: Audio data

[0134] Step 2: Voice Recognition

[0135] The device sends the voice data to a voice recognition engine (for example, a commercial voice recognition API). The voice recognition engine converts the captured voice data into text data. For example, the text data is converted into text data in the format "I want to go from Shibuya to Shinjuku." This text data is temporarily stored in the device.

[0136] Input: Audio data

[0137] Output: Text data

[0138] Step 3: Natural Language Processing (NLP)

[0139] The device sends the text data to a natural language processing (NLP) engine (e.g., a commercial NLP API). The NLP engine analyzes the text data and extracts important parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" is extracted. This parameter information is also temporarily stored on the device.

[0140] Input: Text data

[0141] Output: Extracted parameters (e.g. "Departure: Shibuya", "Destination: Shinjuku")

[0142] Step 4: Select and launch the app

[0143] The device selects the appropriate software based on the analysis results. For example, if the analysis results indicate that the user has the intention of "moving," the device selects transit information software. Next, the device prepares to launch the selected application and collects the necessary parameters (starting point and destination).

[0144] Input: Extracted parameters

[0145] Output: Software selection and launch preparation

[0146] Step 5: Get the results and hear the audio

[0147] The device sends the analysis results (parameters) to the server. The server accesses the specified API endpoint and obtains the information requested by the user. For example, the server requests "route information from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. This collected information is returned from the server to the device. The device then converts the obtained information into voice data using a speech synthesis engine and notifies the user, "The next train leaves at 10:30 and costs 300 yen."

[0148] Input: Analysis results (parameters)

[0149] Output: Acquired information and its spoken notification

[0150] In this way, a series of steps is realized, starting with the user's voice input, followed by voice recognition, natural language processing, application selection and launch, and result acquisition and voice notification. Appropriate data processing and data calculation are performed at each step, ultimately resulting in a system that provides the user with the information they need intuitively and efficiently.

[0151] (Application example 1)

[0152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] Current food delivery services require users to manually operate the application, which is particularly cumbersome for elderly people and those who are not familiar with technology. Also, visually impaired users have difficulty operating smartphones. Therefore, there is a need for a method that allows everyone to easily use food delivery services.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0155] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying the user's intention, means for selecting and launching an appropriate application based on the user's intention, means for acquiring the execution result of the application and notifying the user by voice, and means for including an application related to the food delivery service. This makes it possible to use the food delivery service with just voice input, and enables anyone to easily perform and use the ordering procedure.

[0156] "Means for capturing audio input" refers to a device and its associated software for detecting and recording audio input.

[0157] "Means for converting speech input into text data" refers to software and algorithms for converting speech signals into written information.

[0158] "Means for analyzing text data and identifying user intent" refers to natural language processing technology and related software for analyzing text data and understanding the user's intended operations and requests.

[0159] "Means for selecting and launching an appropriate application" refers to a processing system for automatically selecting and launching a particular application based on a user's intent.

[0160] "Means for obtaining the results of application execution and notifying the user by voice" refers to technology and devices for collecting the results or data of an executed application and providing it to the user as voice.

[0161] "Food delivery service application" refers to a software application that allows a user to voice-order food and process the delivery.

[0162] "Device for capturing an acoustic signal" refers to a hardware device, such as a microphone, for obtaining audio input.

[0163] "Speech recognition engine" refers to software and algorithms for processing captured acoustic signals and converting them into text data.

[0164] This invention is a system that captures voice input, selects and launches an appropriate application according to the user's intention, and notifies the user of the results by voice. Specific embodiments for implementing this system are described below.

[0165] 1. Receiving voice input

[0166] A user speaks to the device, saying, "I want to order a pizza," or, for example, "I want to order a Margherita from a food delivery service." The device captures the voice data through a microphone, a device that captures acoustic signals, and this voice data is temporarily stored within the device.

[0167] 2. Voice Recognition

[0168] The device sends the captured voice data to a speech recognition engine, which converts the voice data into text data. For example, a voice input of "I would like to order a margherita from a food delivery service" is converted into text "I would like to order a margherita from a food delivery service." This text data is temporarily stored for use in the next step.

[0169] 3. Natural Language Processing (NLP)

[0170] The device analyzes the text data using a natural language processing (NLP) engine. The NLP engine identifies the user's intent and extracts necessary parameters. For example, information such as "Vendor: food delivery service" and "Item: Margherita" may be extracted. The analysis results are also temporarily stored.

[0171] 4. Select and launch the app

[0172] The device selects an appropriate application based on the analysis results and prepares to launch it. For example, if the analysis reveals that the user has the intent to "order," a food delivery service application will be selected. The device prepares to pass the necessary parameters (vendor, product, etc.) to the application.

[0173] 5. Result acquisition and voice notification

[0174] The device accesses the specified API endpoint and obtains information according to the user's request. For example, a request such as "I would like to order a margherita from a food delivery service" is sent, and confirmation information about the order is collected. The obtained information is returned from the server to the device. The device then converts the received results into voice data using a speech synthesis engine. For example, a notification such as "Your order has been completed. Your margherita will arrive in 30 minutes" is generated. This voice data is played from the device's speaker, providing the user with an audible notification.

[0175] Specific examples

[0176] For example, let's say a user says, "I want to order a Margherita from a food delivery service." Here's what happens:

[0177] 1. A user says, "I want to order a margherita from a food delivery service."

[0178] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "I'd like to order a margherita from a food delivery service."

[0179] 3. The device analyzes the text using an NLP engine and extracts the parameters "Vendor: food delivery service" and "Item: Margherita."

[0180] 4. The device selects the food delivery service application and prepares to launch it.

[0181] 5. The server retrieves the "Margherita order information from the food delivery service" from the API and returns the details (order confirmation information) to the terminal.

[0182] 6. The terminal converts the execution result into voice and notifies the user by saying, "Your order has been completed. Your Margherita will arrive in 30 minutes."

[0183] 7. The device launches the food delivery service application and displays detailed information.

[0184] Prompt Sentence Examples

[0185] "Using speech recognition, we have received the text 'I would like to order a margherita from a food delivery service.' We will proceed with the order based on the order details extracted using NLP analysis (vendor: food delivery service, product: margherita). Once the order is complete, we will announce the message 'Your order is complete. Your margherita will arrive in 30 minutes.'"

[0186] In this way, the present invention provides a system that allows even elderly or less technically savvy users to easily operate a food delivery service and obtain the information they need.

[0187] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0188] Step 1:

[0189] The user speaks to the device, saying, "I'd like to order a Margherita from a food delivery service." The input is voice data, which is captured through the device's microphone. This voice data is temporarily stored in the device.

[0190] Step 2:

[0191] The device sends the captured voice data to a speech recognition engine, which converts the input voice data into text data, and the resulting text data is "I would like to order a Margherita from a food delivery service." This text data is also temporarily stored.

[0192] Step 3:

[0193] The device analyzes the text data using a natural language processing (NLP) engine. The input is converted text data, and the data is processed through analysis to identify the user's intent. Specific instruction parameters such as "Vendor: food delivery service" and "Item: Margherita" are extracted as output. The analysis results are also temporarily stored.

[0194] Step 4:

[0195] The device selects the appropriate application based on the analysis results and prepares to launch it. The input is the parameters obtained from the analysis, and the application is selected based on the data. The output is the identity of the application to be launched, for example, a food delivery service application is selected. The device prepares to pass the necessary parameters (vendor, product, etc.) to the application.

[0196] Step 5:

[0197] The terminal accesses the specified API endpoint and retrieves the information requested by the user. The input is the instruction parameters selected by the application, and a request based on them is sent to the API endpoint. The output is order confirmation information and other related data. The retrieved information is returned from the server to the terminal.

[0198] Step 6:

[0199] The device converts the received results into voice data using a speech synthesis engine. The input is the order confirmation information obtained from the API, and data processing is performed to convert it into voice data. The output is voice notification data such as "Your order is complete. Your Margherita will arrive in 30 minutes." This voice data is played from the device's speaker, providing the user with an audible notification.

[0200] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0201] The present invention is a system that captures voice input, selects and launches an appropriate application according to the user's intentions and emotions, and notifies the user of the results by voice. Specific embodiments for implementing this system are described below.

[0202] 1. Receiving voice input

[0203] The user speaks to their smartphone, saying, "I want to go from here to where I want to go." For example, they might say, "I want to go from Shibuya to Shinjuku." The device captures the voice data through a microphone. This voice data is temporarily stored on the device.

[0204] 2. Voice Recognition

[0205] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku." This text data is temporarily saved as it will be used in the next step.

[0206] 3. Natural Language Processing (NLP)

[0207] The device sends the text data to a natural language processing (NLP) engine. The NLP engine analyzes the text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted. This analysis result is also temporarily saved.

[0208] 4. Emotion recognition

[0209] The device further transmits the text data to the emotion engine. The emotion engine analyzes the text data and recognizes the user's emotion. For example, if the user is feeling stressed, the emotion engine identifies this as "stress." The emotion recognition result is also temporarily stored.

[0210] 5. Select and launch the app

[0211] Based on the analysis results, the device selects an appropriate application and prepares to launch it. For example, if the user's intention is related to "travel" and "stress" is also recognized, the device will select a transit guide application. The device will pass the necessary parameters (departure point, destination) to the application and instruct it to launch.

[0212] 6. Result acquisition and voice notification

[0213] The server accesses the specified API endpoint and retrieves the information requested by the user. For example, a request for "route from Shibuya to Shinjuku" is made, and information such as the time and fare of the next train is collected. The retrieved information is then returned from the server to the device.

[0214] The device analyzes the received results and extracts the necessary information (such as the next train time and fare). The content of the notification is adjusted based on the emotion engine's results. For example, if the user is feeling stressed, the notification content will be made concise and easy to understand.

[0215] 7. Notification

[0216] The device sends the extracted information to a speech synthesis engine and converts it into voice data. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated. This voice data is played back from the device's speaker to notify the user. The notification is made in an appropriate tone and speed based on the emotion.

[0217] Specific examples

[0218] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[0219] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[0220] 2. The device captures the voice and sends it to a speech recognition engine, which converts it into text such as "I want to go from Shibuya to Shinjuku."

[0221] 3. The device sends this text to the NLP engine, which extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[0222] 4. The device sends the text to the emotion engine, which recognizes that the user is in a "stressed state."

[0223] 5. The device selects the transit guide application, sets parameters, and launches it.

[0224] 6. The server uses the transfer information API to obtain route information and returns the next train time and fare to the terminal.

[0225] 7. The device analyzes the results, generates voice data in a concise format taking into account the stress level, and then uses a voice synthesis engine to announce, "The next train leaves at 10:30 and costs 300 yen."

[0226] 8. The device plays this audio data and notifies the user.

[0227] In this way, the present invention provides a system that combines voice recognition, natural language processing, and emotion recognition based on voice input, allowing even elderly people and users who are unfamiliar with technology to easily operate the desired application and obtain information.

[0228] The processing flow will be explained below.

[0229] Step 1:

[0230] The user speaks into the smartphone, saying, "I want to go from Shibuya to Shinjuku." This voice data is captured by the smartphone's microphone.

[0231] Step 2:

[0232] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku."

[0233] Step 3:

[0234] The device sends the converted text data to a natural language processing (NLP) engine. The NLP engine analyzes this text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted. This analysis result is temporarily saved.

[0235] Step 4:

[0236] The device then sends the text data to the emotion engine, which analyzes the text data and recognizes the user's emotion. For example, if the user is feeling stressed, the emotion engine identifies the user's emotion as "stress." The emotion recognition result is also temporarily saved.

[0237] Step 5:

[0238] The device determines which application is appropriate based on the analysis results and emotion recognition results. For example, if the user's intention is related to "travel" and "stress" is recognized, the device will select a transit guide application.

[0239] Step 6:

[0240] The terminal passes the parameters (starting point, destination) required for the selected transit information application and prepares to launch the application.

[0241] Step 7:

[0242] The server accesses the API endpoint for transit information and retrieves information according to the user's request. For example, a request such as "https: / / api.transit.com / getInfo?saddr=Shibuya&daddr=Shinjuku" is sent to retrieve information such as "The next train departs at 10:30, and the fare is 300 yen." The retrieved information is returned to the terminal in JSON format.

[0243] Step 8:

[0244] The device analyzes the information it receives and extracts the necessary information (time of the next train, fare, etc.). The content of the notification is adjusted taking into account the results of the emotion engine. For example, if the user is feeling stressed, the notification content will be made concise and easy to understand.

[0245] Step 9:

[0246] The device sends the extracted information to a speech synthesis engine, which converts it into voice data. For example, the generated voice data would say, "The next train leaves at 10:30 and costs 300 yen."

[0247] Step 10:

[0248] The device plays the generated voice data over the speaker to notify the user, allowing the user to receive the necessary information through voice. The notification content is played back in an appropriate tone and speed based on the user's emotion.

[0249] Through these steps, the system can accurately recognize the user's voice input, select an appropriate application based on the user's set intention and emotion, and effectively notify the user of the results through voice.

[0250] Example 2

[0251] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0252] Conventional speech recognition systems convert speech input into text and provide a means to select and launch applications based on the user's intent. However, they face the challenge of responding appropriately while taking into account the user's emotions. Providing appropriate and easy-to-understand notifications is particularly important for elderly users and those unfamiliar with technology. Therefore, a system that can recognize not only the user's intent but also their emotions based on speech input and adjust notification content as needed is highly desirable.

[0253] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0254] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying a user's intention, means for analyzing the text data and recognizing the user's emotion, means for selecting and launching an appropriate application based on the user's intention and emotion, means for obtaining the execution result of the application and notifying the user by voice, and means for adjusting the notification based on the user's emotion, thereby enabling elderly people and users who are unfamiliar with technology to obtain information in an optimal form according to their emotions.

[0255] "Voice input" is a method by which a user gives instructions or queries to a system using voice.

[0256] "Text data" is a data format that expresses voice input as text information.

[0257] "Analysis" is the act of processing text data and other data to extract information such as intent and emotion.

[0258] "Intention" is information indicating the purpose for which the user inputs voice.

[0259] "Emotion" is information that indicates the psychological state of the user, and includes states such as stress and joy.

[0260] An "application" is a software program designed to perform a particular function.

[0261] "Launch" is an operation that puts an application into an executable state.

[0262] The "execution result" is the output information generated after the application has performed its processing.

[0263] A "notification" is the act of providing information to a user, often in audio form.

[0264] "Adjustment" refers to the act of changing the information provided and its format to the optimum form depending on the user's condition.

[0265] This invention is a system that allows a user to input voice, analyzes the voice input, recognizes the user's intentions and emotions, and selects and launches an appropriate application. Furthermore, when notifying the user of the results of application execution by voice, it has a function to adjust the notification content based on the user's emotions.

[0266] Hardware and software used

[0267] The system uses the following hardware and software:

[0268] Audio input capture hardware: Microphones built into mobile devices such as smartphones and tablets

[0269] Speech recognition engine: Google (registered trademark) Speech-to-Text API, IBM Watson (registered trademark) Speech to Text, etc.

[0270] Natural Language Processing (NLP) engines: Google Natural Language API, Amazon Comprehend, etc.

[0271] Emotion recognition engine: Microsoft® Azure® Text Analytics for Sentiment Analysis, etc.

[0272] Application launch software: Route guidance applications such as NAVITIME and Google Maps

[0273] Speech synthesis engine: Google Cloud Text-to-Speech API, Amazon Polly, etc.

[0274] Program processing

[0275] Receiving voice input

[0276] The user speaks to the smartphone, saying, "I want to go from here to where I want to go." For example, "I want to go from Shibuya to Shinjuku." The device captures this voice data through the smartphone's microphone and temporarily stores it on the device.

[0277] Voice Recognition

[0278] The device sends the captured voice data to a speech recognition engine, which uses the Google Speech-to-Text API or similar to convert it into text data such as "I want to go from Shibuya to Shinjuku." This text data is temporarily stored for use in the next processing step.

[0279] Natural Language Processing (NLP)

[0280] The device sends the text data obtained through voice recognition to an NLP engine, which analyzes the user's intent. Using Google Natural Language API and other tools, it extracts parameters such as "Departure: Shibuya" and "Arrival: Shinjuku." The analysis results are also temporarily saved.

[0281] emotion recognition

[0282] The device then sends the obtained text data to an emotion recognition engine to recognize the user's emotions. Using Microsoft Azure Text Analytics and other tools, the device determines whether the user is feeling "stressed." The emotion recognition results are also temporarily stored.

[0283] Selecting and launching the app

[0284] Based on these analysis results, the device selects an appropriate application and prepares to launch it. For example, if the user's intention is "travel" and the emotion is recognized as "stress," a route guidance application such as NAVITIME will be selected. The device passes the necessary parameters (starting point, destination) to the application and instructs it to launch.

[0285] Result acquisition and voice notification

[0286] The server accesses the specified API endpoint and retrieves information according to the user's request. For example, it retrieves "route information from Shibuya to Shinjuku" and returns information such as the next train time and fare to the device. The device analyzes these results and adjusts the notification content taking the user's emotions into account.

[0287] notification

[0288] The device then sends the formatted notification to a speech synthesis engine, which converts it into voice data. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated and notified to the user through the device's speaker. The tone and speed of the voice are adjusted based on the user's emotions.

[0289] Specific examples

[0290] For example, let's say a user says, "I want to go from Shibuya to Shinjuku." Here's the specific operation flow:

[0291] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[0292] 2. The device captures the audio and converts it into text using the Google Speech-to-Text API: "I want to go from Shibuya to Shinjuku."

[0293] 3. The device analyzes this text using the Google Natural Language API and extracts "Departure: Shibuya" and "Destination: Shinjuku."

[0294] 4. The device analyzes emotions using Microsoft Azure Text Analytics and recognizes that the user is in a "stressed state."

[0295] 5. The device selects and launches the NAVITIME transit information application and sets the parameters.

[0296] 6. The server obtains route information using NAVITIME's API and returns the information "next train time: 10:30" and "fare: 300 yen" to the terminal.

[0297] 7. The device analyzes this information and, taking into account the stress level, generates a concise voice message such as, "The next train leaves at 10:30 and costs 300 yen."

[0298] 8. The device plays the generated audio data and notifies the user.

[0299] Prompt Sentence Examples

[0300] "Please tell me when the next train leaves and how much it costs."

[0301] This system is particularly useful for elderly people and users who are unfamiliar with technology, as it can understand the user's intentions and emotions and provide appropriate information via voice, simply by inputting the user's voice.

[0302] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0303] Step 1:

[0304] The user speaks to the smartphone, saying, "From where do I want to go to where?" For example, "I want to go from Shibuya to Shinjuku." The device captures this voice data through the smartphone's microphone. The voice data is input and output in a format that is temporarily stored within the device. Specifically, the smartphone's microphone converts the voice signal into digital data, and then records that digital data.

[0305] Step 2:

[0306] The device sends the captured voice data to a voice recognition engine. Here, the Google Speech-to-Text API is used. Voice data is sent to this API as input, and the converted text data is received as output. For example, a voice saying "I want to go from Shibuya to Shinjuku" is converted into text data saying "I want to go from Shibuya to Shinjuku." In concrete terms, the device sends voice data as an API request and receives text data as an API response.

[0307] Step 3:

[0308] The device sends the text data obtained by speech recognition to the NLP engine. Here, the Google Natural Language API is used. The text data is sent as input to the NLP engine, and the analyzed intent and parameters are received as output. For example, parameters such as "Departure: Shibuya" and "Arrival: Shinjuku" are extracted from the text. Specifically, the device sends the text data as an API request and receives response data with the intent and parameters extracted.

[0309] Step 4:

[0310] The device sends the obtained text data to an emotion recognition engine. In this case, Microsoft Azure Text Analytics is used. The text data is sent as input to the emotion recognition engine, and the recognized emotion information is received as output. For example, if the user is feeling "stressed," it is identified as "stress." Specifically, the device sends the text data as an API request and receives the emotion recognition results as response data.

[0311] Step 5:

[0312] The device selects an appropriate application based on the results of natural language processing and emotion recognition, and prepares to launch it. For example, if the user's intent is "travel" and "stress" is recognized, a route guidance application such as NAVITIME will be selected. The device also prepares to pass the necessary parameters (starting point, destination) to the application. Specifically, it determines the application name and parameters, and generates an API request to launch the application.

[0313] Step 6:

[0314] The server accesses the specified API endpoint and retrieves the information requested by the user. For example, it uses the NAVITIME API to retrieve "route information from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. The collected data is then sent back to the device. Specifically, it sends an information request to the API endpoint and retrieves route information as response data.

[0315] Step 7:

[0316] The device analyzes the received information and adjusts the notification content taking into account the user's emotions. For example, if the user is feeling "stressed," the notification content is converted into a concise and easy-to-understand format. Route information (next train time, fare, etc.) is input, and the adjusted voice notification content is output. Specifically, the device analyzes the response data and reformats the text according to the user's emotions.

[0317] Step 8:

[0318] The device sends the adjusted notification content to a speech synthesis engine and converts it into voice data. Here, the Google Cloud Text-to-Speech API is used. The text data of the notification content is sent as input to the speech synthesis engine, and voice data is received as output. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated. Specifically, the text data is sent as a request to the speech synthesis API, and voice data is received as a response.

[0319] Step 9:

[0320] The device plays the generated voice data and notifies the user. The voice data is input and output as voice through the smartphone's speaker. Specifically, the device's voice playback function is used to play the voice from the speaker.

[0321] (Application example 2)

[0322] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] In recent years, there has been a demand for systems that can analyze user intent based on voice input and launch the appropriate application. Interfaces that combine voice input and analysis are particularly useful for users who want to easily order delivery or perform complex operations. However, existing systems often struggle to provide appropriate suggestions that take the user's emotions into account, and the methods of notifying users of information are often not optimal. Therefore, it is necessary to provide a system that can analyze the user's intent and emotions, select and launch the appropriate application based on the results, and notify the user of the results via voice.

[0324] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0325] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying the user's intention and emotion, means for selecting and launching an appropriate application based on the user's intention and emotion, and means for obtaining the execution result of the application and notifying the user by voice, thereby enabling delivery orders and other complex operations to be performed simply and optimally through voice input.

[0326] "Voice input" is the process of capturing the user's spoken words and storing them as digital data.

[0327] "Text data" is an information format in which voice data is converted into a string of characters.

[0328] "User intent" refers to the goal or request that the user is trying to achieve through voice input.

[0329] "User emotions" refers to the psychological and emotional state that can be read from the user's comments.

[0330] "Application selection and launch" is the process of choosing and running the most appropriate software program based on the user's intent and emotions.

[0331] "Application Results" refers to the final information or output provided by a selected and launched application.

[0332] "Voice notification" refers to the act of communicating the results of an application's execution to the user as voice using speech synthesis technology.

[0333] A "transit application" is a software program that allows a user to search for and provide route information from one point to another.

[0334] A "food delivery application" is a software program that allows a user to order food and delivers the food to a specified location.

[0335] An "audio input device" is hardware that captures a user's voice and processes it as a digital signal.

[0336] A "speech recognition engine" is software or algorithms for analyzing captured voice data and converting it into text form.

[0337] This system analyzes the user's intentions and emotions based on the user's voice input, selects and launches an appropriate application, and notifies the user of the results via voice. This system is primarily applied in the food delivery field, helping users to easily order food through voice input.

[0338] Hardware and software used

[0339] Hardware:

[0340] Audio input device: A device that captures a user's voice. Examples include smart glasses and smartphones.

[0341] Server: A central processing unit for processing and analyzing data.

[0342] software:

[0343] Speech recognition engine: Software for converting voice data into text data, such as the Google Cloud Speech-to-Text API.

[0344] Natural Language Processing (NLP) engine: Software for analyzing text data and identifying user intent, such as the Google Cloud Natural Language API.

[0345] Emotion recognition engine: Software that analyzes user emotions from text data, such as IBM Watson Tone Analyzer.

[0346] Speech synthesis engine: Software for converting text data into speech, such as the Google Text-to-Speech API.

[0347] Processing flow

[0348] 1. Audio input capture:

[0349] A voice input device (smart glasses or a smartphone) captures the user's voice, for example, the user says, "I'd like to order a pizza."

[0350] 2. Speech Recognition:

[0351] The captured voice data is sent to a voice recognition engine, which converts the voice into text data.

[0352] 3. Natural Language Processing (NLP):

[0353] The text data is sent to an NLP engine, which analyzes the user's intent and extracts the item to be ordered (in this example, "pizza").

[0354] 4. Emotion recognition:

[0355] Furthermore, the text data is sent to an emotion recognition engine to analyze the user's emotions (e.g., stress state).

[0356] 5. Select and launch the application:

[0357] Based on the analysis results, the system selects and launches an appropriate food delivery application. For example, if the user is feeling stressed, an application that allows for easy ordering will be selected.

[0358] 6. Obtaining and notifying order results:

[0359] The order information is acquired and then a voice synthesis engine delivers it to the user in a relaxed tone, depending on the user's emotions.

[0360] Specific examples

[0361] For example, if a user says "I'd like to order a pizza":

[0362] 1. User: Says, "I want to order a pizza."

[0363] 2. Voice input device: Captures voice and sends it to the voice recognition engine.

[0364] 3. Speech recognition engine: Converts voice data into text data such as "I would like to order pizza."

[0365] 4. Natural language processing engine: Analyzes text data and extracts "Order: Pizza".

[0366] 5. Emotion recognition engine: Analyzes text data and recognizes when the user is in a "stressed state."

[0367] 6. Server: Select a food delivery application (e.g., food delivery service).

[0368] 7. Server: The food delivery application confirms the user's order.

[0369] 8. Speech synthesis engine: Generates the order result as a voice saying, "The pizza will arrive in 20 minutes."

[0370] 9. Audio input device: Plays audio data to the user and notifies them.

[0371] Prompt Sentence Examples

[0372] An example of a prompt is:

[0373] The user says, "I want to order a pizza."

[0374] The voice recognition engine converts the sentence "I would like to order a pizza" into text.

[0375] The NLP engine extracts "What to order: Pizza."

[0376] The emotion engine recognizes the user as being in a "stressed state."

[0377] Select a food delivery application considering stress conditions.

[0378] Suggests pizza types and potential stores.

[0379] Select "Margherita Pizza" and confirm your order.

[0380] The order information is retrieved and a voice notification is given saying, "Your pizza will arrive in 20 minutes."

[0381] The audio input device has completed audio notification.

[0382] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0383] Step 1:

[0384] Voice input capture

[0385] A user speaks into a voice input device (e.g., smart glasses or a smartphone) saying, "I want to order a pizza." The voice input device captures the voice through a microphone and temporarily stores it as voice data.

[0386] Input: User's voice

[0387] Output: Audio data

[0388] Step 2:

[0389] Voice Recognition

[0390] The device sends the captured voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data and returns the text data to the device.

[0391] Input: Audio data

[0392] Output: Text data

[0393] Step 3:

[0394] Natural Language Processing (NLP)

[0395] The device sends the text data to a natural language processing engine (e.g., Google Cloud Natural Language), which analyzes the text data, identifies the user's intent, such as "What to order: Pizza," and returns the analysis results to the device.

[0396] Input: Text data

[0397] Output: User intent (e.g., "Order: Pizza")

[0398] Step 4:

[0399] emotion recognition

[0400] The device further transmits the text data to an emotion recognition engine (e.g., IBM Watson Tone Analyzer), which analyzes the text data and identifies the user's emotion (e.g., "stressed"), and returns the emotion recognition result to the device.

[0401] Input: Text data

[0402] Output: Emotion analysis result (e.g. "stressed")

[0403] Step 5:

[0404] Selecting and Launching an Application

[0405] Based on the analysis results (user intentions and emotions), the device selects an appropriate application (e.g., a food delivery application) from the server, sets the necessary parameters, and launches the application. For example, if the user is feeling stressed, the device will prioritize a food delivery application that is easy to operate.

[0406] Input: User intent, sentiment analysis results

[0407] Output: Launch of selected application

[0408] Step 6:

[0409] Obtaining order results

[0410] The server will then contact the selected application to confirm the order and obtain delivery information (arrival time, shipping costs, etc.), which involves using the application's API to request the required information and return the results.

[0411] Input: Application request

[0412] Output: Delivery information (e.g. arrival time, shipping fee)

[0413] Step 7:

[0414] Audio notifications

[0415] The device sends the acquired delivery information to a speech synthesis engine, which generates concise and easy-to-understand voice data. For example, it generates a voice that says, "Your pizza will arrive in 20 minutes." The device plays this voice data and notifies the user. The notification tone and speed are adjusted based on the user's emotions (e.g., stress level).

[0416] Input: Delivery information

[0417] Output: Audio notification

[0418] Specific examples

[0419] Example prompt sentence:

[0420] The user says, "I want to order a pizza."

[0421] The voice recognition engine converts the sentence "I would like to order a pizza" into text.

[0422] The NLP engine extracts "What to order: Pizza."

[0423] The emotion engine recognizes the user as being in a "stressed state."

[0424] Select a food delivery application considering stress conditions.

[0425] Suggests pizza types and potential stores.

[0426] Select "Margherita Pizza" and confirm your order.

[0427] The order information is retrieved and a voice notification is given saying, "Your pizza will arrive in 20 minutes."

[0428] The audio input device has completed audio notification.

[0429] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0430] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0431] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0432] [Second embodiment]

[0433] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0434] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0435] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0436] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0437] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0438] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0439] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0440] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0441] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0442] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0443] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0444] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0445] This invention is a system that captures voice input, selects and launches an appropriate application according to the user's intention, and notifies the user of the result by voice. Specific embodiments for implementing this system are described below.

[0446] 1. Receiving voice input

[0447] The user speaks to their smartphone, saying, "I want to go from here to where I want to go." For example, they might say, "I want to go from Shibuya to Shinjuku." The device captures the voice data through a microphone. This voice data is temporarily stored on the device.

[0448] 2. Voice Recognition

[0449] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice input of "I want to go from Shibuya to Shinjuku" is converted into text "I want to go from Shibuya to Shinjuku." This text data is temporarily saved as it will be used in the next step.

[0450] 3. Natural Language Processing (NLP)

[0451] The device analyzes the text data using a natural language processing (NLP) engine. The NLP engine identifies the user's intent and extracts the necessary parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" may be extracted. The analysis results are also temporarily saved.

[0452] 4. Select and launch the app

[0453] The device selects an appropriate application based on the analysis results and prepares to launch it. For example, if the analysis determines that the user has the intent to "move," the device selects a transit guide application. The device prepares to pass the necessary parameters (such as departure and arrival points) to the application.

[0454] 5. Result acquisition and voice notification

[0455] The server accesses the specified API endpoint and obtains the information requested by the user. For example, a user requests "route from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. The obtained information is then returned from the server to the device.

[0456] The device converts the received result into voice data using a speech synthesis engine. For example, a notification such as "The next train leaves at 10:30 and costs 300 yen" is generated. This voice data is played back from the device's speaker, providing the user with an audible notification.

[0457] Specific examples

[0458] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[0459] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[0460] 2. The device captures the voice and converts it into text using a speech recognition engine: "I want to go from Shibuya to Shinjuku."

[0461] 3. The device analyzes the text using an NLP engine and extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[0462] 4. The device selects the transit information application and prepares to launch it.

[0463] 5. The server retrieves route information from Shibuya to Shinjuku from the API and returns details (next train time, fare, etc.) to the terminal.

[0464] 6. The terminal converts the result of the operation into voice and announces, "The next train leaves at 10:30 and costs 300 yen."

[0465] 7. The device launches the transit information application and displays detailed information.

[0466] In this way, the present invention provides a system that enables even elderly people and users who are unfamiliar with technology to easily operate a desired application and obtain necessary information.

[0467] The processing flow will be explained below.

[0468] Step 1:

[0469] The user speaks into the smartphone to input the necessary instructions, for example, "I want to go from Shibuya to Shinjuku." This voice data is then captured by the smartphone's microphone.

[0470] Step 2:

[0471] The device sends the captured voice data to its internal voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku."

[0472] Step 3:

[0473] The device sends the converted text data to a natural language processing (NLP) engine, which analyzes the text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted.

[0474] Step 4:

[0475] The device checks the analysis results and determines which application is appropriate. In this case, since the user's intention is related to "travel," the transit guide application is selected.

[0476] Step 5:

[0477] The terminal prepares to launch the selected transit information application by passing necessary parameters (e.g., departure point, arrival point).

[0478] Step 6:

[0479] The server processes API requests for transit information. For example, a request like "https: / / api.transit.com / getInfo?saddr=Shibuya&daddr=Shinjuku" is sent to retrieve information such as "The next train departs at 10:30 and costs 300 yen."

[0480] Step 7:

[0481] The server returns the acquired information to the terminal in JSON format.

[0482] Step 8:

[0483] The terminal analyzes the information it receives and extracts the necessary information (e.g., the time of the next train, the fare).

[0484] Step 9:

[0485] The device sends the extracted information to a speech synthesis engine, which converts it into voice data. For example, the generated voice data would say, "The next train leaves at 10:30 and costs 300 yen."

[0486] Step 10:

[0487] The terminal plays the generated voice data and notifies the user, so that the user receives the necessary information by voice.

[0488] In this way, this system combines voice recognition and natural language processing to select and launch the appropriate application based on the user's voice input, and then returns the results in voice form, implementing a series of processes.

[0489] Example 1

[0490] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0491] In modern society, when elderly people and users unfamiliar with technology use smartphones to obtain information, the operation is often complicated, making it difficult to obtain information. In particular, when obtaining information using voice input, there is a demand for an efficient system that can smoothly and intuitively perform the entire process from voice recognition to application selection and notification of results.

[0492] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0493] In this invention, the server includes: means for capturing voice input; means for converting the voice input into text data; means for analyzing the text data and identifying the user's intent; means for selecting and launching appropriate software based on the user's intent; means for acquiring the execution results of the software and notifying the user by voice; means for capturing the voice input with a microphone and converting it into text data using a voice recognition engine; means for analyzing the text data with a natural language processing engine and extracting necessary parameters; means for sending a request to the server based on the parameters and acquiring necessary information from a specified API endpoint; and means for converting the acquired information into voice data using a voice synthesis engine and notifying the user. This enables even elderly people and users who are not familiar with technology to easily acquire desired information using only voice input.

[0494] "Voice input" refers to the act of a user giving instructions by voice, which is captured as digital voice data via a microphone.

[0495] "Text data" refers to data that has been converted into text form after voice input has been analyzed by a voice recognition engine.

[0496] "Analysis" refers to the process of using a natural language processing engine to identify the user's intent and required information from text data.

[0497] "User intent" refers to the action or information the user is seeking, as determined from the voice input.

[0498] "Software" refers to application programs that are selected and launched based on the user's intent.

[0499] A "microphone" is a hardware device for capturing audio input.

[0500] A "speech recognition engine" is a software or hardware mechanism for converting voice data captured by a microphone into text data.

[0501] A "natural language processing engine" is a software mechanism that analyzes text data and extracts the user's intent and necessary parameters.

[0502] "Parameters" refer to necessary information extracted from the analyzed text data, specifically, the departure and arrival points.

[0503] A "server" is a program or device with a specific function that retrieves information from a specified API based on a user request.

[0504] "API endpoint" refers to the interface of an external service that a server accesses to obtain information.

[0505] A "speech synthesis engine" is software for converting text data into voice data.

[0506] "Voice notification" refers to the act of notifying a user of information using voice data generated by a voice synthesis engine.

[0507] The present invention relates to a system that allows a user to obtain information by voice input and notifies the user of the information by voice. Specific means for implementing this system will be described below.

[0508] overview

[0509] The system of this invention captures and analyzes voice input, selects and launches appropriate software according to the user's intention, and notifies the user of the results by voice, making it easy for even elderly users and those unfamiliar with technology to obtain information.

[0510] Hardware and software configuration

[0511] 1. Receiving voice input

[0512] The user issues specific instructions to the smartphone. For example, the user says, "I want to go from Shibuya to Shinjuku." The device's microphone captures this voice input and generates voice data. This voice data is stored in the device's temporary memory.

[0513] 2. Voice Recognition

[0514] The device sends the voice data to a voice recognition engine (for example, a commercial voice recognition API). The voice recognition engine converts the captured voice data into text data. The converted text data is in the format of, for example, "I want to go from Shibuya to Shinjuku." This text data is also temporarily stored in the device.

[0515] 3. Natural Language Processing (NLP)

[0516] The device sends the text data to a natural language processing (NLP) engine (e.g., a commercial NLP API). The NLP engine analyzes the text data and extracts important parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" is extracted. This parameter information is also temporarily stored on the device.

[0517] 4. Select and launch the app

[0518] The device selects the appropriate software based on the analysis results. For example, if the user has the intent to "move," the device selects transit information software (e.g., a commercial transit information app). The device prepares to launch the selected application and collects the necessary parameters (start and end locations).

[0519] 5. Result acquisition and voice notification

[0520] The device sends the analysis results (parameters) to the server, and the server accesses the specified API endpoint (for example, a commercial API endpoint) to obtain the required information. For example, the server requests "route information from Shibuya to Shinjuku" and obtains the next train time, fare, etc. The obtained information is returned from the server to the device.

[0521] The device converts the acquired information into speech using a speech synthesis engine (for example, a commercial speech synthesis API). The speech data will be in the form of "The next train leaves at 10:30 and costs 300 yen." This speech data is played back to the user through the device's speaker.

[0522] Specific examples

[0523] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[0524] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[0525] 2. The device's microphone captures the audio data and stores it in temporary memory.

[0526] 3. The device sends the voice data to a speech recognition engine and converts it into text.

[0527] 4. The device sends the text data to a natural language processing engine, which extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[0528] 5. The device selects the transit information software and prepares to launch it.

[0529] 6. The terminal sends the extracted parameters to the server.

[0530] 7. The server requests the API endpoint and obtains the route information.

[0531] 8. The server returns the acquired information (next train time, fare) to the terminal.

[0532] 9. The information received by the device is converted into voice using a speech synthesis engine, and the user is notified that "The next train leaves at 10:30 and costs 300 yen."

[0533] 10. The device will launch the transit information software and display detailed information.

[0534] Prompt Sentence Examples

[0535] "If I say I want to go from Shibuya to Shinjuku, please explain the specific process."

[0536] In this way, the present invention enables even elderly people and users who are unfamiliar with technology to quickly and easily obtain desired information using only voice input.

[0537] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0538] Step 1: Receiving voice input

[0539] The user issues specific instructions to the smartphone. For example, the user might say, "I want to go from Shibuya to Shinjuku." The device's microphone captures this voice input and generates digital audio data, which is then stored in the device's temporary memory.

[0540] Input: User voice input

[0541] Output: Audio data

[0542] Step 2: Voice Recognition

[0543] The device sends the voice data to a voice recognition engine (for example, a commercial voice recognition API). The voice recognition engine converts the captured voice data into text data. For example, the text data is converted into text data in the format "I want to go from Shibuya to Shinjuku." This text data is temporarily stored in the device.

[0544] Input: Audio data

[0545] Output: Text data

[0546] Step 3: Natural Language Processing (NLP)

[0547] The device sends the text data to a natural language processing (NLP) engine (e.g., a commercial NLP API). The NLP engine analyzes the text data and extracts important parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" is extracted. This parameter information is also temporarily stored on the device.

[0548] Input: Text data

[0549] Output: Extracted parameters (e.g. "Departure: Shibuya", "Destination: Shinjuku")

[0550] Step 4: Select and launch the app

[0551] The device selects the appropriate software based on the analysis results. For example, if the analysis results indicate that the user has the intention of "moving," the device selects transit information software. Next, the device prepares to launch the selected application and collects the necessary parameters (starting point and destination).

[0552] Input: Extracted parameters

[0553] Output: Software selection and launch preparation

[0554] Step 5: Get the results and hear the audio

[0555] The device sends the analysis results (parameters) to the server. The server accesses the specified API endpoint and obtains the information requested by the user. For example, the server requests "route information from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. This collected information is returned from the server to the device. The device then converts the obtained information into voice data using a speech synthesis engine and notifies the user, "The next train leaves at 10:30 and costs 300 yen."

[0556] Input: Analysis results (parameters)

[0557] Output: Acquired information and its spoken notification

[0558] In this way, a series of steps is realized, starting with the user's voice input, followed by voice recognition, natural language processing, application selection and launch, and result acquisition and voice notification. Appropriate data processing and data calculation are performed at each step, ultimately resulting in a system that provides the user with the information they need intuitively and efficiently.

[0559] (Application example 1)

[0560] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0561] Current food delivery services require users to manually operate the application, which is particularly cumbersome for elderly people and those who are not familiar with technology. Also, visually impaired users have difficulty operating smartphones. Therefore, there is a need for a method that allows everyone to easily use food delivery services.

[0562] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0563] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying the user's intention, means for selecting and launching an appropriate application based on the user's intention, means for acquiring the execution result of the application and notifying the user by voice, and means for including an application related to the food delivery service. This makes it possible to use the food delivery service with just voice input, and enables anyone to easily perform and use the ordering procedure.

[0564] "Means for capturing audio input" refers to a device and its associated software for detecting and recording audio input.

[0565] "Means for converting speech input into text data" refers to software and algorithms for converting speech signals into written information.

[0566] "Means for analyzing text data and identifying user intent" refers to natural language processing technology and related software for analyzing text data and understanding the user's intended operations and requests.

[0567] "Means for selecting and launching an appropriate application" refers to a processing system for automatically selecting and launching a particular application based on a user's intent.

[0568] "Means for obtaining the results of application execution and notifying the user by voice" refers to technology and devices for collecting the results or data of an executed application and providing it to the user as voice.

[0569] "Food delivery service application" refers to a software application that allows a user to voice-order food and process the delivery.

[0570] "Device for capturing an acoustic signal" refers to a hardware device, such as a microphone, for obtaining audio input.

[0571] "Speech recognition engine" refers to software and algorithms for processing captured acoustic signals and converting them into text data.

[0572] This invention is a system that captures voice input, selects and launches an appropriate application according to the user's intention, and notifies the user of the results by voice. Specific embodiments for implementing this system are described below.

[0573] 1. Receiving voice input

[0574] A user speaks to the device, saying, "I want to order a pizza," or, for example, "I want to order a Margherita from a food delivery service." The device captures the voice data through a microphone, a device that captures acoustic signals, and this voice data is temporarily stored within the device.

[0575] 2. Voice Recognition

[0576] The device sends the captured voice data to a speech recognition engine, which converts the voice data into text data. For example, a voice input of "I would like to order a margherita from a food delivery service" is converted into text "I would like to order a margherita from a food delivery service." This text data is temporarily stored for use in the next step.

[0577] 3. Natural Language Processing (NLP)

[0578] The device analyzes the text data using a natural language processing (NLP) engine. The NLP engine identifies the user's intent and extracts necessary parameters. For example, information such as "Vendor: food delivery service" and "Item: Margherita" may be extracted. The analysis results are also temporarily stored.

[0579] 4. Select and launch the app

[0580] The device selects an appropriate application based on the analysis results and prepares to launch it. For example, if the analysis reveals that the user has the intent to "order," a food delivery service application will be selected. The device prepares to pass the necessary parameters (vendor, product, etc.) to the application.

[0581] 5. Result acquisition and voice notification

[0582] The device accesses the specified API endpoint and obtains information according to the user's request. For example, a request such as "I would like to order a margherita from a food delivery service" is sent, and confirmation information about the order is collected. The obtained information is returned from the server to the device. The device then converts the received results into voice data using a speech synthesis engine. For example, a notification such as "Your order has been completed. Your margherita will arrive in 30 minutes" is generated. This voice data is played from the device's speaker, providing the user with an audible notification.

[0583] Specific examples

[0584] For example, let's say a user says, "I want to order a Margherita from a food delivery service." Here's what happens:

[0585] 1. A user says, "I want to order a margherita from a food delivery service."

[0586] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "I'd like to order a margherita from a food delivery service."

[0587] 3. The device analyzes the text using an NLP engine and extracts the parameters "Vendor: food delivery service" and "Item: Margherita."

[0588] 4. The device selects the food delivery service application and prepares to launch it.

[0589] 5. The server retrieves the "Margherita order information from the food delivery service" from the API and returns the details (order confirmation information) to the terminal.

[0590] 6. The terminal converts the execution result into voice and notifies the user by saying, "Your order has been completed. Your Margherita will arrive in 30 minutes."

[0591] 7. The device launches the food delivery service application and displays detailed information.

[0592] Prompt Sentence Examples

[0593] "Using speech recognition, we have received the text 'I would like to order a margherita from a food delivery service.' We will proceed with the order based on the order details extracted using NLP analysis (vendor: food delivery service, product: margherita). Once the order is complete, we will announce the message 'Your order is complete. Your margherita will arrive in 30 minutes.'"

[0594] In this way, the present invention provides a system that allows even elderly or less technically savvy users to easily operate a food delivery service and obtain the information they need.

[0595] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0596] Step 1:

[0597] The user speaks to the device, saying, "I'd like to order a Margherita from a food delivery service." The input is voice data, which is captured through the device's microphone. This voice data is temporarily stored in the device.

[0598] Step 2:

[0599] The device sends the captured voice data to a speech recognition engine, which converts the input voice data into text data, and the resulting text data is "I would like to order a Margherita from a food delivery service." This text data is also temporarily stored.

[0600] Step 3:

[0601] The device analyzes the text data using a natural language processing (NLP) engine. The input is converted text data, and the data is processed through analysis to identify the user's intent. Specific instruction parameters such as "Vendor: food delivery service" and "Item: Margherita" are extracted as output. The analysis results are also temporarily stored.

[0602] Step 4:

[0603] The device selects the appropriate application based on the analysis results and prepares to launch it. The input is the parameters obtained from the analysis, and the application is selected based on the data. The output is the identity of the application to be launched, for example, a food delivery service application is selected. The device prepares to pass the necessary parameters (vendor, product, etc.) to the application.

[0604] Step 5:

[0605] The terminal accesses the specified API endpoint and retrieves the information requested by the user. The input is the instruction parameters selected by the application, and a request based on them is sent to the API endpoint. The output is order confirmation information and other related data. The retrieved information is returned from the server to the terminal.

[0606] Step 6:

[0607] The device converts the received results into voice data using a speech synthesis engine. The input is the order confirmation information obtained from the API, and data processing is performed to convert it into voice data. The output is voice notification data such as "Your order is complete. Your Margherita will arrive in 30 minutes." This voice data is played from the device's speaker, providing the user with an audible notification.

[0608] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0609] The present invention is a system that captures voice input, selects and launches an appropriate application according to the user's intentions and emotions, and notifies the user of the results by voice. Specific embodiments for implementing this system are described below.

[0610] 1. Receiving voice input

[0611] The user speaks to their smartphone, saying, "I want to go from here to where I want to go." For example, they might say, "I want to go from Shibuya to Shinjuku." The device captures the voice data through a microphone. This voice data is temporarily stored on the device.

[0612] 2. Voice Recognition

[0613] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku." This text data is temporarily saved as it will be used in the next step.

[0614] 3. Natural Language Processing (NLP)

[0615] The device sends the text data to a natural language processing (NLP) engine. The NLP engine analyzes the text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted. This analysis result is also temporarily saved.

[0616] 4. Emotion recognition

[0617] The device further transmits the text data to the emotion engine. The emotion engine analyzes the text data and recognizes the user's emotion. For example, if the user is feeling stressed, the emotion engine identifies this as "stress." The emotion recognition result is also temporarily stored.

[0618] 5. Select and launch the app

[0619] Based on the analysis results, the device selects an appropriate application and prepares to launch it. For example, if the user's intention is related to "travel" and "stress" is also recognized, the device will select a transit guide application. The device will pass the necessary parameters (departure point, destination) to the application and instruct it to launch.

[0620] 6. Result acquisition and voice notification

[0621] The server accesses the specified API endpoint and retrieves the information requested by the user. For example, a request for "route from Shibuya to Shinjuku" is made, and information such as the time and fare of the next train is collected. The retrieved information is then returned from the server to the device.

[0622] The device analyzes the received results and extracts the necessary information (such as the next train time and fare). The content of the notification is adjusted based on the emotion engine's results. For example, if the user is feeling stressed, the notification content will be made concise and easy to understand.

[0623] 7. Notification

[0624] The device sends the extracted information to a speech synthesis engine and converts it into voice data. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated. This voice data is played back from the device's speaker to notify the user. The notification is made in an appropriate tone and speed based on the emotion.

[0625] Specific examples

[0626] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[0627] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[0628] 2. The device captures the voice and sends it to a speech recognition engine, which converts it into text such as "I want to go from Shibuya to Shinjuku."

[0629] 3. The device sends this text to the NLP engine, which extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[0630] 4. The device sends the text to the emotion engine, which recognizes that the user is in a "stressed state."

[0631] 5. The device selects the transit guide application, sets parameters, and launches it.

[0632] 6. The server uses the transfer information API to obtain route information and returns the next train time and fare to the terminal.

[0633] 7. The device analyzes the results, generates voice data in a concise format taking into account the stress level, and then uses a voice synthesis engine to announce, "The next train leaves at 10:30 and costs 300 yen."

[0634] 8. The device plays this audio data and notifies the user.

[0635] In this way, the present invention provides a system that combines voice recognition, natural language processing, and emotion recognition based on voice input, allowing even elderly people and users who are unfamiliar with technology to easily operate the desired application and obtain information.

[0636] The processing flow will be explained below.

[0637] Step 1:

[0638] The user speaks into the smartphone, saying, "I want to go from Shibuya to Shinjuku." This voice data is captured by the smartphone's microphone.

[0639] Step 2:

[0640] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku."

[0641] Step 3:

[0642] The device sends the converted text data to a natural language processing (NLP) engine. The NLP engine analyzes this text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted. This analysis result is temporarily saved.

[0643] Step 4:

[0644] The device then sends the text data to the emotion engine, which analyzes the text data and recognizes the user's emotion. For example, if the user is feeling stressed, the emotion engine identifies the user's emotion as "stress." The emotion recognition result is also temporarily saved.

[0645] Step 5:

[0646] The device determines which application is appropriate based on the analysis results and emotion recognition results. For example, if the user's intention is related to "travel" and "stress" is recognized, the device will select a transit guide application.

[0647] Step 6:

[0648] The terminal passes the parameters (starting point, destination) required for the selected transit information application and prepares to launch the application.

[0649] Step 7:

[0650] The server accesses the API endpoint for transit information and retrieves information according to the user's request. For example, a request such as "https: / / api.transit.com / getInfo?saddr=Shibuya&daddr=Shinjuku" is sent to retrieve information such as "The next train departs at 10:30, and the fare is 300 yen." The retrieved information is returned to the terminal in JSON format.

[0651] Step 8:

[0652] The device analyzes the information it receives and extracts the necessary information (time of the next train, fare, etc.). The content of the notification is adjusted taking into account the results of the emotion engine. For example, if the user is feeling stressed, the notification content will be made concise and easy to understand.

[0653] Step 9:

[0654] The device sends the extracted information to a speech synthesis engine, which converts it into voice data. For example, the generated voice data would say, "The next train leaves at 10:30 and costs 300 yen."

[0655] Step 10:

[0656] The device plays the generated voice data over the speaker to notify the user, allowing the user to receive the necessary information through voice. The notification content is played back in an appropriate tone and speed based on the user's emotion.

[0657] Through these steps, the system can accurately recognize the user's voice input, select an appropriate application based on the user's set intention and emotion, and effectively notify the user of the results through voice.

[0658] Example 2

[0659] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0660] Conventional speech recognition systems convert speech input into text and provide a means to select and launch applications based on the user's intent. However, they face the challenge of responding appropriately while taking into account the user's emotions. Providing appropriate and easy-to-understand notifications is particularly important for elderly users and those unfamiliar with technology. Therefore, a system that can recognize not only the user's intent but also their emotions based on speech input and adjust notification content as needed is highly desirable.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0662] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying a user's intention, means for analyzing the text data and recognizing the user's emotion, means for selecting and launching an appropriate application based on the user's intention and emotion, means for obtaining the execution result of the application and notifying the user by voice, and means for adjusting the notification based on the user's emotion, thereby enabling elderly people and users who are unfamiliar with technology to obtain information in an optimal form according to their emotions.

[0663] "Voice input" is a method by which a user gives instructions or queries to a system using voice.

[0664] "Text data" is a data format that expresses voice input as text information.

[0665] "Analysis" is the act of processing text data and other data to extract information such as intent and emotion.

[0666] "Intention" is information indicating the purpose for which the user inputs voice.

[0667] "Emotion" is information that indicates the psychological state of the user, and includes states such as stress and joy.

[0668] An "application" is a software program designed to perform a particular function.

[0669] "Launch" is an operation that puts an application into an executable state.

[0670] The "execution result" is the output information generated after the application has performed its processing.

[0671] A "notification" is the act of providing information to a user, often in audio form.

[0672] "Adjustment" refers to the act of changing the information provided and its format to the optimum form depending on the user's condition.

[0673] This invention is a system that allows a user to input voice, analyzes the voice input, recognizes the user's intentions and emotions, and selects and launches an appropriate application. Furthermore, when notifying the user of the results of application execution by voice, it has a function to adjust the notification content based on the user's emotions.

[0674] Hardware and software used

[0675] The system uses the following hardware and software:

[0676] Audio input capture hardware: Microphones built into mobile devices such as smartphones and tablets

[0677] Speech recognition engine: Google Speech-to-Text API, IBM Watson Speech to Text, etc.

[0678] Natural Language Processing (NLP) engines: Google Natural Language API, Amazon Comprehend, etc.

[0679] Emotion recognition engine: Microsoft Azure Text Analytics for Sentiment Analysis, etc.

[0680] Application launch software: Route guidance applications such as NAVITIME and Google Maps

[0681] Speech synthesis engine: Google Cloud Text-to-Speech API, Amazon Polly, etc.

[0682] Program processing

[0683] Receiving voice input

[0684] The user speaks to the smartphone, saying, "I want to go from here to where I want to go." For example, "I want to go from Shibuya to Shinjuku." The device captures this voice data through the smartphone's microphone and temporarily stores it on the device.

[0685] Voice Recognition

[0686] The device sends the captured voice data to a speech recognition engine, which uses the Google Speech-to-Text API or similar to convert it into text data such as "I want to go from Shibuya to Shinjuku." This text data is temporarily stored for use in the next processing step.

[0687] Natural Language Processing (NLP)

[0688] The device sends the text data obtained through voice recognition to an NLP engine, which analyzes the user's intent. Using Google Natural Language API and other tools, it extracts parameters such as "Departure: Shibuya" and "Arrival: Shinjuku." The analysis results are also temporarily saved.

[0689] emotion recognition

[0690] The device then sends the obtained text data to an emotion recognition engine to recognize the user's emotions. Using Microsoft Azure Text Analytics and other tools, the device determines whether the user is feeling "stressed." The emotion recognition results are also temporarily stored.

[0691] Selecting and launching the app

[0692] Based on these analysis results, the device selects an appropriate application and prepares to launch it. For example, if the user's intention is "travel" and the emotion is recognized as "stress," a route guidance application such as NAVITIME will be selected. The device passes the necessary parameters (starting point, destination) to the application and instructs it to launch.

[0693] Result acquisition and voice notification

[0694] The server accesses the specified API endpoint and retrieves information according to the user's request. For example, it retrieves "route information from Shibuya to Shinjuku" and returns information such as the next train time and fare to the device. The device analyzes these results and adjusts the notification content taking the user's emotions into account.

[0695] notification

[0696] The device then sends the formatted notification to a speech synthesis engine, which converts it into voice data. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated and notified to the user through the device's speaker. The tone and speed of the voice are adjusted based on the user's emotions.

[0697] Specific examples

[0698] For example, let's say a user says, "I want to go from Shibuya to Shinjuku." Here's the specific operation flow:

[0699] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[0700] 2. The device captures the audio and converts it into text using the Google Speech-to-Text API: "I want to go from Shibuya to Shinjuku."

[0701] 3. The device analyzes this text using the Google Natural Language API and extracts "Departure: Shibuya" and "Destination: Shinjuku."

[0702] 4. The device analyzes emotions using Microsoft Azure Text Analytics and recognizes that the user is in a "stressed state."

[0703] 5. The device selects and launches the NAVITIME transit information application and sets the parameters.

[0704] 6. The server obtains route information using NAVITIME's API and returns the information "next train time: 10:30" and "fare: 300 yen" to the terminal.

[0705] 7. The device analyzes this information and, taking into account the stress level, generates a concise voice message such as, "The next train leaves at 10:30 and costs 300 yen."

[0706] 8. The device plays the generated audio data and notifies the user.

[0707] Prompt Sentence Examples

[0708] "Please tell me when the next train leaves and how much it costs."

[0709] This system is particularly useful for elderly people and users who are unfamiliar with technology, as it can understand the user's intentions and emotions and provide appropriate information via voice, simply by inputting the user's voice.

[0710] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0711] Step 1:

[0712] The user speaks to the smartphone, saying, "From where do I want to go to where?" For example, "I want to go from Shibuya to Shinjuku." The device captures this voice data through the smartphone's microphone. The voice data is input and output in a format that is temporarily stored within the device. Specifically, the smartphone's microphone converts the voice signal into digital data, and then records that digital data.

[0713] Step 2:

[0714] The device sends the captured voice data to a voice recognition engine. Here, the Google Speech-to-Text API is used. Voice data is sent to this API as input, and the converted text data is received as output. For example, a voice saying "I want to go from Shibuya to Shinjuku" is converted into text data saying "I want to go from Shibuya to Shinjuku." In concrete terms, the device sends voice data as an API request and receives text data as an API response.

[0715] Step 3:

[0716] The device sends the text data obtained by speech recognition to the NLP engine. Here, the Google Natural Language API is used. The text data is sent as input to the NLP engine, and the analyzed intent and parameters are received as output. For example, parameters such as "Departure: Shibuya" and "Arrival: Shinjuku" are extracted from the text. Specifically, the device sends the text data as an API request and receives response data with the intent and parameters extracted.

[0717] Step 4:

[0718] The device sends the obtained text data to an emotion recognition engine. In this case, Microsoft Azure Text Analytics is used. The text data is sent as input to the emotion recognition engine, and the recognized emotion information is received as output. For example, if the user is feeling "stressed," it is identified as "stress." Specifically, the device sends the text data as an API request and receives the emotion recognition results as response data.

[0719] Step 5:

[0720] The device selects an appropriate application based on the results of natural language processing and emotion recognition, and prepares to launch it. For example, if the user's intent is "travel" and "stress" is recognized, a route guidance application such as NAVITIME will be selected. The device also prepares to pass the necessary parameters (starting point, destination) to the application. Specifically, it determines the application name and parameters, and generates an API request to launch the application.

[0721] Step 6:

[0722] The server accesses the specified API endpoint and retrieves the information requested by the user. For example, it uses the NAVITIME API to retrieve "route information from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. The collected data is then sent back to the device. Specifically, it sends an information request to the API endpoint and retrieves route information as response data.

[0723] Step 7:

[0724] The device analyzes the received information and adjusts the notification content taking into account the user's emotions. For example, if the user is feeling "stressed," the notification content is converted into a concise and easy-to-understand format. Route information (next train time, fare, etc.) is input, and the adjusted voice notification content is output. Specifically, the device analyzes the response data and reformats the text according to the user's emotions.

[0725] Step 8:

[0726] The device sends the adjusted notification content to a speech synthesis engine and converts it into voice data. Here, the Google Cloud Text-to-Speech API is used. The text data of the notification content is sent as input to the speech synthesis engine, and voice data is received as output. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated. Specifically, the text data is sent as a request to the speech synthesis API, and voice data is received as a response.

[0727] Step 9:

[0728] The device plays the generated voice data and notifies the user. The voice data is input and output as voice through the smartphone's speaker. Specifically, the device's voice playback function is used to play the voice from the speaker.

[0729] (Application example 2)

[0730] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0731] In recent years, there has been a demand for systems that can analyze user intent based on voice input and launch the appropriate application. Interfaces that combine voice input and analysis are particularly useful for users who want to easily order delivery or perform complex operations. However, existing systems often struggle to provide appropriate suggestions that take the user's emotions into account, and the methods of notifying users of information are often not optimal. Therefore, it is necessary to provide a system that can analyze the user's intent and emotions, select and launch the appropriate application based on the results, and notify the user of the results via voice.

[0732] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0733] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying the user's intention and emotion, means for selecting and launching an appropriate application based on the user's intention and emotion, and means for obtaining the execution result of the application and notifying the user by voice, thereby enabling delivery orders and other complex operations to be performed simply and optimally through voice input.

[0734] "Voice input" is the process of capturing the user's spoken words and storing them as digital data.

[0735] "Text data" is an information format in which voice data is converted into a string of characters.

[0736] "User intent" refers to the goal or request that the user is trying to achieve through voice input.

[0737] "User emotions" refers to the psychological and emotional state that can be read from the user's comments.

[0738] "Application selection and launch" is the process of choosing and running the most appropriate software program based on the user's intent and emotions.

[0739] "Application Results" refers to the final information or output provided by a selected and launched application.

[0740] "Voice notification" refers to the act of communicating the results of an application's execution to the user as voice using speech synthesis technology.

[0741] A "transit application" is a software program that allows a user to search for and provide route information from one point to another.

[0742] A "food delivery application" is a software program that allows a user to order food and delivers the food to a specified location.

[0743] An "audio input device" is hardware that captures a user's voice and processes it as a digital signal.

[0744] A "speech recognition engine" is software or algorithms for analyzing captured voice data and converting it into text form.

[0745] This system analyzes the user's intentions and emotions based on the user's voice input, selects and launches an appropriate application, and notifies the user of the results via voice. This system is primarily applied in the food delivery field, helping users to easily order food through voice input.

[0746] Hardware and software used

[0747] Hardware:

[0748] Audio input device: A device that captures a user's voice. Examples include smart glasses and smartphones.

[0749] Server: A central processing unit for processing and analyzing data.

[0750] software:

[0751] Speech recognition engine: Software for converting voice data into text data, such as the Google Cloud Speech-to-Text API.

[0752] Natural Language Processing (NLP) engine: Software for analyzing text data and identifying user intent, such as the Google Cloud Natural Language API.

[0753] Emotion recognition engine: Software that analyzes user emotions from text data, such as IBM Watson Tone Analyzer.

[0754] Speech synthesis engine: Software for converting text data into speech, such as the Google Text-to-Speech API.

[0755] Processing flow

[0756] 1. Audio input capture:

[0757] A voice input device (smart glasses or a smartphone) captures the user's voice, for example, the user says, "I'd like to order a pizza."

[0758] 2. Speech Recognition:

[0759] The captured voice data is sent to a voice recognition engine, which converts the voice into text data.

[0760] 3. Natural Language Processing (NLP):

[0761] The text data is sent to an NLP engine, which analyzes the user's intent and extracts the item to be ordered (in this example, "pizza").

[0762] 4. Emotion recognition:

[0763] Furthermore, the text data is sent to an emotion recognition engine to analyze the user's emotions (e.g., stress state).

[0764] 5. Select and launch the application:

[0765] Based on the analysis results, the system selects and launches an appropriate food delivery application. For example, if the user is feeling stressed, an application that allows for easy ordering will be selected.

[0766] 6. Obtaining and notifying order results:

[0767] The order information is acquired and then a voice synthesis engine delivers it to the user in a relaxed tone, depending on the user's emotions.

[0768] Specific examples

[0769] For example, if a user says "I'd like to order a pizza":

[0770] 1. User: Says, "I want to order a pizza."

[0771] 2. Voice input device: Captures voice and sends it to the voice recognition engine.

[0772] 3. Speech recognition engine: Converts voice data into text data such as "I would like to order pizza."

[0773] 4. Natural language processing engine: Analyzes text data and extracts "Order: Pizza".

[0774] 5. Emotion recognition engine: Analyzes text data and recognizes when the user is in a "stressed state."

[0775] 6. Server: Select a food delivery application (e.g., food delivery service).

[0776] 7. Server: The food delivery application confirms the user's order.

[0777] 8. Speech synthesis engine: Generates the order result as a voice saying, "The pizza will arrive in 20 minutes."

[0778] 9. Audio input device: Plays audio data to the user and notifies them.

[0779] Prompt Sentence Examples

[0780] An example of a prompt is:

[0781] The user says, "I want to order a pizza."

[0782] The voice recognition engine converts the sentence "I would like to order a pizza" into text.

[0783] The NLP engine extracts "What to order: Pizza."

[0784] The emotion engine recognizes the user as being in a "stressed state."

[0785] Select a food delivery application considering stress conditions.

[0786] Suggests pizza types and potential stores.

[0787] Select "Margherita Pizza" and confirm your order.

[0788] The order information is retrieved and a voice notification is given saying, "Your pizza will arrive in 20 minutes."

[0789] The audio input device has completed audio notification.

[0790] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0791] Step 1:

[0792] Voice input capture

[0793] A user speaks into a voice input device (e.g., smart glasses or a smartphone) saying, "I want to order a pizza." The voice input device captures the voice through a microphone and temporarily stores it as voice data.

[0794] Input: User's voice

[0795] Output: Audio data

[0796] Step 2:

[0797] Voice Recognition

[0798] The device sends the captured voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data and returns the text data to the device.

[0799] Input: Audio data

[0800] Output: Text data

[0801] Step 3:

[0802] Natural Language Processing (NLP)

[0803] The device sends the text data to a natural language processing engine (e.g., Google Cloud Natural Language), which analyzes the text data, identifies the user's intent, such as "What to order: Pizza," and returns the analysis results to the device.

[0804] Input: Text data

[0805] Output: User intent (e.g., "Order: Pizza")

[0806] Step 4:

[0807] emotion recognition

[0808] The device further transmits the text data to an emotion recognition engine (e.g., IBM Watson Tone Analyzer), which analyzes the text data and identifies the user's emotion (e.g., "stressed"), and returns the emotion recognition result to the device.

[0809] Input: Text data

[0810] Output: Emotion analysis result (e.g. "stressed")

[0811] Step 5:

[0812] Selecting and Launching an Application

[0813] Based on the analysis results (user intentions and emotions), the device selects an appropriate application (e.g., a food delivery application) from the server, sets the necessary parameters, and launches the application. For example, if the user is feeling stressed, the device will prioritize a food delivery application that is easy to operate.

[0814] Input: User intent, sentiment analysis results

[0815] Output: Launch of selected application

[0816] Step 6:

[0817] Obtaining order results

[0818] The server will then contact the selected application to confirm the order and obtain delivery information (arrival time, shipping costs, etc.), which involves using the application's API to request the required information and return the results.

[0819] Input: Application request

[0820] Output: Delivery information (e.g. arrival time, shipping fee)

[0821] Step 7:

[0822] Audio notifications

[0823] The device sends the acquired delivery information to a speech synthesis engine, which generates concise and easy-to-understand voice data. For example, it generates a voice that says, "Your pizza will arrive in 20 minutes." The device plays this voice data and notifies the user. The notification tone and speed are adjusted based on the user's emotions (e.g., stress level).

[0824] Input: Delivery information

[0825] Output: Audio notification

[0826] Specific examples

[0827] Example prompt sentence:

[0828] The user says, "I want to order a pizza."

[0829] The voice recognition engine converts the sentence "I would like to order a pizza" into text.

[0830] The NLP engine extracts "What to order: Pizza."

[0831] The emotion engine recognizes the user as being in a "stressed state."

[0832] Select a food delivery application considering stress conditions.

[0833] Suggests pizza types and potential stores.

[0834] Select "Margherita Pizza" and confirm your order.

[0835] The order information is retrieved and a voice notification is given saying, "Your pizza will arrive in 20 minutes."

[0836] The audio input device has completed audio notification.

[0837] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0838] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0839] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0840] [Third embodiment]

[0841] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0842] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0843] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0844] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0845] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0846] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0847] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0848] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0849] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0850] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0851] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0852] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0853] This invention is a system that captures voice input, selects and launches an appropriate application according to the user's intention, and notifies the user of the result by voice. Specific embodiments for implementing this system are described below.

[0854] 1. Receiving voice input

[0855] The user speaks to their smartphone, saying, "I want to go from here to where I want to go." For example, they might say, "I want to go from Shibuya to Shinjuku." The device captures the voice data through a microphone. This voice data is temporarily stored on the device.

[0856] 2. Voice Recognition

[0857] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice input of "I want to go from Shibuya to Shinjuku" is converted into text "I want to go from Shibuya to Shinjuku." This text data is temporarily saved as it will be used in the next step.

[0858] 3. Natural Language Processing (NLP)

[0859] The device analyzes the text data using a natural language processing (NLP) engine. The NLP engine identifies the user's intent and extracts the necessary parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" may be extracted. The analysis results are also temporarily saved.

[0860] 4. Select and launch the app

[0861] The device selects an appropriate application based on the analysis results and prepares to launch it. For example, if the analysis determines that the user has the intent to "move," the device selects a transit guide application. The device prepares to pass the necessary parameters (such as departure and arrival points) to the application.

[0862] 5. Result acquisition and voice notification

[0863] The server accesses the specified API endpoint and obtains the information requested by the user. For example, a user requests "route from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. The obtained information is then returned from the server to the device.

[0864] The device converts the received result into voice data using a speech synthesis engine. For example, a notification such as "The next train leaves at 10:30 and costs 300 yen" is generated. This voice data is played back from the device's speaker, providing the user with an audible notification.

[0865] Specific examples

[0866] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[0867] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[0868] 2. The device captures the voice and converts it into text using a speech recognition engine: "I want to go from Shibuya to Shinjuku."

[0869] 3. The device analyzes the text using an NLP engine and extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[0870] 4. The device selects the transit information application and prepares to launch it.

[0871] 5. The server retrieves route information from Shibuya to Shinjuku from the API and returns details (next train time, fare, etc.) to the terminal.

[0872] 6. The terminal converts the result of the operation into voice and announces, "The next train leaves at 10:30 and costs 300 yen."

[0873] 7. The device launches the transit information application and displays detailed information.

[0874] In this way, the present invention provides a system that enables even elderly people and users who are unfamiliar with technology to easily operate a desired application and obtain necessary information.

[0875] The processing flow will be explained below.

[0876] Step 1:

[0877] The user speaks into the smartphone to input the necessary instructions, for example, "I want to go from Shibuya to Shinjuku." This voice data is then captured by the smartphone's microphone.

[0878] Step 2:

[0879] The device sends the captured voice data to its internal voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku."

[0880] Step 3:

[0881] The device sends the converted text data to a natural language processing (NLP) engine, which analyzes the text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted.

[0882] Step 4:

[0883] The device checks the analysis results and determines which application is appropriate. In this case, since the user's intention is related to "travel," the transit guide application is selected.

[0884] Step 5:

[0885] The terminal prepares to launch the selected transit information application by passing necessary parameters (e.g., departure point, arrival point).

[0886] Step 6:

[0887] The server processes API requests for transit information. For example, a request like "https: / / api.transit.com / getInfo?saddr=Shibuya&daddr=Shinjuku" is sent to retrieve information such as "The next train departs at 10:30 and costs 300 yen."

[0888] Step 7:

[0889] The server returns the acquired information to the terminal in JSON format.

[0890] Step 8:

[0891] The terminal analyzes the information it receives and extracts the necessary information (e.g., the time of the next train, the fare).

[0892] Step 9:

[0893] The device sends the extracted information to a speech synthesis engine, which converts it into voice data. For example, the generated voice data would say, "The next train leaves at 10:30 and costs 300 yen."

[0894] Step 10:

[0895] The terminal plays the generated voice data and notifies the user, so that the user receives the necessary information by voice.

[0896] In this way, this system combines voice recognition and natural language processing to select and launch the appropriate application based on the user's voice input, and then returns the results in voice form, implementing a series of processes.

[0897] Example 1

[0898] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0899] In modern society, when elderly people and users unfamiliar with technology use smartphones to obtain information, the operation is often complicated, making it difficult to obtain information. In particular, when obtaining information using voice input, there is a demand for an efficient system that can smoothly and intuitively perform the entire process from voice recognition to application selection and notification of results.

[0900] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0901] In this invention, the server includes: means for capturing voice input; means for converting the voice input into text data; means for analyzing the text data and identifying the user's intent; means for selecting and launching appropriate software based on the user's intent; means for acquiring the execution results of the software and notifying the user by voice; means for capturing the voice input with a microphone and converting it into text data using a voice recognition engine; means for analyzing the text data with a natural language processing engine and extracting necessary parameters; means for sending a request to the server based on the parameters and acquiring necessary information from a specified API endpoint; and means for converting the acquired information into voice data using a voice synthesis engine and notifying the user. This enables even elderly people and users who are not familiar with technology to easily acquire desired information using only voice input.

[0902] "Voice input" refers to the act of a user giving instructions by voice, which is captured as digital voice data via a microphone.

[0903] "Text data" refers to data that has been converted into text form after voice input has been analyzed by a voice recognition engine.

[0904] "Analysis" refers to the process of using a natural language processing engine to identify the user's intent and required information from text data.

[0905] "User intent" refers to the action or information the user is seeking, as determined from the voice input.

[0906] "Software" refers to application programs that are selected and launched based on the user's intent.

[0907] A "microphone" is a hardware device for capturing audio input.

[0908] A "speech recognition engine" is a software or hardware mechanism for converting voice data captured by a microphone into text data.

[0909] A "natural language processing engine" is a software mechanism that analyzes text data and extracts the user's intent and necessary parameters.

[0910] "Parameters" refer to necessary information extracted from the analyzed text data, specifically, the departure and arrival points.

[0911] A "server" is a program or device with a specific function that retrieves information from a specified API based on a user request.

[0912] "API endpoint" refers to the interface of an external service that a server accesses to obtain information.

[0913] A "speech synthesis engine" is software for converting text data into voice data.

[0914] "Voice notification" refers to the act of notifying a user of information using voice data generated by a voice synthesis engine.

[0915] The present invention relates to a system that allows a user to obtain information by voice input and notifies the user of the information by voice. Specific means for implementing this system will be described below.

[0916] overview

[0917] The system of this invention captures and analyzes voice input, selects and launches appropriate software according to the user's intention, and notifies the user of the results by voice, making it easy for even elderly users and those unfamiliar with technology to obtain information.

[0918] Hardware and software configuration

[0919] 1. Receiving voice input

[0920] The user issues specific instructions to the smartphone. For example, the user says, "I want to go from Shibuya to Shinjuku." The device's microphone captures this voice input and generates voice data. This voice data is stored in the device's temporary memory.

[0921] 2. Voice Recognition

[0922] The device sends the voice data to a voice recognition engine (for example, a commercial voice recognition API). The voice recognition engine converts the captured voice data into text data. The converted text data is in the format of, for example, "I want to go from Shibuya to Shinjuku." This text data is also temporarily stored in the device.

[0923] 3. Natural Language Processing (NLP)

[0924] The device sends the text data to a natural language processing (NLP) engine (e.g., a commercial NLP API). The NLP engine analyzes the text data and extracts important parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" is extracted. This parameter information is also temporarily stored on the device.

[0925] 4. Select and launch the app

[0926] The device selects the appropriate software based on the analysis results. For example, if the user has the intent to "move," the device selects transit information software (e.g., a commercial transit information app). The device prepares to launch the selected application and collects the necessary parameters (start and end locations).

[0927] 5. Result acquisition and voice notification

[0928] The device sends the analysis results (parameters) to the server, and the server accesses the specified API endpoint (for example, a commercial API endpoint) to obtain the required information. For example, the server requests "route information from Shibuya to Shinjuku" and obtains the next train time, fare, etc. The obtained information is returned from the server to the device.

[0929] The device converts the acquired information into speech using a speech synthesis engine (for example, a commercial speech synthesis API). The speech data will be in the form of "The next train leaves at 10:30 and costs 300 yen." This speech data is played back to the user through the device's speaker.

[0930] Specific examples

[0931] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[0932] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[0933] 2. The device's microphone captures the audio data and stores it in temporary memory.

[0934] 3. The device sends the voice data to a speech recognition engine and converts it into text.

[0935] 4. The device sends the text data to a natural language processing engine, which extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[0936] 5. The device selects the transit information software and prepares to launch it.

[0937] 6. The terminal sends the extracted parameters to the server.

[0938] 7. The server requests the API endpoint and obtains the route information.

[0939] 8. The server returns the acquired information (next train time, fare) to the terminal.

[0940] 9. The information received by the device is converted into voice using a speech synthesis engine, and the user is notified that "The next train leaves at 10:30 and costs 300 yen."

[0941] 10. The device will launch the transit information software and display detailed information.

[0942] Prompt Sentence Examples

[0943] "If I say I want to go from Shibuya to Shinjuku, please explain the specific process."

[0944] In this way, the present invention enables even elderly people and users who are unfamiliar with technology to quickly and easily obtain desired information using only voice input.

[0945] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0946] Step 1: Receiving voice input

[0947] The user issues specific instructions to the smartphone. For example, the user might say, "I want to go from Shibuya to Shinjuku." The device's microphone captures this voice input and generates digital audio data, which is then stored in the device's temporary memory.

[0948] Input: User voice input

[0949] Output: Audio data

[0950] Step 2: Voice Recognition

[0951] The device sends the voice data to a voice recognition engine (for example, a commercial voice recognition API). The voice recognition engine converts the captured voice data into text data. For example, the text data is converted into text data in the format "I want to go from Shibuya to Shinjuku." This text data is temporarily stored in the device.

[0952] Input: Audio data

[0953] Output: Text data

[0954] Step 3: Natural Language Processing (NLP)

[0955] The device sends the text data to a natural language processing (NLP) engine (e.g., a commercial NLP API). The NLP engine analyzes the text data and extracts important parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" is extracted. This parameter information is also temporarily stored on the device.

[0956] Input: Text data

[0957] Output: Extracted parameters (e.g. "Departure: Shibuya", "Destination: Shinjuku")

[0958] Step 4: Select and launch the app

[0959] The device selects the appropriate software based on the analysis results. For example, if the analysis results indicate that the user has the intention of "moving," the device selects transit information software. Next, the device prepares to launch the selected application and collects the necessary parameters (starting point and destination).

[0960] Input: Extracted parameters

[0961] Output: Software selection and launch preparation

[0962] Step 5: Get the results and hear the audio

[0963] The device sends the analysis results (parameters) to the server. The server accesses the specified API endpoint and obtains the information requested by the user. For example, the server requests "route information from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. This collected information is returned from the server to the device. The device then converts the obtained information into voice data using a speech synthesis engine and notifies the user, "The next train leaves at 10:30 and costs 300 yen."

[0964] Input: Analysis results (parameters)

[0965] Output: Acquired information and its spoken notification

[0966] In this way, a series of steps is realized, starting with the user's voice input, followed by voice recognition, natural language processing, application selection and launch, and result acquisition and voice notification. Appropriate data processing and data calculation are performed at each step, ultimately resulting in a system that provides the user with the information they need intuitively and efficiently.

[0967] (Application example 1)

[0968] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0969] Current food delivery services require users to manually operate the application, which is particularly cumbersome for elderly people and those who are not familiar with technology. Also, visually impaired users have difficulty operating smartphones. Therefore, there is a need for a method that allows everyone to easily use food delivery services.

[0970] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0971] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying the user's intention, means for selecting and launching an appropriate application based on the user's intention, means for acquiring the execution result of the application and notifying the user by voice, and means for including an application related to the food delivery service. This makes it possible to use the food delivery service with just voice input, and enables anyone to easily perform and use the ordering procedure.

[0972] "Means for capturing audio input" refers to a device and its associated software for detecting and recording audio input.

[0973] "Means for converting speech input into text data" refers to software and algorithms for converting speech signals into written information.

[0974] "Means for analyzing text data and identifying user intent" refers to natural language processing technology and related software for analyzing text data and understanding the user's intended operations and requests.

[0975] "Means for selecting and launching an appropriate application" refers to a processing system for automatically selecting and launching a particular application based on a user's intent.

[0976] "Means for obtaining the results of application execution and notifying the user by voice" refers to technology and devices for collecting the results or data of an executed application and providing it to the user as voice.

[0977] "Food delivery service application" refers to a software application that allows a user to voice-order food and process the delivery.

[0978] "Device for capturing an acoustic signal" refers to a hardware device, such as a microphone, for obtaining audio input.

[0979] "Speech recognition engine" refers to software and algorithms for processing captured acoustic signals and converting them into text data.

[0980] This invention is a system that captures voice input, selects and launches an appropriate application according to the user's intention, and notifies the user of the results by voice. Specific embodiments for implementing this system are described below.

[0981] 1. Receiving voice input

[0982] A user speaks to the device, saying, "I want to order a pizza," or, for example, "I want to order a Margherita from a food delivery service." The device captures the voice data through a microphone, a device that captures acoustic signals, and this voice data is temporarily stored within the device.

[0983] 2. Voice Recognition

[0984] The device sends the captured voice data to a speech recognition engine, which converts the voice data into text data. For example, a voice input of "I would like to order a margherita from a food delivery service" is converted into text "I would like to order a margherita from a food delivery service." This text data is temporarily stored for use in the next step.

[0985] 3. Natural Language Processing (NLP)

[0986] The device analyzes the text data using a natural language processing (NLP) engine. The NLP engine identifies the user's intent and extracts necessary parameters. For example, information such as "Vendor: food delivery service" and "Item: Margherita" may be extracted. The analysis results are also temporarily stored.

[0987] 4. Select and launch the app

[0988] The device selects an appropriate application based on the analysis results and prepares to launch it. For example, if the analysis reveals that the user has the intent to "order," a food delivery service application will be selected. The device prepares to pass the necessary parameters (vendor, product, etc.) to the application.

[0989] 5. Result acquisition and voice notification

[0990] The device accesses the specified API endpoint and obtains information according to the user's request. For example, a request such as "I would like to order a margherita from a food delivery service" is sent, and confirmation information about the order is collected. The obtained information is returned from the server to the device. The device then converts the received results into voice data using a speech synthesis engine. For example, a notification such as "Your order has been completed. Your margherita will arrive in 30 minutes" is generated. This voice data is played from the device's speaker, providing the user with an audible notification.

[0991] Specific examples

[0992] For example, let's say a user says, "I want to order a Margherita from a food delivery service." Here's what happens:

[0993] 1. A user says, "I want to order a margherita from a food delivery service."

[0994] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "I'd like to order a margherita from a food delivery service."

[0995] 3. The device analyzes the text using an NLP engine and extracts the parameters "Vendor: food delivery service" and "Item: Margherita."

[0996] 4. The device selects the food delivery service application and prepares to launch it.

[0997] 5. The server retrieves the "Margherita order information from the food delivery service" from the API and returns the details (order confirmation information) to the terminal.

[0998] 6. The terminal converts the execution result into voice and notifies the user by saying, "Your order has been completed. Your Margherita will arrive in 30 minutes."

[0999] 7. The device launches the food delivery service application and displays detailed information.

[1000] Prompt Sentence Examples

[1001] "Using speech recognition, we have received the text 'I would like to order a margherita from a food delivery service.' We will proceed with the order based on the order details extracted using NLP analysis (vendor: food delivery service, product: margherita). Once the order is complete, we will announce the message 'Your order is complete. Your margherita will arrive in 30 minutes.'"

[1002] In this way, the present invention provides a system that allows even elderly or less technically savvy users to easily operate a food delivery service and obtain the information they need.

[1003] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1004] Step 1:

[1005] The user speaks to the device, saying, "I'd like to order a Margherita from a food delivery service." The input is voice data, which is captured through the device's microphone. This voice data is temporarily stored in the device.

[1006] Step 2:

[1007] The device sends the captured voice data to a speech recognition engine, which converts the input voice data into text data, and the resulting text data is "I would like to order a Margherita from a food delivery service." This text data is also temporarily stored.

[1008] Step 3:

[1009] The device analyzes the text data using a natural language processing (NLP) engine. The input is converted text data, and the data is processed through analysis to identify the user's intent. Specific instruction parameters such as "Vendor: food delivery service" and "Item: Margherita" are extracted as output. The analysis results are also temporarily stored.

[1010] Step 4:

[1011] The device selects the appropriate application based on the analysis results and prepares to launch it. The input is the parameters obtained from the analysis, and the application is selected based on the data. The output is the identity of the application to be launched, for example, a food delivery service application is selected. The device prepares to pass the necessary parameters (vendor, product, etc.) to the application.

[1012] Step 5:

[1013] The terminal accesses the specified API endpoint and retrieves the information requested by the user. The input is the instruction parameters selected by the application, and a request based on them is sent to the API endpoint. The output is order confirmation information and other related data. The retrieved information is returned from the server to the terminal.

[1014] Step 6:

[1015] The device converts the received results into voice data using a speech synthesis engine. The input is the order confirmation information obtained from the API, and data processing is performed to convert it into voice data. The output is voice notification data such as "Your order is complete. Your Margherita will arrive in 30 minutes." This voice data is played from the device's speaker, providing the user with an audible notification.

[1016] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1017] The present invention is a system that captures voice input, selects and launches an appropriate application according to the user's intentions and emotions, and notifies the user of the results by voice. Specific embodiments for implementing this system are described below.

[1018] 1. Receiving voice input

[1019] The user speaks to their smartphone, saying, "I want to go from here to where I want to go." For example, they might say, "I want to go from Shibuya to Shinjuku." The device captures the voice data through a microphone. This voice data is temporarily stored on the device.

[1020] 2. Voice Recognition

[1021] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku." This text data is temporarily saved as it will be used in the next step.

[1022] 3. Natural Language Processing (NLP)

[1023] The device sends the text data to a natural language processing (NLP) engine. The NLP engine analyzes the text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted. This analysis result is also temporarily saved.

[1024] 4. Emotion recognition

[1025] The device further transmits the text data to the emotion engine. The emotion engine analyzes the text data and recognizes the user's emotion. For example, if the user is feeling stressed, the emotion engine identifies this as "stress." The emotion recognition result is also temporarily stored.

[1026] 5. Select and launch the app

[1027] Based on the analysis results, the device selects an appropriate application and prepares to launch it. For example, if the user's intention is related to "travel" and "stress" is also recognized, the device will select a transit guide application. The device will pass the necessary parameters (departure point, destination) to the application and instruct it to launch.

[1028] 6. Result acquisition and voice notification

[1029] The server accesses the specified API endpoint and retrieves the information requested by the user. For example, a request for "route from Shibuya to Shinjuku" is made, and information such as the time and fare of the next train is collected. The retrieved information is then returned from the server to the device.

[1030] The device analyzes the received results and extracts the necessary information (such as the next train time and fare). The content of the notification is adjusted based on the emotion engine's results. For example, if the user is feeling stressed, the notification content will be made concise and easy to understand.

[1031] 7. Notification

[1032] The device sends the extracted information to a speech synthesis engine and converts it into voice data. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated. This voice data is played back from the device's speaker to notify the user. The notification is made in an appropriate tone and speed based on the emotion.

[1033] Specific examples

[1034] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[1035] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[1036] 2. The device captures the voice and sends it to a speech recognition engine, which converts it into text such as "I want to go from Shibuya to Shinjuku."

[1037] 3. The device sends this text to the NLP engine, which extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[1038] 4. The device sends the text to the emotion engine, which recognizes that the user is in a "stressed state."

[1039] 5. The device selects the transit guide application, sets parameters, and launches it.

[1040] 6. The server uses the transfer information API to obtain route information and returns the next train time and fare to the terminal.

[1041] 7. The device analyzes the results, generates voice data in a concise format taking into account the stress level, and then uses a voice synthesis engine to announce, "The next train leaves at 10:30 and costs 300 yen."

[1042] 8. The device plays this audio data and notifies the user.

[1043] In this way, the present invention provides a system that combines voice recognition, natural language processing, and emotion recognition based on voice input, allowing even elderly people and users who are unfamiliar with technology to easily operate the desired application and obtain information.

[1044] The processing flow will be explained below.

[1045] Step 1:

[1046] The user speaks into the smartphone, saying, "I want to go from Shibuya to Shinjuku." This voice data is captured by the smartphone's microphone.

[1047] Step 2:

[1048] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku."

[1049] Step 3:

[1050] The device sends the converted text data to a natural language processing (NLP) engine. The NLP engine analyzes this text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted. This analysis result is temporarily saved.

[1051] Step 4:

[1052] The device then sends the text data to the emotion engine, which analyzes the text data and recognizes the user's emotion. For example, if the user is feeling stressed, the emotion engine identifies the user's emotion as "stress." The emotion recognition result is also temporarily saved.

[1053] Step 5:

[1054] The device determines which application is appropriate based on the analysis results and emotion recognition results. For example, if the user's intention is related to "travel" and "stress" is recognized, the device will select a transit guide application.

[1055] Step 6:

[1056] The terminal passes the parameters (starting point, destination) required for the selected transit information application and prepares to launch the application.

[1057] Step 7:

[1058] The server accesses the API endpoint for transit information and retrieves information according to the user's request. For example, a request such as "https: / / api.transit.com / getInfo?saddr=Shibuya&daddr=Shinjuku" is sent to retrieve information such as "The next train departs at 10:30, and the fare is 300 yen." The retrieved information is returned to the terminal in JSON format.

[1059] Step 8:

[1060] The device analyzes the information it receives and extracts the necessary information (time of the next train, fare, etc.). The content of the notification is adjusted taking into account the results of the emotion engine. For example, if the user is feeling stressed, the notification content will be made concise and easy to understand.

[1061] Step 9:

[1062] The device sends the extracted information to a speech synthesis engine, which converts it into voice data. For example, the generated voice data would say, "The next train leaves at 10:30 and costs 300 yen."

[1063] Step 10:

[1064] The device plays the generated voice data over the speaker to notify the user, allowing the user to receive the necessary information through voice. The notification content is played back in an appropriate tone and speed based on the user's emotion.

[1065] Through these steps, the system can accurately recognize the user's voice input, select an appropriate application based on the user's set intention and emotion, and effectively notify the user of the results through voice.

[1066] Example 2

[1067] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1068] Conventional speech recognition systems convert speech input into text and provide a means to select and launch applications based on the user's intent. However, they face the challenge of responding appropriately while taking into account the user's emotions. Providing appropriate and easy-to-understand notifications is particularly important for elderly users and those unfamiliar with technology. Therefore, a system that can recognize not only the user's intent but also their emotions based on speech input and adjust notification content as needed is highly desirable.

[1069] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1070] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying a user's intention, means for analyzing the text data and recognizing the user's emotion, means for selecting and launching an appropriate application based on the user's intention and emotion, means for obtaining the execution result of the application and notifying the user by voice, and means for adjusting the notification based on the user's emotion, thereby enabling elderly people and users who are unfamiliar with technology to obtain information in an optimal form according to their emotions.

[1071] "Voice input" is a method by which a user gives instructions or queries to a system using voice.

[1072] "Text data" is a data format that expresses voice input as text information.

[1073] "Analysis" is the act of processing text data and other data to extract information such as intent and emotion.

[1074] "Intention" is information indicating the purpose for which the user inputs voice.

[1075] "Emotion" is information that indicates the psychological state of the user, and includes states such as stress and joy.

[1076] An "application" is a software program designed to perform a particular function.

[1077] "Launch" is an operation that puts an application into an executable state.

[1078] The "execution result" is the output information generated after the application has performed its processing.

[1079] A "notification" is the act of providing information to a user, often in audio form.

[1080] "Adjustment" refers to the act of changing the information provided and its format to the optimum form depending on the user's condition.

[1081] This invention is a system that allows a user to input voice, analyzes the voice input, recognizes the user's intentions and emotions, and selects and launches an appropriate application. Furthermore, when notifying the user of the results of application execution by voice, it has a function to adjust the notification content based on the user's emotions.

[1082] Hardware and software used

[1083] The system uses the following hardware and software:

[1084] Audio input capture hardware: Microphones built into mobile devices such as smartphones and tablets

[1085] Speech recognition engine: Google Speech-to-Text API, IBM Watson Speech to Text, etc.

[1086] Natural Language Processing (NLP) engines: Google Natural Language API, Amazon Comprehend, etc.

[1087] Emotion recognition engine: Microsoft Azure Text Analytics for Sentiment Analysis, etc.

[1088] Application launch software: Route guidance applications such as NAVITIME and Google Maps

[1089] Speech synthesis engine: Google Cloud Text-to-Speech API, Amazon Polly, etc.

[1090] Program processing

[1091] Receiving voice input

[1092] The user speaks to the smartphone, saying, "I want to go from here to where I want to go." For example, "I want to go from Shibuya to Shinjuku." The device captures this voice data through the smartphone's microphone and temporarily stores it on the device.

[1093] Voice Recognition

[1094] The device sends the captured voice data to a speech recognition engine, which uses the Google Speech-to-Text API or similar to convert it into text data such as "I want to go from Shibuya to Shinjuku." This text data is temporarily stored for use in the next processing step.

[1095] Natural Language Processing (NLP)

[1096] The device sends the text data obtained through voice recognition to an NLP engine, which analyzes the user's intent. Using Google Natural Language API and other tools, it extracts parameters such as "Departure: Shibuya" and "Arrival: Shinjuku." The analysis results are also temporarily saved.

[1097] emotion recognition

[1098] The device then sends the obtained text data to an emotion recognition engine to recognize the user's emotions. Using Microsoft Azure Text Analytics and other tools, the device determines whether the user is feeling "stressed." The emotion recognition results are also temporarily stored.

[1099] Selecting and launching the app

[1100] Based on these analysis results, the device selects an appropriate application and prepares to launch it. For example, if the user's intention is "travel" and the emotion is recognized as "stress," a route guidance application such as NAVITIME will be selected. The device passes the necessary parameters (starting point, destination) to the application and instructs it to launch.

[1101] Result acquisition and voice notification

[1102] The server accesses the specified API endpoint and retrieves information according to the user's request. For example, it retrieves "route information from Shibuya to Shinjuku" and returns information such as the next train time and fare to the device. The device analyzes these results and adjusts the notification content taking the user's emotions into account.

[1103] notification

[1104] The device then sends the formatted notification to a speech synthesis engine, which converts it into voice data. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated and notified to the user through the device's speaker. The tone and speed of the voice are adjusted based on the user's emotions.

[1105] Specific examples

[1106] For example, let's say a user says, "I want to go from Shibuya to Shinjuku." Here's the specific operation flow:

[1107] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[1108] 2. The device captures the audio and converts it into text using the Google Speech-to-Text API: "I want to go from Shibuya to Shinjuku."

[1109] 3. The device analyzes this text using the Google Natural Language API and extracts "Departure: Shibuya" and "Destination: Shinjuku."

[1110] 4. The device analyzes emotions using Microsoft Azure Text Analytics and recognizes that the user is in a "stressed state."

[1111] 5. The device selects and launches the NAVITIME transit information application and sets the parameters.

[1112] 6. The server obtains route information using NAVITIME's API and returns the information "next train time: 10:30" and "fare: 300 yen" to the terminal.

[1113] 7. The device analyzes this information and, taking into account the stress level, generates a concise voice message such as, "The next train leaves at 10:30 and costs 300 yen."

[1114] 8. The device plays the generated audio data and notifies the user.

[1115] Prompt Sentence Examples

[1116] "Please tell me when the next train leaves and how much it costs."

[1117] This system is particularly useful for elderly people and users who are unfamiliar with technology, as it can understand the user's intentions and emotions and provide appropriate information via voice, simply by inputting the user's voice.

[1118] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1119] Step 1:

[1120] The user speaks to the smartphone, saying, "From where do I want to go to where?" For example, "I want to go from Shibuya to Shinjuku." The device captures this voice data through the smartphone's microphone. The voice data is input and output in a format that is temporarily stored within the device. Specifically, the smartphone's microphone converts the voice signal into digital data, and then records that digital data.

[1121] Step 2:

[1122] The device sends the captured voice data to a voice recognition engine. Here, the Google Speech-to-Text API is used. Voice data is sent to this API as input, and the converted text data is received as output. For example, a voice saying "I want to go from Shibuya to Shinjuku" is converted into text data saying "I want to go from Shibuya to Shinjuku." In concrete terms, the device sends voice data as an API request and receives text data as an API response.

[1123] Step 3:

[1124] The device sends the text data obtained by speech recognition to the NLP engine. Here, the Google Natural Language API is used. The text data is sent as input to the NLP engine, and the analyzed intent and parameters are received as output. For example, parameters such as "Departure: Shibuya" and "Arrival: Shinjuku" are extracted from the text. Specifically, the device sends the text data as an API request and receives response data with the intent and parameters extracted.

[1125] Step 4:

[1126] The device sends the obtained text data to an emotion recognition engine. In this case, Microsoft Azure Text Analytics is used. The text data is sent as input to the emotion recognition engine, and the recognized emotion information is received as output. For example, if the user is feeling "stressed," it is identified as "stress." Specifically, the device sends the text data as an API request and receives the emotion recognition results as response data.

[1127] Step 5:

[1128] The device selects an appropriate application based on the results of natural language processing and emotion recognition, and prepares to launch it. For example, if the user's intent is "travel" and "stress" is recognized, a route guidance application such as NAVITIME will be selected. The device also prepares to pass the necessary parameters (starting point, destination) to the application. Specifically, it determines the application name and parameters, and generates an API request to launch the application.

[1129] Step 6:

[1130] The server accesses the specified API endpoint and retrieves the information requested by the user. For example, it uses the NAVITIME API to retrieve "route information from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. The collected data is then sent back to the device. Specifically, it sends an information request to the API endpoint and retrieves route information as response data.

[1131] Step 7:

[1132] The device analyzes the received information and adjusts the notification content taking into account the user's emotions. For example, if the user is feeling "stressed," the notification content is converted into a concise and easy-to-understand format. Route information (next train time, fare, etc.) is input, and the adjusted voice notification content is output. Specifically, the device analyzes the response data and reformats the text according to the user's emotions.

[1133] Step 8:

[1134] The device sends the adjusted notification content to a speech synthesis engine and converts it into voice data. Here, the Google Cloud Text-to-Speech API is used. The text data of the notification content is sent as input to the speech synthesis engine, and voice data is received as output. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated. Specifically, the text data is sent as a request to the speech synthesis API, and voice data is received as a response.

[1135] Step 9:

[1136] The device plays the generated voice data and notifies the user. The voice data is input and output as voice through the smartphone's speaker. Specifically, the device's voice playback function is used to play the voice from the speaker.

[1137] (Application example 2)

[1138] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1139] In recent years, there has been a demand for systems that can analyze user intent based on voice input and launch the appropriate application. Interfaces that combine voice input and analysis are particularly useful for users who want to easily order delivery or perform complex operations. However, existing systems often struggle to provide appropriate suggestions that take the user's emotions into account, and the methods of notifying users of information are often not optimal. Therefore, it is necessary to provide a system that can analyze the user's intent and emotions, select and launch the appropriate application based on the results, and notify the user of the results via voice.

[1140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1141] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying the user's intention and emotion, means for selecting and launching an appropriate application based on the user's intention and emotion, and means for obtaining the execution result of the application and notifying the user by voice, thereby enabling delivery orders and other complex operations to be performed simply and optimally through voice input.

[1142] "Voice input" is the process of capturing the user's spoken words and storing them as digital data.

[1143] "Text data" is an information format in which voice data is converted into a string of characters.

[1144] "User intent" refers to the goal or request that the user is trying to achieve through voice input.

[1145] "User emotions" refers to the psychological and emotional state that can be read from the user's comments.

[1146] "Application selection and launch" is the process of choosing and running the most appropriate software program based on the user's intent and emotions.

[1147] "Application Results" refers to the final information or output provided by a selected and launched application.

[1148] "Voice notification" refers to the act of communicating the results of an application's execution to the user as voice using speech synthesis technology.

[1149] A "transit application" is a software program that allows a user to search for and provide route information from one point to another.

[1150] A "food delivery application" is a software program that allows a user to order food and delivers the food to a specified location.

[1151] An "audio input device" is hardware that captures a user's voice and processes it as a digital signal.

[1152] A "speech recognition engine" is software or algorithms for analyzing captured voice data and converting it into text form.

[1153] This system analyzes the user's intentions and emotions based on the user's voice input, selects and launches an appropriate application, and notifies the user of the results via voice. This system is primarily applied in the food delivery field, helping users to easily order food through voice input.

[1154] Hardware and software used

[1155] Hardware:

[1156] Audio input device: A device that captures a user's voice. Examples include smart glasses and smartphones.

[1157] Server: A central processing unit for processing and analyzing data.

[1158] software:

[1159] Speech recognition engine: Software for converting voice data into text data, such as the Google Cloud Speech-to-Text API.

[1160] Natural Language Processing (NLP) engine: Software for analyzing text data and identifying user intent, such as the Google Cloud Natural Language API.

[1161] Emotion recognition engine: Software that analyzes user emotions from text data, such as IBM Watson Tone Analyzer.

[1162] Speech synthesis engine: Software for converting text data into speech, such as the Google Text-to-Speech API.

[1163] Processing flow

[1164] 1. Audio input capture:

[1165] A voice input device (smart glasses or a smartphone) captures the user's voice, for example, the user says, "I'd like to order a pizza."

[1166] 2. Speech Recognition:

[1167] The captured voice data is sent to a voice recognition engine, which converts the voice into text data.

[1168] 3. Natural Language Processing (NLP):

[1169] The text data is sent to an NLP engine, which analyzes the user's intent and extracts the item to be ordered (in this example, "pizza").

[1170] 4. Emotion recognition:

[1171] Furthermore, the text data is sent to an emotion recognition engine to analyze the user's emotions (e.g., stress state).

[1172] 5. Select and launch the application:

[1173] Based on the analysis results, the system selects and launches an appropriate food delivery application. For example, if the user is feeling stressed, an application that allows for easy ordering will be selected.

[1174] 6. Obtaining and notifying order results:

[1175] The order information is acquired and then a voice synthesis engine delivers it to the user in a relaxed tone, depending on the user's emotions.

[1176] Specific examples

[1177] For example, if a user says "I'd like to order a pizza":

[1178] 1. User: Says, "I want to order a pizza."

[1179] 2. Voice input device: Captures voice and sends it to the voice recognition engine.

[1180] 3. Speech recognition engine: Converts voice data into text data such as "I would like to order pizza."

[1181] 4. Natural language processing engine: Analyzes text data and extracts "Order: Pizza".

[1182] 5. Emotion recognition engine: Analyzes text data and recognizes when the user is in a "stressed state."

[1183] 6. Server: Select a food delivery application (e.g., food delivery service).

[1184] 7. Server: The food delivery application confirms the user's order.

[1185] 8. Speech synthesis engine: Generates the order result as a voice saying, "The pizza will arrive in 20 minutes."

[1186] 9. Audio input device: Plays audio data to the user and notifies them.

[1187] Prompt Sentence Examples

[1188] An example of a prompt is:

[1189] The user says, "I want to order a pizza."

[1190] The voice recognition engine converts the sentence "I would like to order a pizza" into text.

[1191] The NLP engine extracts "What to order: Pizza."

[1192] The emotion engine recognizes the user as being in a "stressed state."

[1193] Select a food delivery application considering stress conditions.

[1194] Suggests pizza types and potential stores.

[1195] Select "Margherita Pizza" and confirm your order.

[1196] The order information is retrieved and a voice notification is given saying, "Your pizza will arrive in 20 minutes."

[1197] The audio input device has completed audio notification.

[1198] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1199] Step 1:

[1200] Voice input capture

[1201] A user speaks into a voice input device (e.g., smart glasses or a smartphone) saying, "I want to order a pizza." The voice input device captures the voice through a microphone and temporarily stores it as voice data.

[1202] Input: User's voice

[1203] Output: Audio data

[1204] Step 2:

[1205] Voice Recognition

[1206] The device sends the captured voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data and returns the text data to the device.

[1207] Input: Audio data

[1208] Output: Text data

[1209] Step 3:

[1210] Natural Language Processing (NLP)

[1211] The device sends the text data to a natural language processing engine (e.g., Google Cloud Natural Language), which analyzes the text data, identifies the user's intent, such as "What to order: Pizza," and returns the analysis results to the device.

[1212] Input: Text data

[1213] Output: User intent (e.g., "Order: Pizza")

[1214] Step 4:

[1215] emotion recognition

[1216] The device further transmits the text data to an emotion recognition engine (e.g., IBM Watson Tone Analyzer), which analyzes the text data and identifies the user's emotion (e.g., "stressed"), and returns the emotion recognition result to the device.

[1217] Input: Text data

[1218] Output: Emotion analysis result (e.g. "stressed")

[1219] Step 5:

[1220] Selecting and Launching an Application

[1221] Based on the analysis results (user intentions and emotions), the device selects an appropriate application (e.g., a food delivery application) from the server, sets the necessary parameters, and launches the application. For example, if the user is feeling stressed, the device will prioritize a food delivery application that is easy to operate.

[1222] Input: User intent, sentiment analysis results

[1223] Output: Launch of selected application

[1224] Step 6:

[1225] Obtaining order results

[1226] The server will then contact the selected application to confirm the order and obtain delivery information (arrival time, shipping costs, etc.), which involves using the application's API to request the required information and return the results.

[1227] Input: Application request

[1228] Output: Delivery information (e.g. arrival time, shipping fee)

[1229] Step 7:

[1230] Audio notifications

[1231] The device sends the acquired delivery information to a speech synthesis engine, which generates concise and easy-to-understand voice data. For example, it generates a voice that says, "Your pizza will arrive in 20 minutes." The device plays this voice data and notifies the user. The notification tone and speed are adjusted based on the user's emotions (e.g., stress level).

[1232] Input: Delivery information

[1233] Output: Audio notification

[1234] Specific examples

[1235] Example prompt sentence:

[1236] The user says, "I want to order a pizza."

[1237] The voice recognition engine converts the sentence "I would like to order a pizza" into text.

[1238] The NLP engine extracts "What to order: Pizza."

[1239] The emotion engine recognizes the user as being in a "stressed state."

[1240] Select a food delivery application considering stress conditions.

[1241] Suggests pizza types and potential stores.

[1242] Select "Margherita Pizza" and confirm your order.

[1243] The order information is retrieved and a voice notification is given saying, "Your pizza will arrive in 20 minutes."

[1244] The audio input device has completed audio notification.

[1245] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1246] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1247] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1248] [Fourth embodiment]

[1249] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1250] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1251] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1252] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1253] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1254] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1255] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1256] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1257] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1258] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1259] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1260] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1261] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1262] This invention is a system that captures voice input, selects and launches an appropriate application according to the user's intention, and notifies the user of the result by voice. Specific embodiments for implementing this system are described below.

[1263] 1. Receiving voice input

[1264] The user speaks to their smartphone, saying, "I want to go from here to where I want to go." For example, they might say, "I want to go from Shibuya to Shinjuku." The device captures the voice data through a microphone. This voice data is temporarily stored on the device.

[1265] 2. Voice Recognition

[1266] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice input of "I want to go from Shibuya to Shinjuku" is converted into text "I want to go from Shibuya to Shinjuku." This text data is temporarily saved as it will be used in the next step.

[1267] 3. Natural Language Processing (NLP)

[1268] The device analyzes the text data using a natural language processing (NLP) engine. The NLP engine identifies the user's intent and extracts the necessary parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" may be extracted. The analysis results are also temporarily saved.

[1269] 4. Select and launch the app

[1270] The device selects an appropriate application based on the analysis results and prepares to launch it. For example, if the analysis determines that the user has the intent to "move," the device selects a transit guide application. The device prepares to pass the necessary parameters (such as departure and arrival points) to the application.

[1271] 5. Result acquisition and voice notification

[1272] The server accesses the specified API endpoint and obtains the information requested by the user. For example, a user requests "route from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. The obtained information is then returned from the server to the device.

[1273] The device converts the received result into voice data using a speech synthesis engine. For example, a notification such as "The next train leaves at 10:30 and costs 300 yen" is generated. This voice data is played back from the device's speaker, providing the user with an audible notification.

[1274] Specific examples

[1275] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[1276] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[1277] 2. The device captures the voice and converts it into text using a speech recognition engine: "I want to go from Shibuya to Shinjuku."

[1278] 3. The device analyzes the text using an NLP engine and extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[1279] 4. The device selects the transit information application and prepares to launch it.

[1280] 5. The server retrieves route information from Shibuya to Shinjuku from the API and returns details (next train time, fare, etc.) to the terminal.

[1281] 6. The terminal converts the result of the operation into voice and announces, "The next train leaves at 10:30 and costs 300 yen."

[1282] 7. The device launches the transit information application and displays detailed information.

[1283] In this way, the present invention provides a system that enables even elderly people and users who are unfamiliar with technology to easily operate a desired application and obtain necessary information.

[1284] The processing flow will be explained below.

[1285] Step 1:

[1286] The user speaks into the smartphone to input the necessary instructions, for example, "I want to go from Shibuya to Shinjuku." This voice data is then captured by the smartphone's microphone.

[1287] Step 2:

[1288] The device sends the captured voice data to its internal voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku."

[1289] Step 3:

[1290] The device sends the converted text data to a natural language processing (NLP) engine, which analyzes the text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted.

[1291] Step 4:

[1292] The device checks the analysis results and determines which application is appropriate. In this case, since the user's intention is related to "travel," the transit guide application is selected.

[1293] Step 5:

[1294] The terminal prepares to launch the selected transit information application by passing necessary parameters (e.g., departure point, arrival point).

[1295] Step 6:

[1296] The server processes API requests for transit information. For example, a request like "https: / / api.transit.com / getInfo?saddr=Shibuya&daddr=Shinjuku" is sent to retrieve information such as "The next train departs at 10:30 and costs 300 yen."

[1297] Step 7:

[1298] The server returns the acquired information to the terminal in JSON format.

[1299] Step 8:

[1300] The terminal analyzes the information it receives and extracts the necessary information (e.g., the time of the next train, the fare).

[1301] Step 9:

[1302] The device sends the extracted information to a speech synthesis engine, which converts it into voice data. For example, the generated voice data would say, "The next train leaves at 10:30 and costs 300 yen."

[1303] Step 10:

[1304] The terminal plays the generated voice data and notifies the user, so that the user receives the necessary information by voice.

[1305] In this way, this system combines voice recognition and natural language processing to select and launch the appropriate application based on the user's voice input, and then returns the results in voice form, implementing a series of processes.

[1306] Example 1

[1307] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1308] In modern society, when elderly people and users unfamiliar with technology use smartphones to obtain information, the operation is often complicated, making it difficult to obtain information. In particular, when obtaining information using voice input, there is a demand for an efficient system that can smoothly and intuitively perform the entire process from voice recognition to application selection and notification of results.

[1309] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1310] In this invention, the server includes: means for capturing voice input; means for converting the voice input into text data; means for analyzing the text data and identifying the user's intent; means for selecting and launching appropriate software based on the user's intent; means for acquiring the execution results of the software and notifying the user by voice; means for capturing the voice input with a microphone and converting it into text data using a voice recognition engine; means for analyzing the text data with a natural language processing engine and extracting necessary parameters; means for sending a request to the server based on the parameters and acquiring necessary information from a specified API endpoint; and means for converting the acquired information into voice data using a voice synthesis engine and notifying the user. This enables even elderly people and users who are not familiar with technology to easily acquire desired information using only voice input.

[1311] "Voice input" refers to the act of a user giving instructions by voice, which is captured as digital voice data via a microphone.

[1312] "Text data" refers to data that has been converted into text form after voice input has been analyzed by a voice recognition engine.

[1313] "Analysis" refers to the process of using a natural language processing engine to identify the user's intent and required information from text data.

[1314] "User intent" refers to the action or information the user is seeking, as determined from the voice input.

[1315] "Software" refers to application programs that are selected and launched based on the user's intent.

[1316] A "microphone" is a hardware device for capturing audio input.

[1317] A "speech recognition engine" is a software or hardware mechanism for converting voice data captured by a microphone into text data.

[1318] A "natural language processing engine" is a software mechanism that analyzes text data and extracts the user's intent and necessary parameters.

[1319] "Parameters" refer to necessary information extracted from the analyzed text data, specifically, the departure and arrival points.

[1320] A "server" is a program or device with a specific function that retrieves information from a specified API based on a user request.

[1321] "API endpoint" refers to the interface of an external service that a server accesses to obtain information.

[1322] A "speech synthesis engine" is software for converting text data into voice data.

[1323] "Voice notification" refers to the act of notifying a user of information using voice data generated by a voice synthesis engine.

[1324] The present invention relates to a system that allows a user to obtain information by voice input and notifies the user of the information by voice. Specific means for implementing this system will be described below.

[1325] overview

[1326] The system of this invention captures and analyzes voice input, selects and launches appropriate software according to the user's intention, and notifies the user of the results by voice, making it easy for even elderly users and those unfamiliar with technology to obtain information.

[1327] Hardware and software configuration

[1328] 1. Receiving voice input

[1329] The user issues specific instructions to the smartphone. For example, the user says, "I want to go from Shibuya to Shinjuku." The device's microphone captures this voice input and generates voice data. This voice data is stored in the device's temporary memory.

[1330] 2. Voice Recognition

[1331] The device sends the voice data to a voice recognition engine (for example, a commercial voice recognition API). The voice recognition engine converts the captured voice data into text data. The converted text data is in the format of, for example, "I want to go from Shibuya to Shinjuku." This text data is also temporarily stored in the device.

[1332] 3. Natural Language Processing (NLP)

[1333] The device sends the text data to a natural language processing (NLP) engine (e.g., a commercial NLP API). The NLP engine analyzes the text data and extracts important parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" is extracted. This parameter information is also temporarily stored on the device.

[1334] 4. Select and launch the app

[1335] The device selects the appropriate software based on the analysis results. For example, if the user has the intent to "move," the device selects transit information software (e.g., a commercial transit information app). The device prepares to launch the selected application and collects the necessary parameters (start and end locations).

[1336] 5. Result acquisition and voice notification

[1337] The device sends the analysis results (parameters) to the server, and the server accesses the specified API endpoint (for example, a commercial API endpoint) to obtain the required information. For example, the server requests "route information from Shibuya to Shinjuku" and obtains the next train time, fare, etc. The obtained information is returned from the server to the device.

[1338] The device converts the acquired information into speech using a speech synthesis engine (for example, a commercial speech synthesis API). The speech data will be in the form of "The next train leaves at 10:30 and costs 300 yen." This speech data is played back to the user through the device's speaker.

[1339] Specific examples

[1340] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[1341] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[1342] 2. The device's microphone captures the audio data and stores it in temporary memory.

[1343] 3. The device sends the voice data to a speech recognition engine and converts it into text.

[1344] 4. The device sends the text data to a natural language processing engine, which extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[1345] 5. The device selects the transit information software and prepares to launch it.

[1346] 6. The terminal sends the extracted parameters to the server.

[1347] 7. The server requests the API endpoint and obtains the route information.

[1348] 8. The server returns the acquired information (next train time, fare) to the terminal.

[1349] 9. The information received by the device is converted into voice using a speech synthesis engine, and the user is notified that "The next train leaves at 10:30 and costs 300 yen."

[1350] 10. The device will launch the transit information software and display detailed information.

[1351] Prompt Sentence Examples

[1352] "If I say I want to go from Shibuya to Shinjuku, please explain the specific process."

[1353] In this way, the present invention enables even elderly people and users who are unfamiliar with technology to quickly and easily obtain desired information using only voice input.

[1354] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1355] Step 1: Receiving voice input

[1356] The user issues specific instructions to the smartphone. For example, the user might say, "I want to go from Shibuya to Shinjuku." The device's microphone captures this voice input and generates digital audio data, which is then stored in the device's temporary memory.

[1357] Input: User voice input

[1358] Output: Audio data

[1359] Step 2: Voice Recognition

[1360] The device sends the voice data to a voice recognition engine (for example, a commercial voice recognition API). The voice recognition engine converts the captured voice data into text data. For example, the text data is converted into text data in the format "I want to go from Shibuya to Shinjuku." This text data is temporarily stored in the device.

[1361] Input: Audio data

[1362] Output: Text data

[1363] Step 3: Natural Language Processing (NLP)

[1364] The device sends the text data to a natural language processing (NLP) engine (e.g., a commercial NLP API). The NLP engine analyzes the text data and extracts important parameters. For example, information such as "Departure: Shibuya" and "Arrival: Shinjuku" is extracted. This parameter information is also temporarily stored on the device.

[1365] Input: Text data

[1366] Output: Extracted parameters (e.g. "Departure: Shibuya", "Destination: Shinjuku")

[1367] Step 4: Select and launch the app

[1368] The device selects the appropriate software based on the analysis results. For example, if the analysis results indicate that the user has the intention of "moving," the device selects transit information software. Next, the device prepares to launch the selected application and collects the necessary parameters (starting point and destination).

[1369] Input: Extracted parameters

[1370] Output: Software selection and launch preparation

[1371] Step 5: Get the results and hear the audio

[1372] The device sends the analysis results (parameters) to the server. The server accesses the specified API endpoint and obtains the information requested by the user. For example, the server requests "route information from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. This collected information is returned from the server to the device. The device then converts the obtained information into voice data using a speech synthesis engine and notifies the user, "The next train leaves at 10:30 and costs 300 yen."

[1373] Input: Analysis results (parameters)

[1374] Output: Acquired information and its spoken notification

[1375] In this way, a series of steps is realized, starting with the user's voice input, followed by voice recognition, natural language processing, application selection and launch, and result acquisition and voice notification. Appropriate data processing and data calculation are performed at each step, ultimately resulting in a system that provides the user with the information they need intuitively and efficiently.

[1376] (Application example 1)

[1377] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1378] Current food delivery services require users to manually operate the application, which is particularly cumbersome for elderly people and those who are not familiar with technology. Also, visually impaired users have difficulty operating smartphones. Therefore, there is a need for a method that allows everyone to easily use food delivery services.

[1379] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1380] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying the user's intention, means for selecting and launching an appropriate application based on the user's intention, means for acquiring the execution result of the application and notifying the user by voice, and means for including an application related to the food delivery service. This makes it possible to use the food delivery service with just voice input, and enables anyone to easily perform and use the ordering procedure.

[1381] "Means for capturing audio input" refers to a device and its associated software for detecting and recording audio input.

[1382] "Means for converting speech input into text data" refers to software and algorithms for converting speech signals into written information.

[1383] "Means for analyzing text data and identifying user intent" refers to natural language processing technology and related software for analyzing text data and understanding the user's intended operations and requests.

[1384] "Means for selecting and launching an appropriate application" refers to a processing system for automatically selecting and launching a particular application based on a user's intent.

[1385] "Means for obtaining the results of application execution and notifying the user by voice" refers to technology and devices for collecting the results or data of an executed application and providing it to the user as voice.

[1386] "Food delivery service application" refers to a software application that allows a user to voice-order food and process the delivery.

[1387] "Device for capturing an acoustic signal" refers to a hardware device, such as a microphone, for obtaining audio input.

[1388] "Speech recognition engine" refers to software and algorithms for processing captured acoustic signals and converting them into text data.

[1389] This invention is a system that captures voice input, selects and launches an appropriate application according to the user's intention, and notifies the user of the results by voice. Specific embodiments for implementing this system are described below.

[1390] 1. Receiving voice input

[1391] A user speaks to the device, saying, "I want to order a pizza," or, for example, "I want to order a Margherita from a food delivery service." The device captures the voice data through a microphone, a device that captures acoustic signals, and this voice data is temporarily stored within the device.

[1392] 2. Voice Recognition

[1393] The device sends the captured voice data to a speech recognition engine, which converts the voice data into text data. For example, a voice input of "I would like to order a margherita from a food delivery service" is converted into text "I would like to order a margherita from a food delivery service." This text data is temporarily stored for use in the next step.

[1394] 3. Natural Language Processing (NLP)

[1395] The device analyzes the text data using a natural language processing (NLP) engine. The NLP engine identifies the user's intent and extracts necessary parameters. For example, information such as "Vendor: food delivery service" and "Item: Margherita" may be extracted. The analysis results are also temporarily stored.

[1396] 4. Select and launch the app

[1397] The device selects an appropriate application based on the analysis results and prepares to launch it. For example, if the analysis reveals that the user has the intent to "order," a food delivery service application will be selected. The device prepares to pass the necessary parameters (vendor, product, etc.) to the application.

[1398] 5. Result acquisition and voice notification

[1399] The device accesses the specified API endpoint and obtains information according to the user's request. For example, a request such as "I would like to order a margherita from a food delivery service" is sent, and confirmation information about the order is collected. The obtained information is returned from the server to the device. The device then converts the received results into voice data using a speech synthesis engine. For example, a notification such as "Your order has been completed. Your margherita will arrive in 30 minutes" is generated. This voice data is played from the device's speaker, providing the user with an audible notification.

[1400] Specific examples

[1401] For example, let's say a user says, "I want to order a Margherita from a food delivery service." Here's what happens:

[1402] 1. A user says, "I want to order a margherita from a food delivery service."

[1403] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "I'd like to order a margherita from a food delivery service."

[1404] 3. The device analyzes the text using an NLP engine and extracts the parameters "Vendor: food delivery service" and "Item: Margherita."

[1405] 4. The device selects the food delivery service application and prepares to launch it.

[1406] 5. The server retrieves the "Margherita order information from the food delivery service" from the API and returns the details (order confirmation information) to the terminal.

[1407] 6. The terminal converts the execution result into voice and notifies the user by saying, "Your order has been completed. Your Margherita will arrive in 30 minutes."

[1408] 7. The device launches the food delivery service application and displays detailed information.

[1409] Prompt Sentence Examples

[1410] "Using speech recognition, we have received the text 'I would like to order a margherita from a food delivery service.' We will proceed with the order based on the order details extracted using NLP analysis (vendor: food delivery service, product: margherita). Once the order is complete, we will announce the message 'Your order is complete. Your margherita will arrive in 30 minutes.'"

[1411] In this way, the present invention provides a system that allows even elderly or less technically savvy users to easily operate a food delivery service and obtain the information they need.

[1412] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1413] Step 1:

[1414] The user speaks to the device, saying, "I'd like to order a Margherita from a food delivery service." The input is voice data, which is captured through the device's microphone. This voice data is temporarily stored in the device.

[1415] Step 2:

[1416] The device sends the captured voice data to a speech recognition engine, which converts the input voice data into text data, and the resulting text data is "I would like to order a Margherita from a food delivery service." This text data is also temporarily stored.

[1417] Step 3:

[1418] The device analyzes the text data using a natural language processing (NLP) engine. The input is converted text data, and the data is processed through analysis to identify the user's intent. Specific instruction parameters such as "Vendor: food delivery service" and "Item: Margherita" are extracted as output. The analysis results are also temporarily stored.

[1419] Step 4:

[1420] The device selects the appropriate application based on the analysis results and prepares to launch it. The input is the parameters obtained from the analysis, and the application is selected based on the data. The output is the identity of the application to be launched, for example, a food delivery service application is selected. The device prepares to pass the necessary parameters (vendor, product, etc.) to the application.

[1421] Step 5:

[1422] The terminal accesses the specified API endpoint and retrieves the information requested by the user. The input is the instruction parameters selected by the application, and a request based on them is sent to the API endpoint. The output is order confirmation information and other related data. The retrieved information is returned from the server to the terminal.

[1423] Step 6:

[1424] The device converts the received results into voice data using a speech synthesis engine. The input is the order confirmation information obtained from the API, and data processing is performed to convert it into voice data. The output is voice notification data such as "Your order is complete. Your Margherita will arrive in 30 minutes." This voice data is played from the device's speaker, providing the user with an audible notification.

[1425] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1426] The present invention is a system that captures voice input, selects and launches an appropriate application according to the user's intentions and emotions, and notifies the user of the results by voice. Specific embodiments for implementing this system are described below.

[1427] 1. Receiving voice input

[1428] The user speaks to their smartphone, saying, "I want to go from here to where I want to go." For example, they might say, "I want to go from Shibuya to Shinjuku." The device captures the voice data through a microphone. This voice data is temporarily stored on the device.

[1429] 2. Voice Recognition

[1430] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku." This text data is temporarily saved as it will be used in the next step.

[1431] 3. Natural Language Processing (NLP)

[1432] The device sends the text data to a natural language processing (NLP) engine. The NLP engine analyzes the text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted. This analysis result is also temporarily saved.

[1433] 4. Emotion recognition

[1434] The device further transmits the text data to the emotion engine. The emotion engine analyzes the text data and recognizes the user's emotion. For example, if the user is feeling stressed, the emotion engine identifies this as "stress." The emotion recognition result is also temporarily stored.

[1435] 5. Select and launch the app

[1436] Based on the analysis results, the device selects an appropriate application and prepares to launch it. For example, if the user's intention is related to "travel" and "stress" is also recognized, the device will select a transit guide application. The device will pass the necessary parameters (departure point, destination) to the application and instruct it to launch.

[1437] 6. Result acquisition and voice notification

[1438] The server accesses the specified API endpoint and retrieves the information requested by the user. For example, a request for "route from Shibuya to Shinjuku" is made, and information such as the time and fare of the next train is collected. The retrieved information is then returned from the server to the device.

[1439] The device analyzes the received results and extracts the necessary information (such as the next train time and fare). The content of the notification is adjusted based on the emotion engine's results. For example, if the user is feeling stressed, the notification content will be made concise and easy to understand.

[1440] 7. Notification

[1441] The device sends the extracted information to a speech synthesis engine and converts it into voice data. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated. This voice data is played back from the device's speaker to notify the user. The notification is made in an appropriate tone and speed based on the emotion.

[1442] Specific examples

[1443] Let's take the example of a user saying, "I want to go from Shibuya to Shinjuku." The specific flow of operations is as follows:

[1444] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[1445] 2. The device captures the voice and sends it to a speech recognition engine, which converts it into text such as "I want to go from Shibuya to Shinjuku."

[1446] 3. The device sends this text to the NLP engine, which extracts the parameters "Departure: Shibuya" and "Arrival: Shinjuku."

[1447] 4. The device sends the text to the emotion engine, which recognizes that the user is in a "stressed state."

[1448] 5. The device selects the transit guide application, sets parameters, and launches it.

[1449] 6. The server uses the transfer information API to obtain route information and returns the next train time and fare to the terminal.

[1450] 7. The device analyzes the results, generates voice data in a concise format taking into account the stress level, and then uses a voice synthesis engine to announce, "The next train leaves at 10:30 and costs 300 yen."

[1451] 8. The device plays this audio data and notifies the user.

[1452] In this way, the present invention provides a system that combines voice recognition, natural language processing, and emotion recognition based on voice input, allowing even elderly people and users who are unfamiliar with technology to easily operate the desired application and obtain information.

[1453] The processing flow will be explained below.

[1454] Step 1:

[1455] The user speaks into the smartphone, saying, "I want to go from Shibuya to Shinjuku." This voice data is captured by the smartphone's microphone.

[1456] Step 2:

[1457] The device sends the captured voice data to a voice recognition engine, which converts the voice data into text data. For example, a voice saying "I want to go from Shibuya to Shinjuku" becomes text data saying "I want to go from Shibuya to Shinjuku."

[1458] Step 3:

[1459] The device sends the converted text data to a natural language processing (NLP) engine. The NLP engine analyzes this text data and identifies the user's intent. As a result of the analysis, the parameters "Departure: Shibuya" and "Arrival: Shinjuku" are extracted. This analysis result is temporarily saved.

[1460] Step 4:

[1461] The device then sends the text data to the emotion engine, which analyzes the text data and recognizes the user's emotion. For example, if the user is feeling stressed, the emotion engine identifies the user's emotion as "stress." The emotion recognition result is also temporarily saved.

[1462] Step 5:

[1463] The device determines which application is appropriate based on the analysis results and emotion recognition results. For example, if the user's intention is related to "travel" and "stress" is recognized, the device will select a transit guide application.

[1464] Step 6:

[1465] The terminal passes the parameters (starting point, destination) required for the selected transit information application and prepares to launch the application.

[1466] Step 7:

[1467] The server accesses the API endpoint for transit information and retrieves information according to the user's request. For example, a request such as "https: / / api.transit.com / getInfo?saddr=Shibuya&daddr=Shinjuku" is sent to retrieve information such as "The next train departs at 10:30, and the fare is 300 yen." The retrieved information is returned to the terminal in JSON format.

[1468] Step 8:

[1469] The device analyzes the information it receives and extracts the necessary information (time of the next train, fare, etc.). The content of the notification is adjusted taking into account the results of the emotion engine. For example, if the user is feeling stressed, the notification content will be made concise and easy to understand.

[1470] Step 9:

[1471] The device sends the extracted information to a speech synthesis engine, which converts it into voice data. For example, the generated voice data would say, "The next train leaves at 10:30 and costs 300 yen."

[1472] Step 10:

[1473] The device plays the generated voice data over the speaker to notify the user, allowing the user to receive the necessary information through voice. The notification content is played back in an appropriate tone and speed based on the user's emotion.

[1474] Through these steps, the system can accurately recognize the user's voice input, select an appropriate application based on the user's set intention and emotion, and effectively notify the user of the results through voice.

[1475] Example 2

[1476] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1477] Conventional speech recognition systems convert speech input into text and provide a means to select and launch applications based on the user's intent. However, they face the challenge of responding appropriately while taking into account the user's emotions. Providing appropriate and easy-to-understand notifications is particularly important for elderly users and those unfamiliar with technology. Therefore, a system that can recognize not only the user's intent but also their emotions based on speech input and adjust notification content as needed is highly desirable.

[1478] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1479] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying a user's intention, means for analyzing the text data and recognizing the user's emotion, means for selecting and launching an appropriate application based on the user's intention and emotion, means for obtaining the execution result of the application and notifying the user by voice, and means for adjusting the notification based on the user's emotion, thereby enabling elderly people and users who are unfamiliar with technology to obtain information in an optimal form according to their emotions.

[1480] "Voice input" is a method by which a user gives instructions or queries to a system using voice.

[1481] "Text data" is a data format that expresses voice input as text information.

[1482] "Analysis" is the act of processing text data and other data to extract information such as intent and emotion.

[1483] "Intention" is information indicating the purpose for which the user inputs voice.

[1484] "Emotion" is information that indicates the psychological state of the user, and includes states such as stress and joy.

[1485] An "application" is a software program designed to perform a particular function.

[1486] "Launch" is an operation that puts an application into an executable state.

[1487] The "execution result" is the output information generated after the application has performed its processing.

[1488] A "notification" is the act of providing information to a user, often in audio form.

[1489] "Adjustment" refers to the act of changing the information provided and its format to the optimum form depending on the user's condition.

[1490] This invention is a system that allows a user to input voice, analyzes the voice input, recognizes the user's intentions and emotions, and selects and launches an appropriate application. Furthermore, when notifying the user of the results of application execution by voice, it has a function to adjust the notification content based on the user's emotions.

[1491] Hardware and software used

[1492] The system uses the following hardware and software:

[1493] Audio input capture hardware: Microphones built into mobile devices such as smartphones and tablets

[1494] Speech recognition engine: Google Speech-to-Text API, IBM Watson Speech to Text, etc.

[1495] Natural Language Processing (NLP) engines: Google Natural Language API, Amazon Comprehend, etc.

[1496] Emotion recognition engine: Microsoft Azure Text Analytics for Sentiment Analysis, etc.

[1497] Application launch software: Route guidance applications such as NAVITIME and Google Maps

[1498] Speech synthesis engine: Google Cloud Text-to-Speech API, Amazon Polly, etc.

[1499] Program processing

[1500] Receiving voice input

[1501] The user speaks to the smartphone, saying, "I want to go from here to where I want to go." For example, "I want to go from Shibuya to Shinjuku." The device captures this voice data through the smartphone's microphone and temporarily stores it on the device.

[1502] Voice Recognition

[1503] The device sends the captured voice data to a speech recognition engine, which uses the Google Speech-to-Text API or similar to convert it into text data such as "I want to go from Shibuya to Shinjuku." This text data is temporarily stored for use in the next processing step.

[1504] Natural Language Processing (NLP)

[1505] The device sends the text data obtained through voice recognition to an NLP engine, which analyzes the user's intent. Using Google Natural Language API and other tools, it extracts parameters such as "Departure: Shibuya" and "Arrival: Shinjuku." The analysis results are also temporarily saved.

[1506] emotion recognition

[1507] The device then sends the obtained text data to an emotion recognition engine to recognize the user's emotions. Using Microsoft Azure Text Analytics and other tools, the device determines whether the user is feeling "stressed." The emotion recognition results are also temporarily stored.

[1508] Selecting and launching the app

[1509] Based on these analysis results, the device selects an appropriate application and prepares to launch it. For example, if the user's intention is "travel" and the emotion is recognized as "stress," a route guidance application such as NAVITIME will be selected. The device passes the necessary parameters (starting point, destination) to the application and instructs it to launch.

[1510] Result acquisition and voice notification

[1511] The server accesses the specified API endpoint and retrieves information according to the user's request. For example, it retrieves "route information from Shibuya to Shinjuku" and returns information such as the next train time and fare to the device. The device analyzes these results and adjusts the notification content taking the user's emotions into account.

[1512] notification

[1513] The device then sends the formatted notification to a speech synthesis engine, which converts it into voice data. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated and notified to the user through the device's speaker. The tone and speed of the voice are adjusted based on the user's emotions.

[1514] Specific examples

[1515] For example, let's say a user says, "I want to go from Shibuya to Shinjuku." Here's the specific operation flow:

[1516] 1. The user speaks, "I want to go from Shibuya to Shinjuku."

[1517] 2. The device captures the audio and converts it into text using the Google Speech-to-Text API: "I want to go from Shibuya to Shinjuku."

[1518] 3. The device analyzes this text using the Google Natural Language API and extracts "Departure: Shibuya" and "Destination: Shinjuku."

[1519] 4. The device analyzes emotions using Microsoft Azure Text Analytics and recognizes that the user is in a "stressed state."

[1520] 5. The device selects and launches the NAVITIME transit information application and sets the parameters.

[1521] 6. The server obtains route information using NAVITIME's API and returns the information "next train time: 10:30" and "fare: 300 yen" to the terminal.

[1522] 7. The device analyzes this information and, taking into account the stress level, generates a concise voice message such as, "The next train leaves at 10:30 and costs 300 yen."

[1523] 8. The device plays the generated audio data and notifies the user.

[1524] Prompt Sentence Examples

[1525] "Please tell me when the next train leaves and how much it costs."

[1526] This system is particularly useful for elderly people and users who are unfamiliar with technology, as it can understand the user's intentions and emotions and provide appropriate information via voice, simply by inputting the user's voice.

[1527] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1528] Step 1:

[1529] The user speaks to the smartphone, saying, "From where do I want to go to where?" For example, "I want to go from Shibuya to Shinjuku." The device captures this voice data through the smartphone's microphone. The voice data is input and output in a format that is temporarily stored within the device. Specifically, the smartphone's microphone converts the voice signal into digital data, and then records that digital data.

[1530] Step 2:

[1531] The device sends the captured voice data to a voice recognition engine. Here, the Google Speech-to-Text API is used. Voice data is sent to this API as input, and the converted text data is received as output. For example, a voice saying "I want to go from Shibuya to Shinjuku" is converted into text data saying "I want to go from Shibuya to Shinjuku." In concrete terms, the device sends voice data as an API request and receives text data as an API response.

[1532] Step 3:

[1533] The device sends the text data obtained by speech recognition to the NLP engine. Here, the Google Natural Language API is used. The text data is sent as input to the NLP engine, and the analyzed intent and parameters are received as output. For example, parameters such as "Departure: Shibuya" and "Arrival: Shinjuku" are extracted from the text. Specifically, the device sends the text data as an API request and receives response data with the intent and parameters extracted.

[1534] Step 4:

[1535] The device sends the obtained text data to an emotion recognition engine. In this case, Microsoft Azure Text Analytics is used. The text data is sent as input to the emotion recognition engine, and the recognized emotion information is received as output. For example, if the user is feeling "stressed," it is identified as "stress." Specifically, the device sends the text data as an API request and receives the emotion recognition results as response data.

[1536] Step 5:

[1537] The device selects an appropriate application based on the results of natural language processing and emotion recognition, and prepares to launch it. For example, if the user's intent is "travel" and "stress" is recognized, a route guidance application such as NAVITIME will be selected. The device also prepares to pass the necessary parameters (starting point, destination) to the application. Specifically, it determines the application name and parameters, and generates an API request to launch the application.

[1538] Step 6:

[1539] The server accesses the specified API endpoint and retrieves the information requested by the user. For example, it uses the NAVITIME API to retrieve "route information from Shibuya to Shinjuku" and collects information such as the time and fare of the next train. The collected data is then sent back to the device. Specifically, it sends an information request to the API endpoint and retrieves route information as response data.

[1540] Step 7:

[1541] The device analyzes the received information and adjusts the notification content taking into account the user's emotions. For example, if the user is feeling "stressed," the notification content is converted into a concise and easy-to-understand format. Route information (next train time, fare, etc.) is input, and the adjusted voice notification content is output. Specifically, the device analyzes the response data and reformats the text according to the user's emotions.

[1542] Step 8:

[1543] The device sends the adjusted notification content to a speech synthesis engine and converts it into voice data. Here, the Google Cloud Text-to-Speech API is used. The text data of the notification content is sent as input to the speech synthesis engine, and voice data is received as output. For example, voice data such as "The next train leaves at 10:30 and costs 300 yen" is generated. Specifically, the text data is sent as a request to the speech synthesis API, and voice data is received as a response.

[1544] Step 9:

[1545] The device plays the generated voice data and notifies the user. The voice data is input and output as voice through the smartphone's speaker. Specifically, the device's voice playback function is used to play the voice from the speaker.

[1546] (Application example 2)

[1547] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1548] In recent years, there has been a demand for systems that can analyze user intent based on voice input and launch the appropriate application. Interfaces that combine voice input and analysis are particularly useful for users who want to easily order delivery or perform complex operations. However, existing systems often struggle to provide appropriate suggestions that take the user's emotions into account, and the methods of notifying users of information are often not optimal. Therefore, it is necessary to provide a system that can analyze the user's intent and emotions, select and launch the appropriate application based on the results, and notify the user of the results via voice.

[1549] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1550] In this invention, the server includes means for capturing voice input, means for converting the voice input into text data, means for analyzing the text data and identifying the user's intention and emotion, means for selecting and launching an appropriate application based on the user's intention and emotion, and means for obtaining the execution result of the application and notifying the user by voice, thereby enabling delivery orders and other complex operations to be performed simply and optimally through voice input.

[1551] "Voice input" is the process of capturing the user's spoken words and storing them as digital data.

[1552] "Text data" is an information format in which voice data is converted into a string of characters.

[1553] "User intent" refers to the goal or request that the user is trying to achieve through voice input.

[1554] "User emotions" refers to the psychological and emotional state that can be read from the user's comments.

[1555] "Application selection and launch" is the process of choosing and running the most appropriate software program based on the user's intent and emotions.

[1556] "Application Results" refers to the final information or output provided by a selected and launched application.

[1557] "Voice notification" refers to the act of communicating the results of an application's execution to the user as voice using speech synthesis technology.

[1558] A "transit application" is a software program that allows a user to search for and provide route information from one point to another.

[1559] A "food delivery application" is a software program that allows a user to order food and delivers the food to a specified location.

[1560] An "audio input device" is hardware that captures a user's voice and processes it as a digital signal.

[1561] A "speech recognition engine" is software or algorithms for analyzing captured voice data and converting it into text form.

[1562] This system analyzes the user's intentions and emotions based on the user's voice input, selects and launches an appropriate application, and notifies the user of the results via voice. This system is primarily applied in the food delivery field, helping users to easily order food through voice input.

[1563] Hardware and software used

[1564] Hardware:

[1565] Audio input device: A device that captures a user's voice. Examples include smart glasses and smartphones.

[1566] Server: A central processing unit for processing and analyzing data.

[1567] software:

[1568] Speech recognition engine: Software for converting voice data into text data, such as the Google Cloud Speech-to-Text API.

[1569] Natural Language Processing (NLP) engine: Software for analyzing text data and identifying user intent, such as the Google Cloud Natural Language API.

[1570] Emotion recognition engine: Software that analyzes user emotions from text data, such as IBM Watson Tone Analyzer.

[1571] Speech synthesis engine: Software for converting text data into speech, such as the Google Text-to-Speech API.

[1572] Processing flow

[1573] 1. Audio input capture:

[1574] A voice input device (smart glasses or a smartphone) captures the user's voice, for example, the user says, "I'd like to order a pizza."

[1575] 2. Speech Recognition:

[1576] The captured voice data is sent to a voice recognition engine, which converts the voice into text data.

[1577] 3. Natural Language Processing (NLP):

[1578] The text data is sent to an NLP engine, which analyzes the user's intent and extracts the item to be ordered (in this example, "pizza").

[1579] 4. Emotion recognition:

[1580] Furthermore, the text data is sent to an emotion recognition engine to analyze the user's emotions (e.g., stress state).

[1581] 5. Select and launch the application:

[1582] Based on the analysis results, the system selects and launches an appropriate food delivery application. For example, if the user is feeling stressed, an application that allows for easy ordering will be selected.

[1583] 6. Obtaining and notifying order results:

[1584] The order information is acquired and then a voice synthesis engine delivers it to the user in a relaxed tone, depending on the user's emotions.

[1585] Specific examples

[1586] For example, if a user says "I'd like to order a pizza":

[1587] 1. User: Says, "I want to order a pizza."

[1588] 2. Voice input device: Captures voice and sends it to the voice recognition engine.

[1589] 3. Speech recognition engine: Converts voice data into text data such as "I would like to order pizza."

[1590] 4. Natural language processing engine: Analyzes text data and extracts "Order: Pizza".

[1591] 5. Emotion recognition engine: Analyzes text data and recognizes when the user is in a "stressed state."

[1592] 6. Server: Select a food delivery application (e.g., food delivery service).

[1593] 7. Server: The food delivery application confirms the user's order.

[1594] 8. Speech synthesis engine: Generates the order result as a voice saying, "The pizza will arrive in 20 minutes."

[1595] 9. Audio input device: Plays audio data to the user and notifies them.

[1596] Prompt Sentence Examples

[1597] An example of a prompt is:

[1598] The user says, "I want to order a pizza."

[1599] The voice recognition engine converts the sentence "I would like to order a pizza" into text.

[1600] The NLP engine extracts "What to order: Pizza."

[1601] The emotion engine recognizes the user as being in a "stressed state."

[1602] Select a food delivery application considering stress conditions.

[1603] Suggests pizza types and potential stores.

[1604] Select "Margherita Pizza" and confirm your order.

[1605] The order information is retrieved and a voice notification is given saying, "Your pizza will arrive in 20 minutes."

[1606] The audio input device has completed audio notification.

[1607] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1608] Step 1:

[1609] Voice input capture

[1610] A user speaks into a voice input device (e.g., smart glasses or a smartphone) saying, "I want to order a pizza." The voice input device captures the voice through a microphone and temporarily stores it as voice data.

[1611] Input: User's voice

[1612] Output: Audio data

[1613] Step 2:

[1614] Voice Recognition

[1615] The device sends the captured voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data and returns the text data to the device.

[1616] Input: Audio data

[1617] Output: Text data

[1618] Step 3:

[1619] Natural Language Processing (NLP)

[1620] The device sends the text data to a natural language processing engine (e.g., Google Cloud Natural Language), which analyzes the text data, identifies the user's intent, such as "What to order: Pizza," and returns the analysis results to the device.

[1621] Input: Text data

[1622] Output: User intent (e.g., "Order: Pizza")

[1623] Step 4:

[1624] emotion recognition

[1625] The device further transmits the text data to an emotion recognition engine (e.g., IBM Watson Tone Analyzer), which analyzes the text data and identifies the user's emotion (e.g., "stressed"), and returns the emotion recognition result to the device.

[1626] Input: Text data

[1627] Output: Emotion analysis result (e.g. "stressed")

[1628] Step 5:

[1629] Selecting and Launching an Application

[1630] Based on the analysis results (user intentions and emotions), the device selects an appropriate application (e.g., a food delivery application) from the server, sets the necessary parameters, and launches the application. For example, if the user is feeling stressed, the device will prioritize a food delivery application that is easy to operate.

[1631] Input: User intent, sentiment analysis results

[1632] Output: Launch of selected application

[1633] Step 6:

[1634] Obtaining order results

[1635] The server will then contact the selected application to confirm the order and obtain delivery information (arrival time, shipping costs, etc.), which involves using the application's API to request the required information and return the results.

[1636] Input: Application request

[1637] Output: Delivery information (e.g. arrival time, shipping fee)

[1638] Step 7:

[1639] Audio notifications

[1640] The device sends the acquired delivery information to a speech synthesis engine, which generates concise and easy-to-understand voice data. For example, it generates a voice that says, "Your pizza will arrive in 20 minutes." The device plays this voice data and notifies the user. The notification tone and speed are adjusted based on the user's emotions (e.g., stress level).

[1641] Input: Delivery information

[1642] Output: Audio notification

[1643] Specific examples

[1644] Example prompt sentence:

[1645] The user says, "I want to order a pizza."

[1646] The voice recognition engine converts the sentence "I would like to order a pizza" into text.

[1647] The NLP engine extracts "What to order: Pizza."

[1648] The emotion engine recognizes the user as being in a "stressed state."

[1649] Select a food delivery application considering stress conditions.

[1650] Suggests pizza types and potential stores.

[1651] Select "Margherita Pizza" and confirm your order.

[1652] The order information is retrieved and a voice notification is given saying, "Your pizza will arrive in 20 minutes."

[1653] The audio input device has completed audio notification.

[1654] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1655] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1656] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1657] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1658] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1659] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1660] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1661] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1662] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1663] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1664] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1665] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1666] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1667] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1668] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1669] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1670] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1671] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1672] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1673] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1674] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1675] The following is further disclosed regarding the above embodiment.

[1676] (Claim 1)

[1677] means for capturing audio input;

[1678] means for converting the voice input into text data;

[1679] means for analyzing the text data and identifying the user's intention;

[1680] means for selecting and launching an appropriate application based on the user's intent;

[1681] means for acquiring the execution result of the application and notifying the user by voice;

[1682] A system including:

[1683] (Claim 2)

[1684] 2. The system of claim 1, wherein the means for selecting an appropriate application is a transit guide application or a map application.

[1685] (Claim 3)

[1686] 10. The system of claim 1, wherein the means for receiving speech input comprises a microphone and a speech recognition engine for processing speech signals from the microphone.

[1687] "Example 1"

[1688] (Claim 1)

[1689] means for capturing audio input;

[1690] means for converting the voice input into text data;

[1691] means for analyzing the text data and identifying the user's intention;

[1692] means for selecting and launching appropriate software based on the user's intent;

[1693] means for acquiring the execution result of the software and notifying the user by voice;

[1694] means for capturing said speech input with a microphone and converting it into text data using a speech recognition engine;

[1695] means for analyzing the text data using a natural language processing engine and extracting necessary parameters;

[1696] A means for sending a request to a server based on the parameters and obtaining necessary information from a specified API endpoint;

[1697] means for converting the acquired information into voice data using a voice synthesis engine and notifying the user;

[1698] A system including:

[1699] (Claim 2)

[1700] 2. The system of claim 1, wherein the means for selecting the appropriate software is transit guide software or map software.

[1701] (Claim 3)

[1702] 10. The system of claim 1, wherein the means for receiving speech input comprises a microphone and a speech recognition engine for processing speech signals from the microphone.

[1703] "Application Example 1"

[1704] (Claim 1)

[1705] means for capturing audio input;

[1706] means for converting the voice input into text data;

[1707] means for analyzing the text data and identifying the user's intention;

[1708] means for selecting and launching an appropriate application based on the user's intent;

[1709] means for acquiring the execution result of the application and notifying the user by voice;

[1710] means including an application relating to a food delivery service;

[1711] A system including:

[1712] (Claim 2)

[1713] 10. The system of claim 1, wherein the means for selecting an appropriate application is a food delivery application or an e-commerce application.

[1714] (Claim 3)

[1715] 10. The system of claim 1, wherein the means for receiving speech input includes a device for capturing an acoustic signal and a speech recognition engine for processing the acoustic signal.

[1716] "Example 2: Combining Emotion Engines"

[1717] (Claim 1)

[1718] means for capturing audio input;

[1719] means for converting the voice input into text data;

[1720] means for analyzing the text data and identifying the user's intention;

[1721] means for analyzing the text data and recognizing a user's emotion;

[1722] means for selecting and launching an appropriate application based on the user's intentions and emotions;

[1723] means for acquiring the execution result of the application and notifying the user by voice;

[1724] means for adjusting the notification based on a user's emotion;

[1725] A system including:

[1726] (Claim 2)

[1727] 2. The system of claim 1, wherein the means for selecting an appropriate application is a route guidance application or a map guidance application.

[1728] (Claim 3)

[1729] 10. The system of claim 1, wherein the means for receiving speech input includes an acoustic device and a speech recognition device for processing the acoustic signal from the acoustic device.

[1730] "Application example 2 when combining emotion engines"

[1731] (Claim 1)

[1732] means for capturing audio input;

[1733] means for converting the voice input into text data;

[1734] means for analyzing the text data and identifying a user's intent and emotion;

[1735] means for selecting and launching an appropriate application based on the user's intentions and emotions;

[1736] means for acquiring the execution result of the application and notifying the user by voice;

[1737] A system including:

[1738] (Claim 2)

[1739] 10. The system of claim 1, wherein the means for selecting an appropriate application comprises a transit application or a food delivery application.

[1740] (Claim 3)

[1741] 2. The system of claim 1, wherein the means for receiving speech input comprises a speech input device and a speech recognition engine for processing speech signals from the speech input device. [Explanation of symbols]

[1742] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for capturing audio input; means for converting the voice input into text data; means for analyzing the text data and identifying the user's intention; means for selecting and launching an appropriate application based on the user's intent; means for acquiring the execution result of the application and notifying the user by voice; A system including:

2. The system of claim 1 , wherein the means for selecting an appropriate application is a transit guide application or a map application.

3. 2. The system of claim 1, wherein the means for receiving speech input comprises a microphone and a speech recognition engine for processing speech signals from the microphone.

Images