System

Abstract

A system is provided.SOLUTION: A system, comprising: voice recognition means; natural language processing means; a terminal for capturing a voice command of a user; a server for performing a payment procedure based on the captured voice command; means for confirming account information and a payment amount of the user; means for notifying the user of the payment amount and requesting a confirmation response of "OK"; and means for receiving the confirmation response of the user and completing the payment.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] QR code payments are widespread in modern society, but they require users to scan the QR code and enter the amount, which is time-consuming. Store clerks also need to present and verify the QR code, which can reduce productivity. Furthermore, situations requiring visual confirmation can be slowed down by congestion or obstacles. There is a need for a way to resolve these issues and provide a smoother, more comfortable payment experience. [Means for solving the problem]

[0005] The present invention provides a payment system that uses voice recognition means and natural language processing means and includes a terminal for capturing user voice commands, a server that executes payment procedures based on the captured voice commands, means for confirming the user's account information and payment amount, means for notifying the user of the payment amount and requesting an "OK" confirmation response, and means for completing the payment upon receiving the user's confirmation response. This system eliminates the need for QR code exchange and enables quick and efficient payment using voice.

[0006] "Speech recognition means" refers to technology that converts human speech into digital signals, analyzes the content, and outputs it as text data.

[0007] "Natural language processing means" refers to technology that understands and analyzes text data as natural human language.

[0008] "Terminal" refers to a device for receiving input from a user and transmitting data to a server.

[0009] A "server" refers to a computer system that processes data received from terminals over a network and performs various operations such as payment processing.

[0010] "Account information" refers to important data such as a user's personal information, payment information, and authentication data.

[0011] "Settlement Amount" means the amount due by a User in a particular Transaction.

[0012] "Confirmation response" refers to a response made by a user in response to a confirmation message from a server or terminal.

[0013] "Means for completing payment" refers to the process or technology for actually debiting the user's account and completing the transaction after receiving the user's confirmation response. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention relates to a system that uses voice recognition and natural language processing to analyze the conversation between the user and the store clerk and executes payments by voice without using a QR code. This system provides smooth and efficient payments by linking the user's voice instructions, server processing, and terminal operation.

[0036] Program Overview

[0037] The system operates as follows:

[0038] User voice instructions

[0039] The user issues a voice command such as "Please use voice." The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet.

[0040] Server-based voice analysis

[0041] The server passes the received voice data to a speech recognition module, converts it into text, and uses a natural language processing module to analyze the text data and confirm the user's intent.The server then retrieves the user's account information and payment amount from the database.

[0042] Generate and notify confirmation messages

[0043] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[0044] User Acknowledgment

[0045] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[0046] Server response confirmation

[0047] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[0048] Notification of payment completion

[0049] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[0050] Specific examples

[0051] scenario

[0052] Consider the example of a user purchasing a coffee at a cafe.

[0053] User:

[0054] "Audio please."

[0055] Device:

[0056] The user's voice is captured and sent to the server.

[0057] server:

[0058] The received voice data is converted into text format and recognized as "Voice please." Next, the user account information and payment amount (for example, 500 yen) are obtained from the database. A message is generated saying "500 yen, right? Say OK," and sent to the terminal.

[0059] Device:

[0060] The user is notified by voice, "It's 500 yen. Please say OK."

[0061] User:

[0062] Respond with "OK."

[0063] Device:

[0064] Capture the user's "OK" and send it to the server.

[0065] server:

[0066] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[0067] Device:

[0068] The user is notified by voice that "Payment has been completed."

[0069] This process allows payments to be completed quickly and efficiently using only voice, without the need for a QR code. This system is particularly easy to use for people with visual impairments or those who have difficulty operating digital devices, and simplifies the payment process at stores.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user issues a voice command such as "Please speak to me." The device captures this voice command and stores it as voice data.

[0073] Step 2:

[0074] The device sends the captured audio data to the server, and the server receives the audio data.

[0075] Step 3:

[0076] The server passes the voice data to the speech recognition module, converts it into text, and verifies that the text is "Please speak."

[0077] Step 4:

[0078] The server retrieves the user's account information and transaction amount from a database, then generates a text response message saying, "That's XX yen. Say OK."

[0079] Step 5:

[0080] The server generates a confirmation message and sends it to the terminal. The terminal receives the confirmation message, converts it into speech using a speech synthesis engine, and notifies the user.

[0081] Step 6:

[0082] The user responds by saying "OK." The device recaptures the user's voice data and saves it as new voice data.

[0083] Step 7:

[0084] The device sends newly captured audio data to the server, which receives it.

[0085] Step 8:

[0086] The server passes the received voice data to the voice recognition module and converts it into text format as "OK." If "OK" is confirmed, the payment process begins.

[0087] Step 9:

[0088] The server debits the user's account for the specified amount and verifies that the transaction was successful.

[0089] Step 10:

[0090] The server generates a message saying "Payment has been completed" and sends it to the terminal. The terminal converts the received message into voice and notifies the user that "Payment has been completed."

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] Traditional payment systems require the use of QR codes or manual input of information, making them difficult to use, especially for the visually impaired and those with limited digital literacy. There is also a lack of voice-based systems to make payment procedures quick and efficient.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes means for capturing a user's voice command, means for transmitting the captured voice command over a network, speech recognition means for converting the voice data into text format, natural language processing means for analyzing the text data, means for confirming the user's intention and acquiring account information and the payment amount, means for notifying the user of the payment amount and requesting an "OK" confirmation response, means for capturing the user's confirmation response and transmitting it again to the server, means for confirming the converted text data and starting the payment process, and means for generating a payment completion message and notifying the user. This enables the user to complete payment quickly and efficiently using only voice, without using a QR code.

[0096] The "means for capturing user voice commands" is hardware or software for obtaining voice commands uttered by a user.

[0097] The "means for transmitting the captured voice command over a network" refers to a means for transmitting the captured voice data to a server over a communication network.

[0098] The "voice recognition means for converting voice data into text format" is a technology that can convert acquired voice data into text information.

[0099] "Natural language processing means for analyzing text data" is a technology for analyzing the meaning of converted text data and understanding the user's intentions.

[0100] "Means for confirming the user's intent and obtaining account information and payment amount" refers to means for determining the user's intent from the analyzed text and obtaining the necessary account information and payment amount from a database or the like.

[0101] The "means for notifying the user of the payment amount and requesting a confirmation response of "OK"" is a means for transmitting the acquired payment amount to the user and requesting a response of "OK" as confirmation to proceed with the payment.

[0102] The "means for capturing the user's confirmation response and sending it again to the server" refers to a means for capturing the user's voice response again and sending that data to the server.

[0103] The "means for verifying the converted text data and initiating the payment process" is a means for analyzing the user's response and initiating the payment procedure if appropriate verification is obtained.

[0104] The "means for generating a payment completion message and notifying the user" is a means for generating a message indicating that the payment procedure has been completed normally and conveying the contents of the message to the user.

[0105] This invention is a system that utilizes voice recognition and natural language processing technology to perform smooth and efficient payments between users and the system. This system is realized by coordinating user voice instructions, server processing, and terminal operation.

[0106] System configuration

[0107] Hardware and Software

[0108] 1. Terminal: A device that allows users to issue voice commands. The terminal is equipped with a high-sensitivity microphone that captures the user's voice. The terminal also has the function of transmitting voice data to a server via the Internet.

[0109] 2. Server: A central processing unit that receives voice data and performs speech recognition and natural language processing. Specifically, it uses the following modules:

[0110] Speech recognition module: Converts voice data into text using the Google Cloud Speech-to-Text API or similar.

[0111] Natural language processing module: Analyzes text data using IBM Watson NLP and other technologies to understand user intent.

[0112] Database management system: Manages user account information and payment amounts using a database such as MySQL.

[0113] 3. Speech synthesis engine: This technology uses Amazon Polly and other technologies to notify users of confirmation messages and payment completion messages sent from the server via voice.

[0114] Processing flow

[0115] The user issues a voice command to the device, saying, "Please use voice." The device captures this voice and sends the generated voice data to the server.

[0116] The server converts the received voice data into text using the Google Cloud Speech-to-Text API, which is then analyzed by IBM Watson NLP to understand the user's intent.

[0117] The server retrieves the user's account information and payment amount from the database, generates a confirmation message saying, "It's XX yen. Please say OK." This message is sent to the terminal.

[0118] The device uses the Amazon Polly speech synthesis engine to audibly notify the user of a confirmation message. If the user responds "OK," the device captures the audio again and sends the new audio data to the server.

[0119] The server performs voice recognition again, confirms the user's "OK", and then starts the payment process. The specified amount is deducted from the user's account and the payment is completed.

[0120] The server generates a message saying "Payment completed" and sends it to the terminal, which converts this message into speech using the Amazon Polly speech synthesis engine and notifies the user.

[0121] Specific examples

[0122] scenario

[0123] Take the example of a user buying coffee at a cafe.

[0124] User

[0125] Give the voice command "Voice please."

[0126] Terminal

[0127] The user's voice is captured and sent to the server.

[0128] server

[0129] The Google Cloud Speech-to-Text API is used to convert the voice data into text and recognize the phrase "Please speak." Next, IBM Watson NLP is used to analyze the text and retrieve the user account information and payment amount (e.g., 500 yen) from the database. A confirmation message saying "500 yen, right? Say OK" is sent to the terminal.

[0130] Terminal

[0131] The user is notified by voice, "It's 500 yen. Please say OK."

[0132] User

[0133] Respond with a voice saying "OK."

[0134] Terminal

[0135] Capture the user's "OK" and send it to the server.

[0136] server

[0137] After confirming "OK" with the Google Cloud Speech-to-Text API, the payment process is initiated and 500 yen is deducted from the user's account. The completion of the payment is confirmed and a message stating "Payment completed" is sent to the terminal.

[0138] Terminal

[0139] The user is notified by voice that "payment has been completed."

[0140] The system uses voice recognition and natural language processing to provide fast and efficient payments, and is particularly convenient for users with visual impairments and those who have difficulty operating digital devices.

[0141] Prompt Sentence Examples

[0142] "Please explain a system that allows users to use voice commands to make payments without using a QR code. The system uses voice recognition and natural language processing, and includes the user's voice instructions, processing by the server, and terminal operation. Please provide a specific example and explain the detailed steps."

[0143] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0144] Step 1:

[0145] The user issues a voice command at a store, saying, "Please use voice." The device captures this voice command with a high-sensitivity microphone. The input is the user's voice, and the output is audio data in WAV format.

[0146] Step 2:

[0147] The terminal sends the generated WAV format audio data to a server via the Internet. The input is WAV format audio data, and the output is audio data sent via the network.

[0148] Step 3:

[0149] The server passes the received voice data to the Google Cloud Speech-to-Text API. The input is WAV format voice data, which is converted into text format. The output is the text data "Please speak by voice."

[0150] Step 4:

[0151] The server passes the text data to the IBM Watson NLP module, which analyzes the user's intent. Specifically, it understands from the text data "Voice please" that the user's intent is to start payment. The input is the text data "Voice please," and the output is the analysis result that the user's intent is to start payment.

[0152] Step 5:

[0153] The server retrieves the user's account information and payment amount from the MySQL database. Specifically, it uses an API to retrieve the account ID and balance information, and extracts the payment amount. The input is the user's account ID, and the output is the account information and payment amount (for example, 500 yen).

[0154] Step 6:

[0155] The server generates a confirmation message saying "It's 500 yen. Please say OK" based on the acquired account information and payment amount. The input is the user's account information and payment amount, and the output is the text data of the confirmation message.

[0156] Step 7:

[0157] The server generates a confirmation message and sends it to the terminal in JSON format. The input is the text data of the confirmation message, and the output is the JSON format data sent over the network.

[0158] Step 8:

[0159] The device uses the Amazon Polly speech synthesis engine to notify the user by voice of the received confirmation message, "It's 500 yen. Please say OK." The input is the text data of the confirmation message, and the output is a voice notification.

[0160] Step 9:

[0161] The user responds verbally with "OK." Once again, the device captures this audio with a high-sensitivity microphone. The input is the user's voice, and the output is audio data in a new WAV format.

[0162] Step 10:

[0163] The device then sends the new WAV-formatted audio data to the server again via the Internet. The input is the new WAV-formatted audio data, and the output is the audio data sent over the network.

[0164] Step 11:

[0165] The server again calls the Google Cloud Speech-to-Text API to convert the received audio data into text format. The input is the new WAV audio data, and the output is the text data "OK".

[0166] Step 12:

[0167] The server verifies that the text data is "OK." The input is the text data "OK," and the output is the verification result.

[0168] Step 13:

[0169] The server initiates the payment process and debits the specified amount (500 yen) from the user's account via the API. The input is the user's account information and the payment amount, and the output is confirmation that the debit has been completed.

[0170] Step 14:

[0171] The server confirms that the payment has been completed and generates a message saying "Payment completed." The input is the confirmation of the withdrawal completion, and the output is the text data of the completion message.

[0172] Step 15:

[0173] The server sends the generated completion message to the terminal in JSON format. The input is the text data of the completion message, and the output is the JSON format data sent over the network.

[0174] Step 16:

[0175] The terminal uses the Amazon Polly speech synthesis engine to notify the user by voice of the message "Payment completed." The input is the text data of the completion message, and the output is the voice notification.

[0176] Through the above processing steps, payment can be completed quickly and efficiently using only voice, without using a QR code.

[0177] (Application example 1)

[0178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] While the convenience of electronic payment services has improved in recent years, many systems require visual input, such as QR codes or touch operations. This poses a challenge for users with visual impairments and those unfamiliar with operating digital devices. Furthermore, existing voice payment systems lack sufficient speech recognition accuracy and natural language processing effectiveness, making it difficult to achieve smooth and prompt payments.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0181] In this invention, the server includes a voice recognition unit, a natural language processing unit, and a terminal for capturing the user's voice commands. This allows the user to make payments quickly and accurately using only their voice, without the need for visual input. Furthermore, the user experience can be improved by using voice synthesis to deliver confirmation messages and payment completion notifications.

[0182] "Speech recognition means" is a technology for capturing a user's speech and converting it into text data.

[0183] "Natural language processing means" is a technology for analyzing text data and understanding the user's intent.

[0184] A "terminal for capturing user's voice commands" is a device for collecting user's voice and transmitting it to a server.

[0185] The "payment processing server" is a central control unit that processes payments based on captured voice commands.

[0186] "Means for confirming user account information and payment amount" refers to technology that allows the server to obtain and confirm the user's account information and payment amount.

[0187] "Means for notifying the user of the payment amount and requesting a confirmation response of 'OK'" refers to a technique for notifying the user of the payment amount by voice and requesting confirmation by voice.

[0188] The "means for receiving a user's confirmation response and completing the payment" refers to a technique for obtaining a user's confirmation response and completing the actual payment process.

[0189] "Means for notifying the user using voice synthesis" is a technology that converts a message from the server into voice and notifies the user.

[0190] "Means for notifying the user by voice after payment is completed" refers to a technique for notifying the user by voice that payment has been completed.

[0191] The present invention relates to a system that uses voice recognition and natural language processing to analyze a user's voice commands and execute payments by voice without using a QR code. This system is realized through cooperation between the user, terminals, and a server.

[0192] The system is configured as follows:

[0193] 1. A device that captures the user's voice command: When a user says "pay with voice," this device captures the voice and sends it over the internet to a server. For example, this could be a smartphone or smart glasses.

[0194] 2. Speech recognition method: The server uses speech recognition technology such as the Google Speech-to-Text API to convert the received speech into text data.

[0195] 3. Natural language processing means: Analyze text data using Google NLP API or similar to confirm the user's intent.

[0196] 4. Server that executes the payment process: The server obtains the user's account information and payment amount, and generates a confirmation message in the format "It's XX yen. Please say OK."

[0197] 5. Notifying the user using voice synthesis: The confirmation message from the server is converted into voice using the Google Text-to-Speech API and notified to the user.

[0198] 6. Means for receiving user confirmation response: When the user responds "OK", the terminal again captures this voice and sends it to the server.

[0199] 7. Completion of payment process: The server again uses voice recognition to confirm the user's response, then processes the payment and deducts the specified amount from the user's account.

[0200] 8. A means of notifying the user by voice after payment is completed: When payment is completed, the server generates a completion notification stating "Payment has been completed" and notifies the user by voice via the terminal.

[0201] Specific examples

[0202] Below is an example of a user purchasing a coffee at a cafe.

[0203] A user at a cafe says, "Pay by voice." The device captures this speech and sends it to the server. The server uses speech recognition to transcribe "Pay by voice" and uses natural language processing to understand the user's intent. The server then obtains the user's account information and payment amount (e.g., 500 yen), generates a confirmation message saying, "500 yen, right? Say OK," and converts this into voice to notify the user.

[0204] If the user responds "OK," the device captures the audio again and sends it to the server. After the server confirms this "OK," it starts the payment process, deducting 500 yen from the user's account. Finally, the server generates a message saying "Payment completed," converts it into audio, and notifies the user via the device.

[0205] This system allows even visually impaired people or users who have difficulty operating digital devices to make payments quickly and efficiently using only their voice.

[0206] Example prompt sentence:

[0207] "Voice command to pay"

[0208] "It's 500 yen. Just say OK."

[0209] "Payment completed"

[0210] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0211] Step 1:

[0212] The user says "Pay with voice." This voice command is captured by the terminal.

[0213] Input: The user's voice command.

[0214] Output: The captured audio data.

[0215] Step 2:

[0216] The device transmits the captured audio data to the server.

[0217] Input: The captured audio data.

[0218] Output: Sending audio data to the server.

[0219] Step 3:

[0220] The server converts the received voice data into text data using a voice recognition tool (Google Speech-to-Text API).

[0221] Input: The audio data sent to the server.

[0222] Output: Text data.

[0223] Step 4:

[0224] The server analyzes the text data using natural language processing tools (Google NLP API) and confirms the user's intent.

[0225] Input: Text data.

[0226] Output: Data that represents the user's intent.

[0227] Step 5:

[0228] The server obtains the user's account information and payment amount and generates a confirmation message saying, "It's 500 yen. Please say OK."

[0229] Input: Data representing the user's intent, and a database of account information on the server.

[0230] Output: Confirmation message (text format).

[0231] Step 6:

[0232] The server converts the confirmation message into voice data using a voice synthesis method (Google Text-to-Speech API) and sends it to the terminal.

[0233] Input: Confirmation message (in text format).

[0234] Output: Confirmation message (audio data).

[0235] Step 7:

[0236] The terminal notifies the user of the audio data.

[0237] Input: Confirmation message (audio data).

[0238] Output: Audio notification to the user.

[0239] Step 8:

[0240] The user says "OK." This speech is again captured by the device and sent to the server.

[0241] Input: User's spoken acknowledgment.

[0242] Output: The captured audio data of the acknowledgment.

[0243] Step 9:

[0244] The server converts the received voice data of the confirmation response into text data using a voice recognition method (Google Speech-to-Text API).

[0245] Input: Acknowledgment audio data.

[0246] Output: Acknowledgement text data.

[0247] Step 10:

[0248] After the server confirms "OK," it starts the payment process and deducts the specified amount from the user's account.

[0249] Input: Text data of confirmation response, and account information database in the server.

[0250] Output: Payment processing completed.

[0251] Step 11:

[0252] The server generates a payment completion message, converts it into voice data using a voice synthesis method (Google Text-to-Speech API), and sends it to the terminal.

[0253] Input: Payment completion message (text format).

[0254] Output: Payment completion message (audio data).

[0255] Step 12:

[0256] The terminal notifies the user by voice of a payment completion message.

[0257] Input: Payment completion message (audio data).

[0258] Output: Notification to the user that payment has been completed.

[0259] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0260] This invention relates to a system that combines voice recognition, natural language processing, and emotion recognition to analyze conversations between users and store clerks and execute voice payments without using QR codes. By linking the user's voice instructions, server processing, terminal operation, and emotion recognition, this system provides a smooth, efficient payment experience that takes emotions into consideration for the user.

[0261] Program Overview

[0262] The system operates as follows:

[0263] User voice instructions

[0264] The user issues a voice command such as "Please use voice." The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet.

[0265] Server-based voice analysis

[0266] The server passes the received voice data to a speech recognition module, converts it into text, and uses a natural language processing module to analyze the text data and confirm the user's intention.The server then uses an emotion engine to identify and analyze the user's emotion from the voice data.

[0267] Emotion-Based Processing

[0268] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is anxious, it will choose calming words. On the other hand, if the user is in a calm state, it will respond normally.

[0269] Generate and notify confirmation messages

[0270] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[0271] User Acknowledgment

[0272] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[0273] Server response confirmation

[0274] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[0275] Notification of payment completion

[0276] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[0277] Specific examples

[0278] scenario

[0279] Consider the example of a user purchasing a coffee at a cafe.

[0280] User:

[0281] "Audio please."

[0282] Device:

[0283] The user's voice is captured and sent to the server.

[0284] server:

[0285] The received voice data is converted into text and recognized as "Voice please." Next, the emotion engine is used to analyze the user's tone of voice and determine that the user is a little nervous. The server retrieves the user's account information and payment amount (for example, 500 yen) from the database. It generates a message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon," and sends it to the terminal.

[0286] Device:

[0287] The user is notified by voice, "It's 500 yen. Say OK. Don't worry, it will be over soon."

[0288] User:

[0289] Respond with "OK."

[0290] Device:

[0291] Capture the user's "OK" and send it to the server.

[0292] server:

[0293] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[0294] Device:

[0295] The user is notified by voice that "Payment has been completed."

[0296] This process allows users to experience an emotionally sensitive voice payment experience, particularly reducing confusion and tension. It also allows store clerks to complete payments quickly and efficiently. The system is also easy to use for people with visual impairments and those who have difficulty operating digital devices, simplifying the payment process in stores.

[0297] The processing flow will be explained below.

[0298] Step 1:

[0299] The user issues a voice command such as "Please speak to me." The device captures this voice command through the microphone and stores it as voice data.

[0300] Step 2:

[0301] The device sends the captured audio data to the server, and the server receives the audio data.

[0302] Step 3:

[0303] The server passes the received voice data to the voice recognition module, which converts the voice data into text. The server then analyzes the converted text, "Please speak to me."

[0304] Step 4:

[0305] The server retrieves the user's account information and payment amount from the database, and then uses an emotion engine to identify the user's emotion from the voice data.

[0306] Step 5:

[0307] The server analyzes the emotion engine results and determines that the user is in a tense state. Based on this result, it generates a confirmation message with additional words to reassure the user.

[0308] Step 6:

[0309] The server generates a confirmation message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon." and sends it to the device. The device passes the received confirmation message to the speech synthesis engine, converts it into speech, and notifies the user.

[0310] Step 7:

[0311] The user responds by saying "OK." The device recaptures this audio and saves it as new audio data.

[0312] Step 8:

[0313] The device sends the newly captured audio data to the server, and the server receives the audio data.

[0314] Step 9:

[0315] The server passes the received voice data to the voice recognition module again and converts it into text format. The server confirms "OK".

[0316] Step 10:

[0317] If the server receives the "OK" confirmation, it starts the payment process: the server debits the specified amount from the user's account.

[0318] Step 11:

[0319] The server confirms that the payment process was successful and generates a "Payment completed" message and sends it to the terminal.

[0320] Step 12:

[0321] The terminal passes the received payment completion message to a speech synthesis engine, which converts it into speech and notifies the user, "Payment has been completed."

[0322] Example 2

[0323] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0324] Conventional payment systems require users to use a visual or physical interface to make payments, which is inconvenient for the visually impaired and those who have difficulty operating digital devices. Furthermore, they are unable to provide services that respond to the user's emotional state, leaving room for improvement in the user experience. Furthermore, voice payment methods that do not use physical media such as QR codes are immature, preventing smooth payments.

[0325] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice recognition means, a natural language processing means, a means for capturing a user's voice instruction, a processing device that executes payment based on the captured voice instruction, a means for confirming the user's account information and payment amount, a means for notifying the user of the payment amount and requesting a confirmation response of "OK," a means for receiving the user's confirmation response and completing the payment, and a means for identifying the user's emotional state using an emotion recognition means and generating an appropriate response. This makes it easy for visually impaired people and users who have difficulty operating digital devices to use, and enables smooth voice payment that takes the user's emotions into consideration.

[0326] "Speech recognition means" is a technology that converts voice data into text data.

[0327] "Natural language processing means" is a technology that analyzes text data and understands context and intent.

[0328] A "means for capturing user voice instructions" is any device or technology that collects a user's voice and converts it into a digital form.

[0329] A "processing device that executes a payment procedure based on captured voice instructions" is an electronic device or system that receives voice instruction data and initiates a payment process.

[0330] "Means for confirming user account information and payment amount" refers to technology for obtaining and confirming user account information and payment amount from a database or the like.

[0331] "Means for notifying the user of the payment amount and requesting a confirmation response of 'OK'" refers to a technique or device that notifies the user of the payment amount and requests a verbal response of 'OK'.

[0332] The "means for receiving a user's confirmation response and completing the payment" refers to a technique or device that receives a user's confirmation response of "OK" and completes the payment process.

[0333] "Emotion recognition means" is a technology for identifying emotions from the user's voice and analyzing their state.

[0334] The "means for generating an appropriate response" refers to a technique or device that takes into account the emotional state of the user and generates an optimal response message.

[0335] MODE FOR CARRYING OUT THE INVENTION

[0336] This invention relates to a system that combines voice recognition, natural language processing, and emotion recognition to analyze conversations between users and store clerks and execute voice payments without using QR codes. By linking the user's voice instructions, server processing, terminal operation, and emotion recognition, this system provides a smooth, efficient payment experience that takes emotions into consideration for the user.

[0337] Hardware and Software Usage

[0338] User voice instructions

[0339] A user issues a voice command such as "Please speak to me" at a store. The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet. A high-performance microphone is used as the hardware for capturing the voice.

[0340] Server-based voice analysis

[0341] The server passes the received voice data to a speech recognition module such as Google Cloud Speech-to-Text and converts it into text. It also uses a natural language processing module such as GPT-3 to analyze the text data and confirm the user's intent. The server then uses an emotion engine such as IBM Watson Tone Analyzer to identify and analyze the user's emotions from the voice data.

[0342] Emotion-Based Processing

[0343] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is anxious, it will choose calming words. On the other hand, if the user is in a calm state, it will respond normally.

[0344] Generate and notify confirmation messages

[0345] The server generates a confirmation message saying, "It's XX yen. Please say OK," and sends it to the device. The device receives this confirmation message and uses Microsoft Azure Text-to-Speech as a speech synthesis engine to notify the user by voice.

[0346] User Acknowledgment

[0347] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[0348] Server response confirmation

[0349] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[0350] Notification of payment completion

[0351] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[0352] Specific examples

[0353] scenario

[0354] Consider the example of a user purchasing a coffee at a cafe.

[0355] User:

[0356] "Audio please."

[0357] Device:

[0358] The user's voice is captured and sent to the server.

[0359] server:

[0360] The received voice data is converted into text and recognized as "Voice please." Next, the emotion engine is used to analyze the user's tone of voice and determine that the user is a little nervous. The server retrieves the user's account information and payment amount (for example, 500 yen) from the database. It generates a message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon," and sends it to the terminal.

[0361] Device:

[0362] The user is notified by voice, "It's 500 yen. Say OK. Don't worry, it will be over soon."

[0363] User:

[0364] Respond with "OK."

[0365] Device:

[0366] Capture the user's "OK" and send it to the server.

[0367] server:

[0368] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[0369] Device:

[0370] The user is notified by voice that "Payment has been completed."

[0371] This process allows users to experience an emotionally sensitive voice payment experience, particularly reducing confusion and tension. It also allows store clerks to complete payments quickly and efficiently. The system is also easy to use for people with visual impairments and those who have difficulty operating digital devices, simplifying the payment process in stores.

[0372] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0373] Step 1:

[0374] The user issues a voice command such as "Voice please." The device uses the user's voice instruction as input and captures voice data with a microphone. The device converts the captured voice data into a digital format and sends it to a server via the Internet. The output is the voice data sent to the server.

[0375] Step 2:

[0376] The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) and converts it into text data. In this step, the input voice data is converted into text. The output is in text format, "Please speak by voice."

[0377] Step 3:

[0378] The server passes the text data to a natural language processing module (e.g., GPT-3) to analyze the user's intent. Here, the text data is used as input to analyze and understand the user's intent and request. This process confirms that the user wishes to make a voice payment. The output is the analysis result regarding the user's intent.

[0379] Step 4:

[0380] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to identify and analyze the user's emotion from the voice data. The input is the voice data, and the voice tone and emotional state are analyzed. This process identifies the user's emotional state (e.g., nervousness). The output is the emotion recognition result.

[0381] Step 5:

[0382] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. The input is the emotion recognition results and the analysis results of the user's intention. For example, if the user is nervous, a calming message is generated. The output is a response message.

[0383] Step 6:

[0384] The server retrieves the payment amount (e.g., 500 yen) from the database and creates a confirmation message saying, "It's 500 yen. Please say OK." It also adds an additional message that takes emotion into consideration. The input is the user's account information and the payment amount. The output is a confirmation message.

[0385] Step 7:

[0386] The server sends the generated confirmation message to the terminal. The input is the confirmation message. The terminal receives this message. The output is the terminal that received the confirmation message.

[0387] Step 8:

[0388] The terminal converts the received confirmation message into speech using a speech synthesis engine (e.g., Microsoft Azure Text-to-Speech). The input is the text data of the confirmation message. The terminal notifies the user by voice, "That's 500 yen. Say OK. Don't worry, it will be done soon." The output is a voice notification.

[0389] Step 9:

[0390] The user responds with "OK" by voice. The terminal captures the voice data again using the user's confirmation response as input. The terminal converts this data into a digital format and sends it back to the server. The output is the voice data of the confirmation response sent to the server.

[0391] Step 10:

[0392] The server passes the sent confirmation response voice data to the voice recognition module again and converts it into text data. The input is the confirmation response voice data. The converted text data "OK" is output.

[0393] Step 11:

[0394] The server confirms the textual "OK" and starts the payment process. The input is "OK" as text data and the user's account information. The server deducts the specified amount from the user's account and makes the payment. The output is payment completion data.

[0395] Step 12:

[0396] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." The input is the payment completion data. The output is the payment completion message.

[0397] Step 13:

[0398] The server sends the generated payment completion message to the terminal. The input is the payment completion message. The terminal receives this message. The output is the terminal that received the payment completion message.

[0399] Step 14:

[0400] The terminal converts the received payment completion message into speech using a speech synthesis engine. The input is the text data of the payment completion message. The terminal notifies the user by voice, "Payment has been completed." The output is a voice notification.

[0401] (Application example 2)

[0402] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0403] Conventional voice payment systems rely solely on voice recognition and natural language processing, and have the problem of being unable to provide appropriate responses that take into account the user's emotional state. Furthermore, the payment process often does not proceed smoothly when the user feels stressed or nervous. Furthermore, certain user groups, such as the visually impaired and elderly, have the problem of difficulty operating the system. There is a need to solve these issues and provide users with a comfortable and smooth payment experience.

[0404] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means, a natural language processing means, and an emotion recognition means. This makes it possible to accurately recognize the user's voice command, generate an appropriate response that takes their emotional state into consideration, and execute a smooth payment process that takes the user's emotions into consideration.

[0405] "Speech recognition means" is a general term for a device or program that captures a user's voice commands and converts them into digital text data.

[0406] "Natural language processing means" is a general term for technology or programs that analyze the user's intentions and meaning from text data and determine appropriate responses and actions.

[0407] "Terminal" is a general term for an electronic device that captures a user's voice and transmits that data to a server.

[0408] "Emotion recognition means" is a general term for technology or programs for identifying and analyzing a user's emotional state from voice data or other input.

[0409] "Server" is a general term for a computer system that performs speech recognition, natural language processing, and emotion recognition, and manages and executes the payment process.

[0410] "Account Information" means, collectively, your personal identification information and payment-related data.

[0411] "Payment amount" refers to the amount to be paid by the user.

[0412] "Response generation means" is a general term for a technology or program for generating an appropriate response message based on the analysis results.

[0413] "Acknowledgement" refers to the voice instructions given by the user to proceed with the payment process.

[0414] This invention is a system that combines speech recognition, natural language processing, and emotion recognition, allowing users to make payments quickly and smoothly using voice commands. The system includes a speech recognition unit, a natural language processing unit, an emotion recognition unit, a server, and a terminal that captures the user's voice.

[0415] The specific operation of the system is configured as follows.

[0416] Hardware and Software Configuration

[0417] Device: A smartphone to capture the user's voice.

[0418] Server: A server for processing voice data. A computer system required for speech recognition, natural language processing, and emotion recognition.

[0419] Speech Recognition Tool: Software that converts speech to text using the SpeechRecognition library.

[0420] Natural language processing tools: Software that uses TextBlob to analyze spoken text and understand user intent.

[0421] Emotion Recognition: Software that uses EmotionClassifier (emopy library) to identify user emotions.

[0422] Program processing overview

[0423] 1. User issues a voice command

[0424] The user issues a voice command to the device, saying, "Please use voice." The device captures this voice and sends the voice data to the server.

[0425] 2. Audio analysis by the server

[0426] The server converts the received voice data into text format using a voice recognition unit, then analyzes the text data using a natural language processing unit to confirm the user's intention, and further identifies the user's emotion from the voice data using an emotion recognition unit.

[0427] 3. Emotion-based response generation

[0428] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is nervous, it will choose calming words, and if the user is in a relaxed state, it will respond normally.

[0429] 4. Generating and notifying confirmation messages

[0430] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[0431] 5. User Acknowledgment

[0432] The user responds by saying "OK." This voice is also captured by the device, and new voice data is created and sent to the server.

[0433] 6. The server confirms the response and starts the payment process

[0434] The server again uses the speech recognition means to convert the user's response into text data and confirms "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[0435] 7. Notification of payment completion

[0436] The server confirms that the payment has been completed, generates a completion message, and sends it to the terminal. The terminal then notifies the user of the received message by voice.

[0437] Specific examples

[0438] As a concrete example, let's consider the process when a user purchases coffee at a cafe. In this case, the process is as follows:

[0439] 1. User: "Voice please."

[0440] 2. Device: Captures the user's voice and sends it to the server.

[0441] 3. Server: Converts the received voice data into text and recognizes "Voice please." Analyzes the user's tone of voice using an emotion recognition engine and determines that the user is a little nervous. Retrieves the user's account information and payment amount (e.g., 500 yen) from the database. Generates a message saying, "500 yen, right? Say OK. Don't worry, it will be done soon," and sends it to the device.

[0442] 4. Terminal: A voice message will say, "It's 500 yen. Say OK. Don't worry, it will be done soon."

[0443] 5. User: Responds "OK."

[0444] 6. Terminal: Captures the user's "OK" and sends it to the server.

[0445] 7. Server: After confirming "OK", it deducts 500 yen and starts the payment process. It confirms that the payment has been completed, generates a completion message, and sends it to the terminal.

[0446] 8. Terminal: The user is notified by voice that "Payment has been completed."

[0447] This allows users to experience a smooth payment experience that takes their emotions into consideration.

[0448] Example prompts for generative AI models

[0449] "Generate a calming response message for users who are a little nervous when paying by voice."

[0450] Such a system will enable voice payments that are easy to use, especially for those with visual impairments or who have difficulty operating digital devices.

[0451] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0452] Step 1:

[0453] The user issues a voice command to the device, saying, "Please speak to me." The device captures this voice and generates voice data, which is then sent directly to the server.

[0454] Input: User's voice command

[0455] Output: Audio data (sent to server)

[0456] How it works: When a user speaks into the smartphone application, the microphone captures the voice, and the voice data is transmitted to the server in real time.

[0457] Step 2:

[0458] The server converts the received voice data into text using speech recognition means, which is then analyzed by natural language processing means to ascertain the user's intent.

[0459] Input: Audio data

[0460] Output: Text data

[0461] How it works: The SpeechRecognition library on the server converts the audio data into text, then uses the TextBlob library to analyze the text and extract the user's intent.

[0462] Step 3:

[0463] The server analyzes the user's emotional state using the text data and the voice data with an emotion recognition means, and the analyzed emotional information is further processed.

[0464] Input: Text and audio data

[0465] Output: Emotion recognition result

[0466] Specific operation: The emopy library is used to analyze the user's emotional state from audio and text data, and the results are obtained. For example, states such as "tense" or "calm" are identified.

[0467] Step 4:

[0468] The server generates an appropriate response message based on the emotion recognition results, depending on the user's emotional state. For example, if the user is nervous, it generates a message containing calming words.

[0469] Input: Emotion recognition results, text data

[0470] Output: Response message

[0471] Specific behavior: The server uses the text template to generate a response message appropriate to the emotional state, such as "500 yen. Say OK. Don't worry, it will be over soon."

[0472] Step 5:

[0473] The server sends the generated response message to the terminal, and the terminal uses a speech synthesis engine to notify the user by voice.

[0474] Input: Response message

[0475] Output: Audio notification

[0476] Specific operation: When a device receives a text message sent from the server, it uses a text-to-speech engine to convert the message into audio and convey it to the user.

[0477] Step 6:

[0478] The user responds by saying "OK." This voice is also captured by the device, and new voice data is generated and sent to the server again.

[0479] Input: User's voice response

[0480] Output: Audio data (sent to server)

[0481] What happens: The user says "OK" to confirm. The device's microphone captures this sound and sends it to the server.

[0482] Step 7:

[0483] The server again uses the speech recognition means to convert the user's response into text data and confirms "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[0484] Input: Audio data

[0485] Output: Confirmation "OK" and start of payment process

[0486] What happens: The SpeechRecognition library on the server converts the voice data into text and uses natural language processing to confirm "OK." Once this confirmation is received, the payment process is carried out.

[0487] Step 8:

[0488] The server confirms that the payment has been completed, generates a message stating "Payment has been completed," and sends it to the terminal.

[0489] Input: Confirm payment completion

[0490] Output: Completion message

[0491] Specific operation: Once the payment process is complete, the server generates a confirmation message and sends it to the terminal, which is then notified to the user by the speech synthesis engine.

[0492] This series of steps provides users with an emotionally sensitive and smooth payment experience.

[0493] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0494] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0495] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0496] [Second embodiment]

[0497] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0498] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0499] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0500] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0501] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0502] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0503] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0504] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0505] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0506] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0507] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0508] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0509] This invention relates to a system that uses voice recognition and natural language processing to analyze the conversation between the user and the store clerk and executes payments by voice without using a QR code. This system provides smooth and efficient payments by linking the user's voice instructions, server processing, and terminal operation.

[0510] Program Overview

[0511] The system operates as follows:

[0512] User voice instructions

[0513] The user issues a voice command such as "Please use voice." The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet.

[0514] Server-based voice analysis

[0515] The server passes the received voice data to a speech recognition module, converts it into text, and uses a natural language processing module to analyze the text data and confirm the user's intent.The server then retrieves the user's account information and payment amount from the database.

[0516] Generate and notify confirmation messages

[0517] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[0518] User Acknowledgment

[0519] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[0520] Server response confirmation

[0521] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[0522] Notification of payment completion

[0523] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[0524] Specific examples

[0525] scenario

[0526] Consider the example of a user purchasing a coffee at a cafe.

[0527] User:

[0528] "Audio please."

[0529] Device:

[0530] The user's voice is captured and sent to the server.

[0531] server:

[0532] The received voice data is converted into text format and recognized as "Voice please." Next, the user account information and payment amount (for example, 500 yen) are obtained from the database. A message is generated saying "500 yen, right? Say OK," and sent to the terminal.

[0533] Device:

[0534] The user is notified by voice, "It's 500 yen. Please say OK."

[0535] User:

[0536] Respond with "OK."

[0537] Device:

[0538] Capture the user's "OK" and send it to the server.

[0539] server:

[0540] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[0541] Device:

[0542] The user is notified by voice that "Payment has been completed."

[0543] This process allows payments to be completed quickly and efficiently using only voice, without the need for a QR code. This system is particularly easy to use for people with visual impairments or those who have difficulty operating digital devices, and simplifies the payment process at stores.

[0544] The processing flow will be explained below.

[0545] Step 1:

[0546] The user issues a voice command such as "Please speak to me." The device captures this voice command and stores it as voice data.

[0547] Step 2:

[0548] The device sends the captured audio data to the server, and the server receives the audio data.

[0549] Step 3:

[0550] The server passes the voice data to the speech recognition module, converts it into text, and verifies that the text is "Please speak."

[0551] Step 4:

[0552] The server retrieves the user's account information and transaction amount from a database, then generates a text response message saying, "That's XX yen. Say OK."

[0553] Step 5:

[0554] The server generates a confirmation message and sends it to the terminal. The terminal receives the confirmation message, converts it into speech using a speech synthesis engine, and notifies the user.

[0555] Step 6:

[0556] The user responds by saying "OK." The device recaptures the user's voice data and saves it as new voice data.

[0557] Step 7:

[0558] The device sends newly captured audio data to the server, which receives it.

[0559] Step 8:

[0560] The server passes the received voice data to the voice recognition module and converts it into text format as "OK." If "OK" is confirmed, the payment process begins.

[0561] Step 9:

[0562] The server debits the user's account for the specified amount and verifies that the transaction was successful.

[0563] Step 10:

[0564] The server generates a message saying "Payment has been completed" and sends it to the terminal. The terminal converts the received message into voice and notifies the user that "Payment has been completed."

[0565] Example 1

[0566] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0567] Traditional payment systems require the use of QR codes or manual input of information, making them difficult to use, especially for the visually impaired and those with limited digital literacy. There is also a lack of voice-based systems to make payment procedures quick and efficient.

[0568] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0569] In this invention, the server includes means for capturing a user's voice command, means for transmitting the captured voice command over a network, speech recognition means for converting the voice data into text format, natural language processing means for analyzing the text data, means for confirming the user's intention and acquiring account information and the payment amount, means for notifying the user of the payment amount and requesting an "OK" confirmation response, means for capturing the user's confirmation response and transmitting it again to the server, means for confirming the converted text data and starting the payment process, and means for generating a payment completion message and notifying the user. This enables the user to complete payment quickly and efficiently using only voice, without using a QR code.

[0570] The "means for capturing user voice commands" is hardware or software for obtaining voice commands uttered by a user.

[0571] The "means for transmitting the captured voice command over a network" refers to a means for transmitting the captured voice data to a server over a communication network.

[0572] The "voice recognition means for converting voice data into text format" is a technology that can convert acquired voice data into text information.

[0573] "Natural language processing means for analyzing text data" is a technology for analyzing the meaning of converted text data and understanding the user's intentions.

[0574] "Means for confirming the user's intent and obtaining account information and payment amount" refers to means for determining the user's intent from the analyzed text and obtaining the necessary account information and payment amount from a database or the like.

[0575] The "means for notifying the user of the payment amount and requesting a confirmation response of "OK"" is a means for transmitting the acquired payment amount to the user and requesting a response of "OK" as confirmation to proceed with the payment.

[0576] The "means for capturing the user's confirmation response and sending it again to the server" refers to a means for capturing the user's voice response again and sending that data to the server.

[0577] The "means for verifying the converted text data and initiating the payment process" is a means for analyzing the user's response and initiating the payment procedure if appropriate verification is obtained.

[0578] The "means for generating a payment completion message and notifying the user" is a means for generating a message indicating that the payment procedure has been completed normally and conveying the contents of the message to the user.

[0579] This invention is a system that utilizes voice recognition and natural language processing technology to perform smooth and efficient payments between users and the system. This system is realized by coordinating user voice instructions, server processing, and terminal operation.

[0580] System configuration

[0581] Hardware and Software

[0582] 1. Terminal: A device that allows users to issue voice commands. The terminal is equipped with a high-sensitivity microphone that captures the user's voice. The terminal also has the function of transmitting voice data to a server via the Internet.

[0583] 2. Server: A central processing unit that receives voice data and performs speech recognition and natural language processing. Specifically, it uses the following modules:

[0584] Speech recognition module: Converts voice data into text using the Google Cloud Speech-to-Text API or similar.

[0585] Natural language processing module: Analyzes text data using IBM Watson NLP and other technologies to understand user intent.

[0586] Database management system: Manages user account information and payment amounts using a database such as MySQL.

[0587] 3. Speech synthesis engine: This technology uses Amazon Polly and other technologies to notify users of confirmation messages and payment completion messages sent from the server via voice.

[0588] Processing flow

[0589] The user issues a voice command to the device, saying, "Please use voice." The device captures this voice and sends the generated voice data to the server.

[0590] The server converts the received voice data into text using the Google Cloud Speech-to-Text API, which is then analyzed by IBM Watson NLP to understand the user's intent.

[0591] The server retrieves the user's account information and payment amount from the database, generates a confirmation message saying, "It's XX yen. Please say OK." This message is sent to the terminal.

[0592] The device uses the Amazon Polly speech synthesis engine to audibly notify the user of a confirmation message. If the user responds "OK," the device captures the audio again and sends the new audio data to the server.

[0593] The server performs voice recognition again, confirms the user's "OK", and then starts the payment process. The specified amount is deducted from the user's account and the payment is completed.

[0594] The server generates a message saying "Payment completed" and sends it to the terminal, which converts this message into speech using the Amazon Polly speech synthesis engine and notifies the user.

[0595] Specific examples

[0596] scenario

[0597] Take the example of a user buying coffee at a cafe.

[0598] User

[0599] Give the voice command "Voice please."

[0600] Terminal

[0601] The user's voice is captured and sent to the server.

[0602] server

[0603] The Google Cloud Speech-to-Text API is used to convert the voice data into text and recognize the phrase "Please speak." Next, IBM Watson NLP is used to analyze the text and retrieve the user account information and payment amount (e.g., 500 yen) from the database. A confirmation message saying "500 yen, right? Say OK" is sent to the terminal.

[0604] Terminal

[0605] The user is notified by voice, "It's 500 yen. Please say OK."

[0606] User

[0607] Respond with a voice saying "OK."

[0608] Terminal

[0609] Capture the user's "OK" and send it to the server.

[0610] server

[0611] After confirming "OK" with the Google Cloud Speech-to-Text API, the payment process is initiated and 500 yen is deducted from the user's account. The completion of the payment is confirmed and a message stating "Payment completed" is sent to the terminal.

[0612] Terminal

[0613] The user is notified by voice that "payment has been completed."

[0614] The system uses voice recognition and natural language processing to provide fast and efficient payments, and is particularly convenient for users with visual impairments and those who have difficulty operating digital devices.

[0615] Prompt Sentence Examples

[0616] "Please explain a system that allows users to use voice commands to make payments without using a QR code. The system uses voice recognition and natural language processing, and includes the user's voice instructions, processing by the server, and terminal operation. Please provide a specific example and explain the detailed steps."

[0617] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0618] Step 1:

[0619] The user issues a voice command at a store, saying, "Please use voice." The device captures this voice command with a high-sensitivity microphone. The input is the user's voice, and the output is audio data in WAV format.

[0620] Step 2:

[0621] The terminal sends the generated WAV format audio data to a server via the Internet. The input is WAV format audio data, and the output is audio data sent via the network.

[0622] Step 3:

[0623] The server passes the received voice data to the Google Cloud Speech-to-Text API. The input is WAV format voice data, which is converted into text format. The output is the text data "Please speak by voice."

[0624] Step 4:

[0625] The server passes the text data to the IBM Watson NLP module, which analyzes the user's intent. Specifically, it understands from the text data "Voice please" that the user's intent is to start payment. The input is the text data "Voice please," and the output is the analysis result that the user's intent is to start payment.

[0626] Step 5:

[0627] The server retrieves the user's account information and payment amount from the MySQL database. Specifically, it uses an API to retrieve the account ID and balance information, and extracts the payment amount. The input is the user's account ID, and the output is the account information and payment amount (for example, 500 yen).

[0628] Step 6:

[0629] The server generates a confirmation message saying "It's 500 yen. Please say OK" based on the acquired account information and payment amount. The input is the user's account information and payment amount, and the output is the text data of the confirmation message.

[0630] Step 7:

[0631] The server generates a confirmation message and sends it to the terminal in JSON format. The input is the text data of the confirmation message, and the output is the JSON format data sent over the network.

[0632] Step 8:

[0633] The device uses the Amazon Polly speech synthesis engine to notify the user by voice of the received confirmation message, "It's 500 yen. Please say OK." The input is the text data of the confirmation message, and the output is a voice notification.

[0634] Step 9:

[0635] The user responds verbally with "OK." Once again, the device captures this audio with a high-sensitivity microphone. The input is the user's voice, and the output is audio data in a new WAV format.

[0636] Step 10:

[0637] The device then sends the new WAV-formatted audio data to the server again via the Internet. The input is the new WAV-formatted audio data, and the output is the audio data sent over the network.

[0638] Step 11:

[0639] The server again calls the Google Cloud Speech-to-Text API to convert the received audio data into text format. The input is the new WAV audio data, and the output is the text data "OK".

[0640] Step 12:

[0641] The server verifies that the text data is "OK." The input is the text data "OK," and the output is the verification result.

[0642] Step 13:

[0643] The server initiates the payment process and debits the specified amount (500 yen) from the user's account via the API. The input is the user's account information and the payment amount, and the output is confirmation that the debit has been completed.

[0644] Step 14:

[0645] The server confirms that the payment has been completed and generates a message saying "Payment completed." The input is the confirmation of the withdrawal completion, and the output is the text data of the completion message.

[0646] Step 15:

[0647] The server sends the generated completion message to the terminal in JSON format. The input is the text data of the completion message, and the output is the JSON format data sent over the network.

[0648] Step 16:

[0649] The terminal uses the Amazon Polly speech synthesis engine to notify the user by voice of the message "Payment completed." The input is the text data of the completion message, and the output is the voice notification.

[0650] Through the above processing steps, payment can be completed quickly and efficiently using only voice, without using a QR code.

[0651] (Application example 1)

[0652] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0653] While the convenience of electronic payment services has improved in recent years, many systems require visual input, such as QR codes or touch operations. This poses a challenge for users with visual impairments and those unfamiliar with operating digital devices. Furthermore, existing voice payment systems lack sufficient speech recognition accuracy and natural language processing effectiveness, making it difficult to achieve smooth and prompt payments.

[0654] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0655] In this invention, the server includes a voice recognition unit, a natural language processing unit, and a terminal for capturing the user's voice commands. This allows the user to make payments quickly and accurately using only their voice, without the need for visual input. Furthermore, the user experience can be improved by using voice synthesis to deliver confirmation messages and payment completion notifications.

[0656] "Speech recognition means" is a technology for capturing a user's speech and converting it into text data.

[0657] "Natural language processing means" is a technology for analyzing text data and understanding the user's intent.

[0658] A "terminal for capturing user's voice commands" is a device for collecting user's voice and transmitting it to a server.

[0659] The "payment processing server" is a central control unit that processes payments based on captured voice commands.

[0660] "Means for confirming user account information and payment amount" refers to technology that allows the server to obtain and confirm the user's account information and payment amount.

[0661] "Means for notifying the user of the payment amount and requesting a confirmation response of 'OK'" refers to a technique for notifying the user of the payment amount by voice and requesting confirmation by voice.

[0662] The "means for receiving a user's confirmation response and completing the payment" refers to a technique for obtaining a user's confirmation response and completing the actual payment process.

[0663] "Means for notifying the user using voice synthesis" is a technology that converts a message from the server into voice and notifies the user.

[0664] "Means for notifying the user by voice after payment is completed" refers to a technique for notifying the user by voice that payment has been completed.

[0665] The present invention relates to a system that uses voice recognition and natural language processing to analyze a user's voice commands and execute payments by voice without using a QR code. This system is realized through cooperation between the user, terminals, and a server.

[0666] The system is configured as follows:

[0667] 1. A device that captures the user's voice command: When a user says "pay with voice," this device captures the voice and sends it over the internet to a server. For example, this could be a smartphone or smart glasses.

[0668] 2. Speech recognition method: The server uses speech recognition technology such as the Google Speech-to-Text API to convert the received speech into text data.

[0669] 3. Natural language processing means: Analyze text data using Google NLP API or similar to confirm the user's intent.

[0670] 4. Server that executes the payment process: The server obtains the user's account information and payment amount, and generates a confirmation message in the format "It's XX yen. Please say OK."

[0671] 5. Notifying the user using voice synthesis: The confirmation message from the server is converted into voice using the Google Text-to-Speech API and notified to the user.

[0672] 6. Means for receiving user confirmation response: When the user responds "OK", the terminal again captures this voice and sends it to the server.

[0673] 7. Completion of payment process: The server again uses voice recognition to confirm the user's response, then processes the payment and deducts the specified amount from the user's account.

[0674] 8. A means of notifying the user by voice after payment is completed: When payment is completed, the server generates a completion notification stating "Payment has been completed" and notifies the user by voice via the terminal.

[0675] Specific examples

[0676] Below is an example of a user purchasing a coffee at a cafe.

[0677] A user at a cafe says, "Pay by voice." The device captures this speech and sends it to the server. The server uses speech recognition to transcribe "Pay by voice" and uses natural language processing to understand the user's intent. The server then obtains the user's account information and payment amount (e.g., 500 yen), generates a confirmation message saying, "500 yen, right? Say OK," and converts this into voice to notify the user.

[0678] If the user responds "OK," the device captures the audio again and sends it to the server. After the server confirms this "OK," it starts the payment process, deducting 500 yen from the user's account. Finally, the server generates a message saying "Payment completed," converts it into audio, and notifies the user via the device.

[0679] This system allows even visually impaired people or users who have difficulty operating digital devices to make payments quickly and efficiently using only their voice.

[0680] Example prompt sentence:

[0681] "Voice command to pay"

[0682] "It's 500 yen. Just say OK."

[0683] "Payment completed"

[0684] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0685] Step 1:

[0686] The user says "Pay with voice." This voice command is captured by the terminal.

[0687] Input: The user's voice command.

[0688] Output: The captured audio data.

[0689] Step 2:

[0690] The device transmits the captured audio data to the server.

[0691] Input: The captured audio data.

[0692] Output: Sending audio data to the server.

[0693] Step 3:

[0694] The server converts the received voice data into text data using a voice recognition tool (Google Speech-to-Text API).

[0695] Input: The audio data sent to the server.

[0696] Output: Text data.

[0697] Step 4:

[0698] The server analyzes the text data using natural language processing tools (Google NLP API) and confirms the user's intent.

[0699] Input: Text data.

[0700] Output: Data that represents the user's intent.

[0701] Step 5:

[0702] The server obtains the user's account information and payment amount and generates a confirmation message saying, "It's 500 yen. Please say OK."

[0703] Input: Data representing the user's intent, and a database of account information on the server.

[0704] Output: Confirmation message (text format).

[0705] Step 6:

[0706] The server converts the confirmation message into voice data using a voice synthesis method (Google Text-to-Speech API) and sends it to the terminal.

[0707] Input: Confirmation message (in text format).

[0708] Output: Confirmation message (audio data).

[0709] Step 7:

[0710] The terminal notifies the user of the audio data.

[0711] Input: Confirmation message (audio data).

[0712] Output: Audio notification to the user.

[0713] Step 8:

[0714] The user says "OK." This speech is again captured by the device and sent to the server.

[0715] Input: User's spoken acknowledgment.

[0716] Output: The captured audio data of the acknowledgment.

[0717] Step 9:

[0718] The server converts the received voice data of the confirmation response into text data using a voice recognition method (Google Speech-to-Text API).

[0719] Input: Acknowledgment audio data.

[0720] Output: Acknowledgement text data.

[0721] Step 10:

[0722] After the server confirms "OK," it starts the payment process and deducts the specified amount from the user's account.

[0723] Input: Text data of confirmation response, and account information database in the server.

[0724] Output: Payment processing completed.

[0725] Step 11:

[0726] The server generates a payment completion message, converts it into voice data using a voice synthesis method (Google Text-to-Speech API), and sends it to the terminal.

[0727] Input: Payment completion message (text format).

[0728] Output: Payment completion message (audio data).

[0729] Step 12:

[0730] The terminal notifies the user by voice of a payment completion message.

[0731] Input: Payment completion message (audio data).

[0732] Output: Notification to the user that payment has been completed.

[0733] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0734] This invention relates to a system that combines voice recognition, natural language processing, and emotion recognition to analyze conversations between users and store clerks and execute voice payments without using QR codes. By linking the user's voice instructions, server processing, terminal operation, and emotion recognition, this system provides a smooth, efficient payment experience that takes emotions into consideration for the user.

[0735] Program Overview

[0736] The system operates as follows:

[0737] User voice instructions

[0738] The user issues a voice command such as "Please use voice." The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet.

[0739] Server-based voice analysis

[0740] The server passes the received voice data to a speech recognition module, converts it into text, and uses a natural language processing module to analyze the text data and confirm the user's intention.The server then uses an emotion engine to identify and analyze the user's emotion from the voice data.

[0741] Emotion-Based Processing

[0742] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is anxious, it will choose calming words. On the other hand, if the user is in a calm state, it will respond normally.

[0743] Generate and notify confirmation messages

[0744] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[0745] User Acknowledgment

[0746] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[0747] Server response confirmation

[0748] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[0749] Notification of payment completion

[0750] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[0751] Specific examples

[0752] scenario

[0753] Consider the example of a user purchasing a coffee at a cafe.

[0754] User:

[0755] "Audio please."

[0756] Device:

[0757] The user's voice is captured and sent to the server.

[0758] server:

[0759] The received voice data is converted into text and recognized as "Voice please." Next, the emotion engine is used to analyze the user's tone of voice and determine that the user is a little nervous. The server retrieves the user's account information and payment amount (for example, 500 yen) from the database. It generates a message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon," and sends it to the terminal.

[0760] Device:

[0761] The user is notified by voice, "It's 500 yen. Say OK. Don't worry, it will be over soon."

[0762] User:

[0763] Respond with "OK."

[0764] Device:

[0765] Capture the user's "OK" and send it to the server.

[0766] server:

[0767] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[0768] Device:

[0769] The user is notified by voice that "Payment has been completed."

[0770] This process allows users to experience an emotionally sensitive voice payment experience, particularly reducing confusion and tension. It also allows store clerks to complete payments quickly and efficiently. The system is also easy to use for people with visual impairments and those who have difficulty operating digital devices, simplifying the payment process in stores.

[0771] The processing flow will be explained below.

[0772] Step 1:

[0773] The user issues a voice command such as "Please speak to me." The device captures this voice command through the microphone and stores it as voice data.

[0774] Step 2:

[0775] The device sends the captured audio data to the server, and the server receives the audio data.

[0776] Step 3:

[0777] The server passes the received voice data to the voice recognition module, which converts the voice data into text. The server then analyzes the converted text, "Please speak to me."

[0778] Step 4:

[0779] The server retrieves the user's account information and payment amount from the database, and then uses an emotion engine to identify the user's emotion from the voice data.

[0780] Step 5:

[0781] The server analyzes the emotion engine results and determines that the user is in a tense state. Based on this result, it generates a confirmation message with additional words to reassure the user.

[0782] Step 6:

[0783] The server generates a confirmation message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon." and sends it to the device. The device passes the received confirmation message to the speech synthesis engine, converts it into speech, and notifies the user.

[0784] Step 7:

[0785] The user responds by saying "OK." The device recaptures this audio and saves it as new audio data.

[0786] Step 8:

[0787] The device sends the newly captured audio data to the server, and the server receives the audio data.

[0788] Step 9:

[0789] The server passes the received voice data to the voice recognition module again and converts it into text format. The server confirms "OK".

[0790] Step 10:

[0791] If the server receives the "OK" confirmation, it starts the payment process: the server debits the specified amount from the user's account.

[0792] Step 11:

[0793] The server confirms that the payment process was successful and generates a "Payment completed" message and sends it to the terminal.

[0794] Step 12:

[0795] The terminal passes the received payment completion message to a speech synthesis engine, which converts it into speech and notifies the user, "Payment has been completed."

[0796] Example 2

[0797] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0798] Conventional payment systems require users to use a visual or physical interface to make payments, which is inconvenient for the visually impaired and those who have difficulty operating digital devices. Furthermore, they are unable to provide services that respond to the user's emotional state, leaving room for improvement in the user experience. Furthermore, voice payment methods that do not use physical media such as QR codes are immature, preventing smooth payments.

[0799] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice recognition means, a natural language processing means, a means for capturing a user's voice instruction, a processing device that executes payment based on the captured voice instruction, a means for confirming the user's account information and payment amount, a means for notifying the user of the payment amount and requesting a confirmation response of "OK," a means for receiving the user's confirmation response and completing the payment, and a means for identifying the user's emotional state using an emotion recognition means and generating an appropriate response. This makes it easy for visually impaired people and users who have difficulty operating digital devices to use, and enables smooth voice payment that takes the user's emotions into consideration.

[0800] "Speech recognition means" is a technology that converts voice data into text data.

[0801] "Natural language processing means" is a technology that analyzes text data and understands context and intent.

[0802] A "means for capturing user voice instructions" is any device or technology that collects a user's voice and converts it into a digital form.

[0803] A "processing device that executes a payment procedure based on captured voice instructions" is an electronic device or system that receives voice instruction data and initiates a payment process.

[0804] "Means for confirming user account information and payment amount" refers to technology for obtaining and confirming user account information and payment amount from a database or the like.

[0805] "Means for notifying the user of the payment amount and requesting a confirmation response of 'OK'" refers to a technique or device that notifies the user of the payment amount and requests a verbal response of 'OK'.

[0806] The "means for receiving a user's confirmation response and completing the payment" refers to a technique or device that receives a user's confirmation response of "OK" and completes the payment process.

[0807] "Emotion recognition means" is a technology for identifying emotions from the user's voice and analyzing their state.

[0808] The "means for generating an appropriate response" refers to a technique or device that takes into account the emotional state of the user and generates an optimal response message.

[0809] MODE FOR CARRYING OUT THE INVENTION

[0810] This invention relates to a system that combines voice recognition, natural language processing, and emotion recognition to analyze conversations between users and store clerks and execute voice payments without using QR codes. By linking the user's voice instructions, server processing, terminal operation, and emotion recognition, this system provides a smooth, efficient payment experience that takes emotions into consideration for the user.

[0811] Hardware and Software Usage

[0812] User voice instructions

[0813] A user issues a voice command such as "Please speak to me" at a store. The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet. A high-performance microphone is used as the hardware for capturing the voice.

[0814] Server-based voice analysis

[0815] The server passes the received voice data to a speech recognition module such as Google Cloud Speech-to-Text and converts it into text. It also uses a natural language processing module such as GPT-3 to analyze the text data and confirm the user's intent. The server then uses an emotion engine such as IBM Watson Tone Analyzer to identify and analyze the user's emotions from the voice data.

[0816] Emotion-Based Processing

[0817] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is anxious, it will choose calming words. On the other hand, if the user is in a calm state, it will respond normally.

[0818] Generate and notify confirmation messages

[0819] The server generates a confirmation message saying, "It's XX yen. Please say OK," and sends it to the device. The device receives this confirmation message and uses Microsoft Azure Text-to-Speech as a speech synthesis engine to notify the user by voice.

[0820] User Acknowledgment

[0821] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[0822] Server response confirmation

[0823] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[0824] Notification of payment completion

[0825] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[0826] Specific examples

[0827] scenario

[0828] Consider the example of a user purchasing a coffee at a cafe.

[0829] User:

[0830] "Audio please."

[0831] Device:

[0832] The user's voice is captured and sent to the server.

[0833] server:

[0834] The received voice data is converted into text and recognized as "Voice please." Next, the emotion engine is used to analyze the user's tone of voice and determine that the user is a little nervous. The server retrieves the user's account information and payment amount (for example, 500 yen) from the database. It generates a message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon," and sends it to the terminal.

[0835] Device:

[0836] The user is notified by voice, "It's 500 yen. Say OK. Don't worry, it will be over soon."

[0837] User:

[0838] Respond with "OK."

[0839] Device:

[0840] Capture the user's "OK" and send it to the server.

[0841] server:

[0842] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[0843] Device:

[0844] The user is notified by voice that "Payment has been completed."

[0845] This process allows users to experience an emotionally sensitive voice payment experience, particularly reducing confusion and tension. It also allows store clerks to complete payments quickly and efficiently. The system is also easy to use for people with visual impairments and those who have difficulty operating digital devices, simplifying the payment process in stores.

[0846] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0847] Step 1:

[0848] The user issues a voice command such as "Voice please." The device uses the user's voice instruction as input and captures voice data with a microphone. The device converts the captured voice data into a digital format and sends it to a server via the Internet. The output is the voice data sent to the server.

[0849] Step 2:

[0850] The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) and converts it into text data. In this step, the input voice data is converted into text. The output is in text format, "Please speak by voice."

[0851] Step 3:

[0852] The server passes the text data to a natural language processing module (e.g., GPT-3) to analyze the user's intent. Here, the text data is used as input to analyze and understand the user's intent and request. This process confirms that the user wishes to make a voice payment. The output is the analysis result regarding the user's intent.

[0853] Step 4:

[0854] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to identify and analyze the user's emotion from the voice data. The input is the voice data, and the voice tone and emotional state are analyzed. This process identifies the user's emotional state (e.g., nervousness). The output is the emotion recognition result.

[0855] Step 5:

[0856] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. The input is the emotion recognition results and the analysis results of the user's intention. For example, if the user is nervous, a calming message is generated. The output is a response message.

[0857] Step 6:

[0858] The server retrieves the payment amount (e.g., 500 yen) from the database and creates a confirmation message saying, "It's 500 yen. Please say OK." It also adds an additional message that takes emotion into consideration. The input is the user's account information and the payment amount. The output is a confirmation message.

[0859] Step 7:

[0860] The server sends the generated confirmation message to the terminal. The input is the confirmation message. The terminal receives this message. The output is the terminal that received the confirmation message.

[0861] Step 8:

[0862] The terminal converts the received confirmation message into speech using a speech synthesis engine (e.g., Microsoft Azure Text-to-Speech). The input is the text data of the confirmation message. The terminal notifies the user by voice, "That's 500 yen. Say OK. Don't worry, it will be done soon." The output is a voice notification.

[0863] Step 9:

[0864] The user responds with "OK" by voice. The terminal captures the voice data again using the user's confirmation response as input. The terminal converts this data into a digital format and sends it back to the server. The output is the voice data of the confirmation response sent to the server.

[0865] Step 10:

[0866] The server passes the sent confirmation response voice data to the voice recognition module again and converts it into text data. The input is the confirmation response voice data. The converted text data "OK" is output.

[0867] Step 11:

[0868] The server confirms the textual "OK" and starts the payment process. The input is "OK" as text data and the user's account information. The server deducts the specified amount from the user's account and makes the payment. The output is payment completion data.

[0869] Step 12:

[0870] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." The input is the payment completion data. The output is the payment completion message.

[0871] Step 13:

[0872] The server sends the generated payment completion message to the terminal. The input is the payment completion message. The terminal receives this message. The output is the terminal that received the payment completion message.

[0873] Step 14:

[0874] The terminal converts the received payment completion message into speech using a speech synthesis engine. The input is the text data of the payment completion message. The terminal notifies the user by voice, "Payment has been completed." The output is a voice notification.

[0875] (Application example 2)

[0876] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0877] Conventional voice payment systems rely solely on voice recognition and natural language processing, and have the problem of being unable to provide appropriate responses that take into account the user's emotional state. Furthermore, the payment process often does not proceed smoothly when the user feels stressed or nervous. Furthermore, certain user groups, such as the visually impaired and elderly, have the problem of difficulty operating the system. There is a need to solve these issues and provide users with a comfortable and smooth payment experience.

[0878] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means, a natural language processing means, and an emotion recognition means. This makes it possible to accurately recognize the user's voice command, generate an appropriate response that takes their emotional state into consideration, and execute a smooth payment process that takes the user's emotions into consideration.

[0879] "Speech recognition means" is a general term for a device or program that captures a user's voice commands and converts them into digital text data.

[0880] "Natural language processing means" is a general term for technology or programs that analyze the user's intentions and meaning from text data and determine appropriate responses and actions.

[0881] "Terminal" is a general term for an electronic device that captures a user's voice and transmits that data to a server.

[0882] "Emotion recognition means" is a general term for technology or programs for identifying and analyzing a user's emotional state from voice data or other input.

[0883] "Server" is a general term for a computer system that performs speech recognition, natural language processing, and emotion recognition, and manages and executes the payment process.

[0884] "Account Information" means, collectively, your personal identification information and payment-related data.

[0885] "Payment amount" refers to the amount to be paid by the user.

[0886] "Response generation means" is a general term for a technology or program for generating an appropriate response message based on the analysis results.

[0887] "Acknowledgement" refers to the voice instructions given by the user to proceed with the payment process.

[0888] This invention is a system that combines speech recognition, natural language processing, and emotion recognition, allowing users to make payments quickly and smoothly using voice commands. The system includes a speech recognition unit, a natural language processing unit, an emotion recognition unit, a server, and a terminal that captures the user's voice.

[0889] The specific operation of the system is configured as follows.

[0890] Hardware and Software Configuration

[0891] Device: A smartphone to capture the user's voice.

[0892] Server: A server for processing voice data. A computer system required for speech recognition, natural language processing, and emotion recognition.

[0893] Speech Recognition Tool: Software that converts speech to text using the SpeechRecognition library.

[0894] Natural language processing tools: Software that uses TextBlob to analyze spoken text and understand user intent.

[0895] Emotion Recognition: Software that uses EmotionClassifier (emopy library) to identify user emotions.

[0896] Program processing overview

[0897] 1. User issues a voice command

[0898] The user issues a voice command to the device, saying, "Please use voice." The device captures this voice and sends the voice data to the server.

[0899] 2. Audio analysis by the server

[0900] The server converts the received voice data into text format using a voice recognition unit, then analyzes the text data using a natural language processing unit to confirm the user's intention, and further identifies the user's emotion from the voice data using an emotion recognition unit.

[0901] 3. Emotion-based response generation

[0902] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is nervous, it will choose calming words, and if the user is in a relaxed state, it will respond normally.

[0903] 4. Generating and notifying confirmation messages

[0904] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[0905] 5. User Acknowledgment

[0906] The user responds by saying "OK." This voice is also captured by the device, and new voice data is created and sent to the server.

[0907] 6. The server confirms the response and starts the payment process

[0908] The server again uses the speech recognition means to convert the user's response into text data and confirms "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[0909] 7. Notification of payment completion

[0910] The server confirms that the payment has been completed, generates a completion message, and sends it to the terminal. The terminal then notifies the user of the received message by voice.

[0911] Specific examples

[0912] As a concrete example, let's consider the process when a user purchases coffee at a cafe. In this case, the process is as follows:

[0913] 1. User: "Voice please."

[0914] 2. Device: Captures the user's voice and sends it to the server.

[0915] 3. Server: Converts the received voice data into text and recognizes "Voice please." Analyzes the user's tone of voice using an emotion recognition engine and determines that the user is a little nervous. Retrieves the user's account information and payment amount (e.g., 500 yen) from the database. Generates a message saying, "500 yen, right? Say OK. Don't worry, it will be done soon," and sends it to the device.

[0916] 4. Terminal: A voice message will say, "It's 500 yen. Say OK. Don't worry, it will be done soon."

[0917] 5. User: Responds "OK."

[0918] 6. Terminal: Captures the user's "OK" and sends it to the server.

[0919] 7. Server: After confirming "OK", it deducts 500 yen and starts the payment process. It confirms that the payment has been completed, generates a completion message, and sends it to the terminal.

[0920] 8. Terminal: The user is notified by voice that "Payment has been completed."

[0921] This allows users to experience a smooth payment experience that takes their emotions into consideration.

[0922] Example prompts for generative AI models

[0923] "Generate a calming response message for users who are a little nervous when paying by voice."

[0924] Such a system will enable voice payments that are easy to use, especially for those with visual impairments or who have difficulty operating digital devices.

[0925] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0926] Step 1:

[0927] The user issues a voice command to the device, saying, "Please speak to me." The device captures this voice and generates voice data, which is then sent directly to the server.

[0928] Input: User's voice command

[0929] Output: Audio data (sent to server)

[0930] How it works: When a user speaks into the smartphone application, the microphone captures the voice, and the voice data is transmitted to the server in real time.

[0931] Step 2:

[0932] The server converts the received voice data into text using speech recognition means, which is then analyzed by natural language processing means to ascertain the user's intent.

[0933] Input: Audio data

[0934] Output: Text data

[0935] How it works: The SpeechRecognition library on the server converts the audio data into text, then uses the TextBlob library to analyze the text and extract the user's intent.

[0936] Step 3:

[0937] The server analyzes the user's emotional state using the text data and the voice data with an emotion recognition means, and the analyzed emotional information is further processed.

[0938] Input: Text and audio data

[0939] Output: Emotion recognition result

[0940] Specific operation: The emopy library is used to analyze the user's emotional state from audio and text data, and the results are obtained. For example, states such as "tense" or "calm" are identified.

[0941] Step 4:

[0942] The server generates an appropriate response message based on the emotion recognition results, depending on the user's emotional state. For example, if the user is nervous, it generates a message containing calming words.

[0943] Input: Emotion recognition results, text data

[0944] Output: Response message

[0945] Specific behavior: The server uses the text template to generate a response message appropriate to the emotional state, such as "500 yen. Say OK. Don't worry, it will be over soon."

[0946] Step 5:

[0947] The server sends the generated response message to the terminal, and the terminal uses a speech synthesis engine to notify the user by voice.

[0948] Input: Response message

[0949] Output: Audio notification

[0950] Specific operation: When a device receives a text message sent from the server, it uses a text-to-speech engine to convert the message into audio and convey it to the user.

[0951] Step 6:

[0952] The user responds by saying "OK." This voice is also captured by the device, and new voice data is generated and sent to the server again.

[0953] Input: User's voice response

[0954] Output: Audio data (sent to server)

[0955] What happens: The user says "OK" to confirm. The device's microphone captures this sound and sends it to the server.

[0956] Step 7:

[0957] The server again uses the speech recognition means to convert the user's response into text data and confirms "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[0958] Input: Audio data

[0959] Output: Confirmation "OK" and start of payment process

[0960] What happens: The SpeechRecognition library on the server converts the voice data into text and uses natural language processing to confirm "OK." Once this confirmation is received, the payment process is carried out.

[0961] Step 8:

[0962] The server confirms that the payment has been completed, generates a message stating "Payment has been completed," and sends it to the terminal.

[0963] Input: Confirm payment completion

[0964] Output: Completion message

[0965] Specific operation: Once the payment process is complete, the server generates a confirmation message and sends it to the terminal, which is then notified to the user by the speech synthesis engine.

[0966] This series of steps provides users with an emotionally sensitive and smooth payment experience.

[0967] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0968] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0969] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0970] [Third embodiment]

[0971] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0972] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0973] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0974] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0975] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0976] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0977] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0978] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0979] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0980] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0981] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0982] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0983] This invention relates to a system that uses voice recognition and natural language processing to analyze the conversation between the user and the store clerk and executes payments by voice without using a QR code. This system provides smooth and efficient payments by linking the user's voice instructions, server processing, and terminal operation.

[0984] Program Overview

[0985] The system operates as follows:

[0986] User voice instructions

[0987] The user issues a voice command such as "Please use voice." The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet.

[0988] Server-based voice analysis

[0989] The server passes the received voice data to a speech recognition module, converts it into text, and uses a natural language processing module to analyze the text data and confirm the user's intent.The server then retrieves the user's account information and payment amount from the database.

[0990] Generate and notify confirmation messages

[0991] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[0992] User Acknowledgment

[0993] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[0994] Server response confirmation

[0995] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[0996] Notification of payment completion

[0997] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[0998] Specific examples

[0999] scenario

[1000] Consider the example of a user purchasing a coffee at a cafe.

[1001] User:

[1002] "Audio please."

[1003] Device:

[1004] The user's voice is captured and sent to the server.

[1005] server:

[1006] The received voice data is converted into text format and recognized as "Voice please." Next, the user account information and payment amount (for example, 500 yen) are obtained from the database. A message is generated saying "500 yen, right? Say OK," and sent to the terminal.

[1007] Device:

[1008] The user is notified by voice, "It's 500 yen. Please say OK."

[1009] User:

[1010] Respond with "OK."

[1011] Device:

[1012] Capture the user's "OK" and send it to the server.

[1013] server:

[1014] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[1015] Device:

[1016] The user is notified by voice that "Payment has been completed."

[1017] This process allows payments to be completed quickly and efficiently using only voice, without the need for a QR code. This system is particularly easy to use for people with visual impairments or those who have difficulty operating digital devices, and simplifies the payment process at stores.

[1018] The processing flow will be explained below.

[1019] Step 1:

[1020] The user issues a voice command such as "Please speak to me." The device captures this voice command and stores it as voice data.

[1021] Step 2:

[1022] The device sends the captured audio data to the server, and the server receives the audio data.

[1023] Step 3:

[1024] The server passes the voice data to the speech recognition module, converts it into text, and verifies that the text is "Please speak."

[1025] Step 4:

[1026] The server retrieves the user's account information and transaction amount from a database, then generates a text response message saying, "That's XX yen. Say OK."

[1027] Step 5:

[1028] The server generates a confirmation message and sends it to the terminal. The terminal receives the confirmation message, converts it into speech using a speech synthesis engine, and notifies the user.

[1029] Step 6:

[1030] The user responds by saying "OK." The device recaptures the user's voice data and saves it as new voice data.

[1031] Step 7:

[1032] The device sends newly captured audio data to the server, which receives it.

[1033] Step 8:

[1034] The server passes the received voice data to the voice recognition module and converts it into text format as "OK." If "OK" is confirmed, the payment process begins.

[1035] Step 9:

[1036] The server debits the user's account for the specified amount and verifies that the transaction was successful.

[1037] Step 10:

[1038] The server generates a message saying "Payment has been completed" and sends it to the terminal. The terminal converts the received message into voice and notifies the user that "Payment has been completed."

[1039] Example 1

[1040] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1041] Traditional payment systems require the use of QR codes or manual input of information, making them difficult to use, especially for the visually impaired and those with limited digital literacy. There is also a lack of voice-based systems to make payment procedures quick and efficient.

[1042] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1043] In this invention, the server includes means for capturing a user's voice command, means for transmitting the captured voice command over a network, speech recognition means for converting the voice data into text format, natural language processing means for analyzing the text data, means for confirming the user's intention and acquiring account information and the payment amount, means for notifying the user of the payment amount and requesting an "OK" confirmation response, means for capturing the user's confirmation response and transmitting it again to the server, means for confirming the converted text data and starting the payment process, and means for generating a payment completion message and notifying the user. This enables the user to complete payment quickly and efficiently using only voice, without using a QR code.

[1044] The "means for capturing user voice commands" is hardware or software for obtaining voice commands uttered by a user.

[1045] The "means for transmitting the captured voice command over a network" refers to a means for transmitting the captured voice data to a server over a communication network.

[1046] The "voice recognition means for converting voice data into text format" is a technology that can convert acquired voice data into text information.

[1047] "Natural language processing means for analyzing text data" is a technology for analyzing the meaning of converted text data and understanding the user's intentions.

[1048] "Means for confirming the user's intent and obtaining account information and payment amount" refers to means for determining the user's intent from the analyzed text and obtaining the necessary account information and payment amount from a database or the like.

[1049] The "means for notifying the user of the payment amount and requesting a confirmation response of "OK"" is a means for transmitting the acquired payment amount to the user and requesting a response of "OK" as confirmation to proceed with the payment.

[1050] The "means for capturing the user's confirmation response and sending it again to the server" refers to a means for capturing the user's voice response again and sending that data to the server.

[1051] The "means for verifying the converted text data and initiating the payment process" is a means for analyzing the user's response and initiating the payment procedure if appropriate verification is obtained.

[1052] The "means for generating a payment completion message and notifying the user" is a means for generating a message indicating that the payment procedure has been completed normally and conveying the contents of the message to the user.

[1053] This invention is a system that utilizes voice recognition and natural language processing technology to perform smooth and efficient payments between users and the system. This system is realized by coordinating user voice instructions, server processing, and terminal operation.

[1054] System configuration

[1055] Hardware and Software

[1056] 1. Terminal: A device that allows users to issue voice commands. The terminal is equipped with a high-sensitivity microphone that captures the user's voice. The terminal also has the function of transmitting voice data to a server via the Internet.

[1057] 2. Server: A central processing unit that receives voice data and performs speech recognition and natural language processing. Specifically, it uses the following modules:

[1058] Speech recognition module: Converts voice data into text using the Google Cloud Speech-to-Text API or similar.

[1059] Natural language processing module: Analyzes text data using IBM Watson NLP and other technologies to understand user intent.

[1060] Database management system: Manages user account information and payment amounts using a database such as MySQL.

[1061] 3. Speech synthesis engine: This technology uses Amazon Polly and other technologies to notify users of confirmation messages and payment completion messages sent from the server via voice.

[1062] Processing flow

[1063] The user issues a voice command to the device, saying, "Please use voice." The device captures this voice and sends the generated voice data to the server.

[1064] The server converts the received voice data into text using the Google Cloud Speech-to-Text API, which is then analyzed by IBM Watson NLP to understand the user's intent.

[1065] The server retrieves the user's account information and payment amount from the database, generates a confirmation message saying, "It's XX yen. Please say OK." This message is sent to the terminal.

[1066] The device uses the Amazon Polly speech synthesis engine to audibly notify the user of a confirmation message. If the user responds "OK," the device captures the audio again and sends the new audio data to the server.

[1067] The server performs voice recognition again, confirms the user's "OK", and then starts the payment process. The specified amount is deducted from the user's account and the payment is completed.

[1068] The server generates a message saying "Payment completed" and sends it to the terminal, which converts this message into speech using the Amazon Polly speech synthesis engine and notifies the user.

[1069] Specific examples

[1070] scenario

[1071] Take the example of a user buying coffee at a cafe.

[1072] User

[1073] Give the voice command "Voice please."

[1074] Terminal

[1075] The user's voice is captured and sent to the server.

[1076] server

[1077] The Google Cloud Speech-to-Text API is used to convert the voice data into text and recognize the phrase "Please speak." Next, IBM Watson NLP is used to analyze the text and retrieve the user account information and payment amount (e.g., 500 yen) from the database. A confirmation message saying "500 yen, right? Say OK" is sent to the terminal.

[1078] Terminal

[1079] The user is notified by voice, "It's 500 yen. Please say OK."

[1080] User

[1081] Respond with a voice saying "OK."

[1082] Terminal

[1083] Capture the user's "OK" and send it to the server.

[1084] server

[1085] After confirming "OK" with the Google Cloud Speech-to-Text API, the payment process is initiated and 500 yen is deducted from the user's account. The completion of the payment is confirmed and a message stating "Payment completed" is sent to the terminal.

[1086] Terminal

[1087] The user is notified by voice that "payment has been completed."

[1088] The system uses voice recognition and natural language processing to provide fast and efficient payments, and is particularly convenient for users with visual impairments and those who have difficulty operating digital devices.

[1089] Prompt Sentence Examples

[1090] "Please explain a system that allows users to use voice commands to make payments without using a QR code. The system uses voice recognition and natural language processing, and includes the user's voice instructions, processing by the server, and terminal operation. Please provide a specific example and explain the detailed steps."

[1091] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1092] Step 1:

[1093] The user issues a voice command at a store, saying, "Please use voice." The device captures this voice command with a high-sensitivity microphone. The input is the user's voice, and the output is audio data in WAV format.

[1094] Step 2:

[1095] The terminal sends the generated WAV format audio data to a server via the Internet. The input is WAV format audio data, and the output is audio data sent via the network.

[1096] Step 3:

[1097] The server passes the received voice data to the Google Cloud Speech-to-Text API. The input is WAV format voice data, which is converted into text format. The output is the text data "Please speak by voice."

[1098] Step 4:

[1099] The server passes the text data to the IBM Watson NLP module, which analyzes the user's intent. Specifically, it understands from the text data "Voice please" that the user's intent is to start payment. The input is the text data "Voice please," and the output is the analysis result that the user's intent is to start payment.

[1100] Step 5:

[1101] The server retrieves the user's account information and payment amount from the MySQL database. Specifically, it uses an API to retrieve the account ID and balance information, and extracts the payment amount. The input is the user's account ID, and the output is the account information and payment amount (for example, 500 yen).

[1102] Step 6:

[1103] The server generates a confirmation message saying "It's 500 yen. Please say OK" based on the acquired account information and payment amount. The input is the user's account information and payment amount, and the output is the text data of the confirmation message.

[1104] Step 7:

[1105] The server generates a confirmation message and sends it to the terminal in JSON format. The input is the text data of the confirmation message, and the output is the JSON format data sent over the network.

[1106] Step 8:

[1107] The device uses the Amazon Polly speech synthesis engine to notify the user by voice of the received confirmation message, "It's 500 yen. Please say OK." The input is the text data of the confirmation message, and the output is a voice notification.

[1108] Step 9:

[1109] The user responds verbally with "OK." Once again, the device captures this audio with a high-sensitivity microphone. The input is the user's voice, and the output is audio data in a new WAV format.

[1110] Step 10:

[1111] The device then sends the new WAV-formatted audio data to the server again via the Internet. The input is the new WAV-formatted audio data, and the output is the audio data sent over the network.

[1112] Step 11:

[1113] The server again calls the Google Cloud Speech-to-Text API to convert the received audio data into text format. The input is the new WAV audio data, and the output is the text data "OK".

[1114] Step 12:

[1115] The server verifies that the text data is "OK." The input is the text data "OK," and the output is the verification result.

[1116] Step 13:

[1117] The server initiates the payment process and debits the specified amount (500 yen) from the user's account via the API. The input is the user's account information and the payment amount, and the output is confirmation that the debit has been completed.

[1118] Step 14:

[1119] The server confirms that the payment has been completed and generates a message saying "Payment completed." The input is the confirmation of the withdrawal completion, and the output is the text data of the completion message.

[1120] Step 15:

[1121] The server sends the generated completion message to the terminal in JSON format. The input is the text data of the completion message, and the output is the JSON format data sent over the network.

[1122] Step 16:

[1123] The terminal uses the Amazon Polly speech synthesis engine to notify the user by voice of the message "Payment completed." The input is the text data of the completion message, and the output is the voice notification.

[1124] Through the above processing steps, payment can be completed quickly and efficiently using only voice, without using a QR code.

[1125] (Application example 1)

[1126] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1127] While the convenience of electronic payment services has improved in recent years, many systems require visual input, such as QR codes or touch operations. This poses a challenge for users with visual impairments and those unfamiliar with operating digital devices. Furthermore, existing voice payment systems lack sufficient speech recognition accuracy and natural language processing effectiveness, making it difficult to achieve smooth and prompt payments.

[1128] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1129] In this invention, the server includes a voice recognition unit, a natural language processing unit, and a terminal for capturing the user's voice commands. This allows the user to make payments quickly and accurately using only their voice, without the need for visual input. Furthermore, the user experience can be improved by using voice synthesis to deliver confirmation messages and payment completion notifications.

[1130] "Speech recognition means" is a technology for capturing a user's speech and converting it into text data.

[1131] "Natural language processing means" is a technology for analyzing text data and understanding the user's intent.

[1132] A "terminal for capturing user's voice commands" is a device for collecting user's voice and transmitting it to a server.

[1133] The "payment processing server" is a central control unit that processes payments based on captured voice commands.

[1134] "Means for confirming user account information and payment amount" refers to technology that allows the server to obtain and confirm the user's account information and payment amount.

[1135] "Means for notifying the user of the payment amount and requesting a confirmation response of 'OK'" refers to a technique for notifying the user of the payment amount by voice and requesting confirmation by voice.

[1136] The "means for receiving a user's confirmation response and completing the payment" refers to a technique for obtaining a user's confirmation response and completing the actual payment process.

[1137] "Means for notifying the user using voice synthesis" is a technology that converts a message from the server into voice and notifies the user.

[1138] "Means for notifying the user by voice after payment is completed" refers to a technique for notifying the user by voice that payment has been completed.

[1139] The present invention relates to a system that uses voice recognition and natural language processing to analyze a user's voice commands and execute payments by voice without using a QR code. This system is realized through cooperation between the user, terminals, and a server.

[1140] The system is configured as follows:

[1141] 1. A device that captures the user's voice command: When a user says "pay with voice," this device captures the voice and sends it over the internet to a server. For example, this could be a smartphone or smart glasses.

[1142] 2. Speech recognition method: The server uses speech recognition technology such as the Google Speech-to-Text API to convert the received speech into text data.

[1143] 3. Natural language processing means: Analyze text data using Google NLP API or similar to confirm the user's intent.

[1144] 4. Server that executes the payment process: The server obtains the user's account information and payment amount, and generates a confirmation message in the format "It's XX yen. Please say OK."

[1145] 5. Notifying the user using voice synthesis: The confirmation message from the server is converted into voice using the Google Text-to-Speech API and notified to the user.

[1146] 6. Means for receiving user confirmation response: When the user responds "OK", the terminal again captures this voice and sends it to the server.

[1147] 7. Completion of payment process: The server again uses voice recognition to confirm the user's response, then processes the payment and deducts the specified amount from the user's account.

[1148] 8. A means of notifying the user by voice after payment is completed: When payment is completed, the server generates a completion notification stating "Payment has been completed" and notifies the user by voice via the terminal.

[1149] Specific examples

[1150] Below is an example of a user purchasing a coffee at a cafe.

[1151] A user at a cafe says, "Pay by voice." The device captures this speech and sends it to the server. The server uses speech recognition to transcribe "Pay by voice" and uses natural language processing to understand the user's intent. The server then obtains the user's account information and payment amount (e.g., 500 yen), generates a confirmation message saying, "500 yen, right? Say OK," and converts this into voice to notify the user.

[1152] If the user responds "OK," the device captures the audio again and sends it to the server. After the server confirms this "OK," it starts the payment process, deducting 500 yen from the user's account. Finally, the server generates a message saying "Payment completed," converts it into audio, and notifies the user via the device.

[1153] This system allows even visually impaired people or users who have difficulty operating digital devices to make payments quickly and efficiently using only their voice.

[1154] Example prompt sentence:

[1155] "Voice command to pay"

[1156] "It's 500 yen. Just say OK."

[1157] "Payment completed"

[1158] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1159] Step 1:

[1160] The user says "Pay with voice." This voice command is captured by the terminal.

[1161] Input: The user's voice command.

[1162] Output: The captured audio data.

[1163] Step 2:

[1164] The device transmits the captured audio data to the server.

[1165] Input: The captured audio data.

[1166] Output: Sending audio data to the server.

[1167] Step 3:

[1168] The server converts the received voice data into text data using a voice recognition tool (Google Speech-to-Text API).

[1169] Input: The audio data sent to the server.

[1170] Output: Text data.

[1171] Step 4:

[1172] The server analyzes the text data using natural language processing tools (Google NLP API) and confirms the user's intent.

[1173] Input: Text data.

[1174] Output: Data that represents the user's intent.

[1175] Step 5:

[1176] The server obtains the user's account information and payment amount and generates a confirmation message saying, "It's 500 yen. Please say OK."

[1177] Input: Data representing the user's intent, and a database of account information on the server.

[1178] Output: Confirmation message (text format).

[1179] Step 6:

[1180] The server converts the confirmation message into voice data using a voice synthesis method (Google Text-to-Speech API) and sends it to the terminal.

[1181] Input: Confirmation message (in text format).

[1182] Output: Confirmation message (audio data).

[1183] Step 7:

[1184] The terminal notifies the user of the audio data.

[1185] Input: Confirmation message (audio data).

[1186] Output: Audio notification to the user.

[1187] Step 8:

[1188] The user says "OK." This speech is again captured by the device and sent to the server.

[1189] Input: User's spoken acknowledgment.

[1190] Output: The captured audio data of the acknowledgment.

[1191] Step 9:

[1192] The server converts the received voice data of the confirmation response into text data using a voice recognition method (Google Speech-to-Text API).

[1193] Input: Acknowledgment audio data.

[1194] Output: Acknowledgement text data.

[1195] Step 10:

[1196] After the server confirms "OK," it starts the payment process and deducts the specified amount from the user's account.

[1197] Input: Text data of confirmation response, and account information database in the server.

[1198] Output: Payment processing completed.

[1199] Step 11:

[1200] The server generates a payment completion message, converts it into voice data using a voice synthesis method (Google Text-to-Speech API), and sends it to the terminal.

[1201] Input: Payment completion message (text format).

[1202] Output: Payment completion message (audio data).

[1203] Step 12:

[1204] The terminal notifies the user by voice of a payment completion message.

[1205] Input: Payment completion message (audio data).

[1206] Output: Notification to the user that payment has been completed.

[1207] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1208] This invention relates to a system that combines voice recognition, natural language processing, and emotion recognition to analyze conversations between users and store clerks and execute voice payments without using QR codes. By linking the user's voice instructions, server processing, terminal operation, and emotion recognition, this system provides a smooth, efficient payment experience that takes emotions into consideration for the user.

[1209] Program Overview

[1210] The system operates as follows:

[1211] User voice instructions

[1212] The user issues a voice command such as "Please use voice." The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet.

[1213] Server-based voice analysis

[1214] The server passes the received voice data to a speech recognition module, converts it into text, and uses a natural language processing module to analyze the text data and confirm the user's intention.The server then uses an emotion engine to identify and analyze the user's emotion from the voice data.

[1215] Emotion-Based Processing

[1216] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is anxious, it will choose calming words. On the other hand, if the user is in a calm state, it will respond normally.

[1217] Generate and notify confirmation messages

[1218] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[1219] User Acknowledgment

[1220] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[1221] Server response confirmation

[1222] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[1223] Notification of payment completion

[1224] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[1225] Specific examples

[1226] scenario

[1227] Consider the example of a user purchasing a coffee at a cafe.

[1228] User:

[1229] "Audio please."

[1230] Device:

[1231] The user's voice is captured and sent to the server.

[1232] server:

[1233] The received voice data is converted into text and recognized as "Voice please." Next, the emotion engine is used to analyze the user's tone of voice and determine that the user is a little nervous. The server retrieves the user's account information and payment amount (for example, 500 yen) from the database. It generates a message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon," and sends it to the terminal.

[1234] Device:

[1235] The user is notified by voice, "It's 500 yen. Say OK. Don't worry, it will be over soon."

[1236] User:

[1237] Respond with "OK."

[1238] Device:

[1239] Capture the user's "OK" and send it to the server.

[1240] server:

[1241] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[1242] Device:

[1243] The user is notified by voice that "Payment has been completed."

[1244] This process allows users to experience an emotionally sensitive voice payment experience, particularly reducing confusion and tension. It also allows store clerks to complete payments quickly and efficiently. The system is also easy to use for people with visual impairments and those who have difficulty operating digital devices, simplifying the payment process in stores.

[1245] The processing flow will be explained below.

[1246] Step 1:

[1247] The user issues a voice command such as "Please speak to me." The device captures this voice command through the microphone and stores it as voice data.

[1248] Step 2:

[1249] The device sends the captured audio data to the server, and the server receives the audio data.

[1250] Step 3:

[1251] The server passes the received voice data to the voice recognition module, which converts the voice data into text. The server then analyzes the converted text, "Please speak to me."

[1252] Step 4:

[1253] The server retrieves the user's account information and payment amount from the database, and then uses an emotion engine to identify the user's emotion from the voice data.

[1254] Step 5:

[1255] The server analyzes the emotion engine results and determines that the user is in a tense state. Based on this result, it generates a confirmation message with additional words to reassure the user.

[1256] Step 6:

[1257] The server generates a confirmation message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon." and sends it to the device. The device passes the received confirmation message to the speech synthesis engine, converts it into speech, and notifies the user.

[1258] Step 7:

[1259] The user responds by saying "OK." The device recaptures this audio and saves it as new audio data.

[1260] Step 8:

[1261] The device sends the newly captured audio data to the server, and the server receives the audio data.

[1262] Step 9:

[1263] The server passes the received voice data to the voice recognition module again and converts it into text format. The server confirms "OK".

[1264] Step 10:

[1265] If the server receives the "OK" confirmation, it starts the payment process: the server debits the specified amount from the user's account.

[1266] Step 11:

[1267] The server confirms that the payment process was successful and generates a "Payment completed" message and sends it to the terminal.

[1268] Step 12:

[1269] The terminal passes the received payment completion message to a speech synthesis engine, which converts it into speech and notifies the user, "Payment has been completed."

[1270] Example 2

[1271] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1272] Conventional payment systems require users to use a visual or physical interface to make payments, which is inconvenient for the visually impaired and those who have difficulty operating digital devices. Furthermore, they are unable to provide services that respond to the user's emotional state, leaving room for improvement in the user experience. Furthermore, voice payment methods that do not use physical media such as QR codes are immature, preventing smooth payments.

[1273] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice recognition means, a natural language processing means, a means for capturing a user's voice instruction, a processing device that executes payment based on the captured voice instruction, a means for confirming the user's account information and payment amount, a means for notifying the user of the payment amount and requesting a confirmation response of "OK," a means for receiving the user's confirmation response and completing the payment, and a means for identifying the user's emotional state using an emotion recognition means and generating an appropriate response. This makes it easy for visually impaired people and users who have difficulty operating digital devices to use, and enables smooth voice payment that takes the user's emotions into consideration.

[1274] "Speech recognition means" is a technology that converts voice data into text data.

[1275] "Natural language processing means" is a technology that analyzes text data and understands context and intent.

[1276] A "means for capturing user voice instructions" is any device or technology that collects a user's voice and converts it into a digital form.

[1277] A "processing device that executes a payment procedure based on captured voice instructions" is an electronic device or system that receives voice instruction data and initiates a payment process.

[1278] "Means for confirming user account information and payment amount" refers to technology for obtaining and confirming user account information and payment amount from a database or the like.

[1279] "Means for notifying the user of the payment amount and requesting a confirmation response of 'OK'" refers to a technique or device that notifies the user of the payment amount and requests a verbal response of 'OK'.

[1280] The "means for receiving a user's confirmation response and completing the payment" refers to a technique or device that receives a user's confirmation response of "OK" and completes the payment process.

[1281] "Emotion recognition means" is a technology for identifying emotions from the user's voice and analyzing their state.

[1282] The "means for generating an appropriate response" refers to a technique or device that takes into account the emotional state of the user and generates an optimal response message.

[1283] MODE FOR CARRYING OUT THE INVENTION

[1284] This invention relates to a system that combines voice recognition, natural language processing, and emotion recognition to analyze conversations between users and store clerks and execute voice payments without using QR codes. By linking the user's voice instructions, server processing, terminal operation, and emotion recognition, this system provides a smooth, efficient payment experience that takes emotions into consideration for the user.

[1285] Hardware and Software Usage

[1286] User voice instructions

[1287] A user issues a voice command such as "Please speak to me" at a store. The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet. A high-performance microphone is used as the hardware for capturing the voice.

[1288] Server-based voice analysis

[1289] The server passes the received voice data to a speech recognition module such as Google Cloud Speech-to-Text and converts it into text. It also uses a natural language processing module such as GPT-3 to analyze the text data and confirm the user's intent. The server then uses an emotion engine such as IBM Watson Tone Analyzer to identify and analyze the user's emotions from the voice data.

[1290] Emotion-Based Processing

[1291] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is anxious, it will choose calming words. On the other hand, if the user is in a calm state, it will respond normally.

[1292] Generate and notify confirmation messages

[1293] The server generates a confirmation message saying, "It's XX yen. Please say OK," and sends it to the device. The device receives this confirmation message and uses Microsoft Azure Text-to-Speech as a speech synthesis engine to notify the user by voice.

[1294] User Acknowledgment

[1295] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[1296] Server response confirmation

[1297] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[1298] Notification of payment completion

[1299] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[1300] Specific examples

[1301] scenario

[1302] Consider the example of a user purchasing a coffee at a cafe.

[1303] User:

[1304] "Audio please."

[1305] Device:

[1306] The user's voice is captured and sent to the server.

[1307] server:

[1308] The received voice data is converted into text and recognized as "Voice please." Next, the emotion engine is used to analyze the user's tone of voice and determine that the user is a little nervous. The server retrieves the user's account information and payment amount (for example, 500 yen) from the database. It generates a message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon," and sends it to the terminal.

[1309] Device:

[1310] The user is notified by voice, "It's 500 yen. Say OK. Don't worry, it will be over soon."

[1311] User:

[1312] Respond with "OK."

[1313] Device:

[1314] Capture the user's "OK" and send it to the server.

[1315] server:

[1316] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[1317] Device:

[1318] The user is notified by voice that "Payment has been completed."

[1319] This process allows users to experience an emotionally sensitive voice payment experience, particularly reducing confusion and tension. It also allows store clerks to complete payments quickly and efficiently. The system is also easy to use for people with visual impairments and those who have difficulty operating digital devices, simplifying the payment process in stores.

[1320] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1321] Step 1:

[1322] The user issues a voice command such as "Voice please." The device uses the user's voice instruction as input and captures voice data with a microphone. The device converts the captured voice data into a digital format and sends it to a server via the Internet. The output is the voice data sent to the server.

[1323] Step 2:

[1324] The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) and converts it into text data. In this step, the input voice data is converted into text. The output is in text format, "Please speak by voice."

[1325] Step 3:

[1326] The server passes the text data to a natural language processing module (e.g., GPT-3) to analyze the user's intent. Here, the text data is used as input to analyze and understand the user's intent and request. This process confirms that the user wishes to make a voice payment. The output is the analysis result regarding the user's intent.

[1327] Step 4:

[1328] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to identify and analyze the user's emotion from the voice data. The input is the voice data, and the voice tone and emotional state are analyzed. This process identifies the user's emotional state (e.g., nervousness). The output is the emotion recognition result.

[1329] Step 5:

[1330] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. The input is the emotion recognition results and the analysis results of the user's intention. For example, if the user is nervous, a calming message is generated. The output is a response message.

[1331] Step 6:

[1332] The server retrieves the payment amount (e.g., 500 yen) from the database and creates a confirmation message saying, "It's 500 yen. Please say OK." It also adds an additional message that takes emotion into consideration. The input is the user's account information and the payment amount. The output is a confirmation message.

[1333] Step 7:

[1334] The server sends the generated confirmation message to the terminal. The input is the confirmation message. The terminal receives this message. The output is the terminal that received the confirmation message.

[1335] Step 8:

[1336] The terminal converts the received confirmation message into speech using a speech synthesis engine (e.g., Microsoft Azure Text-to-Speech). The input is the text data of the confirmation message. The terminal notifies the user by voice, "That's 500 yen. Say OK. Don't worry, it will be done soon." The output is a voice notification.

[1337] Step 9:

[1338] The user responds with "OK" by voice. The terminal captures the voice data again using the user's confirmation response as input. The terminal converts this data into a digital format and sends it back to the server. The output is the voice data of the confirmation response sent to the server.

[1339] Step 10:

[1340] The server passes the sent confirmation response voice data to the voice recognition module again and converts it into text data. The input is the confirmation response voice data. The converted text data "OK" is output.

[1341] Step 11:

[1342] The server confirms the textual "OK" and starts the payment process. The input is "OK" as text data and the user's account information. The server deducts the specified amount from the user's account and makes the payment. The output is payment completion data.

[1343] Step 12:

[1344] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." The input is the payment completion data. The output is the payment completion message.

[1345] Step 13:

[1346] The server sends the generated payment completion message to the terminal. The input is the payment completion message. The terminal receives this message. The output is the terminal that received the payment completion message.

[1347] Step 14:

[1348] The terminal converts the received payment completion message into speech using a speech synthesis engine. The input is the text data of the payment completion message. The terminal notifies the user by voice, "Payment has been completed." The output is a voice notification.

[1349] (Application example 2)

[1350] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1351] Conventional voice payment systems rely solely on voice recognition and natural language processing, and have the problem of being unable to provide appropriate responses that take into account the user's emotional state. Furthermore, the payment process often does not proceed smoothly when the user feels stressed or nervous. Furthermore, certain user groups, such as the visually impaired and elderly, have the problem of difficulty operating the system. There is a need to solve these issues and provide users with a comfortable and smooth payment experience.

[1352] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means, a natural language processing means, and an emotion recognition means. This makes it possible to accurately recognize the user's voice command, generate an appropriate response that takes their emotional state into consideration, and execute a smooth payment process that takes the user's emotions into consideration.

[1353] "Speech recognition means" is a general term for a device or program that captures a user's voice commands and converts them into digital text data.

[1354] "Natural language processing means" is a general term for technology or programs that analyze the user's intentions and meaning from text data and determine appropriate responses and actions.

[1355] "Terminal" is a general term for an electronic device that captures a user's voice and transmits that data to a server.

[1356] "Emotion recognition means" is a general term for technology or programs for identifying and analyzing a user's emotional state from voice data or other input.

[1357] "Server" is a general term for a computer system that performs speech recognition, natural language processing, and emotion recognition, and manages and executes the payment process.

[1358] "Account Information" means, collectively, your personal identification information and payment-related data.

[1359] "Payment amount" refers to the amount to be paid by the user.

[1360] "Response generation means" is a general term for a technology or program for generating an appropriate response message based on the analysis results.

[1361] "Acknowledgement" refers to the voice instructions given by the user to proceed with the payment process.

[1362] This invention is a system that combines speech recognition, natural language processing, and emotion recognition, allowing users to make payments quickly and smoothly using voice commands. The system includes a speech recognition unit, a natural language processing unit, an emotion recognition unit, a server, and a terminal that captures the user's voice.

[1363] The specific operation of the system is configured as follows.

[1364] Hardware and Software Configuration

[1365] Device: A smartphone to capture the user's voice.

[1366] Server: A server for processing voice data. A computer system required for speech recognition, natural language processing, and emotion recognition.

[1367] Speech Recognition Tool: Software that converts speech to text using the SpeechRecognition library.

[1368] Natural language processing tools: Software that uses TextBlob to analyze spoken text and understand user intent.

[1369] Emotion Recognition: Software that uses EmotionClassifier (emopy library) to identify user emotions.

[1370] Program processing overview

[1371] 1. User issues a voice command

[1372] The user issues a voice command to the device, saying, "Please use voice." The device captures this voice and sends the voice data to the server.

[1373] 2. Audio analysis by the server

[1374] The server converts the received voice data into text format using a voice recognition unit, then analyzes the text data using a natural language processing unit to confirm the user's intention, and further identifies the user's emotion from the voice data using an emotion recognition unit.

[1375] 3. Emotion-based response generation

[1376] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is nervous, it will choose calming words, and if the user is in a relaxed state, it will respond normally.

[1377] 4. Generating and notifying confirmation messages

[1378] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[1379] 5. User Acknowledgment

[1380] The user responds by saying "OK." This voice is also captured by the device, and new voice data is created and sent to the server.

[1381] 6. The server confirms the response and starts the payment process

[1382] The server again uses the speech recognition means to convert the user's response into text data and confirms "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[1383] 7. Notification of payment completion

[1384] The server confirms that the payment has been completed, generates a completion message, and sends it to the terminal. The terminal then notifies the user of the received message by voice.

[1385] Specific examples

[1386] As a concrete example, let's consider the process when a user purchases coffee at a cafe. In this case, the process is as follows:

[1387] 1. User: "Voice please."

[1388] 2. Device: Captures the user's voice and sends it to the server.

[1389] 3. Server: Converts the received voice data into text and recognizes "Voice please." Analyzes the user's tone of voice using an emotion recognition engine and determines that the user is a little nervous. Retrieves the user's account information and payment amount (e.g., 500 yen) from the database. Generates a message saying, "500 yen, right? Say OK. Don't worry, it will be done soon," and sends it to the device.

[1390] 4. Terminal: A voice message will say, "It's 500 yen. Say OK. Don't worry, it will be done soon."

[1391] 5. User: Responds "OK."

[1392] 6. Terminal: Captures the user's "OK" and sends it to the server.

[1393] 7. Server: After confirming "OK", it deducts 500 yen and starts the payment process. It confirms that the payment has been completed, generates a completion message, and sends it to the terminal.

[1394] 8. Terminal: The user is notified by voice that "Payment has been completed."

[1395] This allows users to experience a smooth payment experience that takes their emotions into consideration.

[1396] Example prompts for generative AI models

[1397] "Generate a calming response message for users who are a little nervous when paying by voice."

[1398] Such a system will enable voice payments that are easy to use, especially for those with visual impairments or who have difficulty operating digital devices.

[1399] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1400] Step 1:

[1401] The user issues a voice command to the device, saying, "Please speak to me." The device captures this voice and generates voice data, which is then sent directly to the server.

[1402] Input: User's voice command

[1403] Output: Audio data (sent to server)

[1404] How it works: When a user speaks into the smartphone application, the microphone captures the voice, and the voice data is transmitted to the server in real time.

[1405] Step 2:

[1406] The server converts the received voice data into text using speech recognition means, which is then analyzed by natural language processing means to ascertain the user's intent.

[1407] Input: Audio data

[1408] Output: Text data

[1409] How it works: The SpeechRecognition library on the server converts the audio data into text, then uses the TextBlob library to analyze the text and extract the user's intent.

[1410] Step 3:

[1411] The server analyzes the user's emotional state using the text data and the voice data with an emotion recognition means, and the analyzed emotional information is further processed.

[1412] Input: Text and audio data

[1413] Output: Emotion recognition result

[1414] Specific operation: The emopy library is used to analyze the user's emotional state from audio and text data, and the results are obtained. For example, states such as "tense" or "calm" are identified.

[1415] Step 4:

[1416] The server generates an appropriate response message based on the emotion recognition results, depending on the user's emotional state. For example, if the user is nervous, it generates a message containing calming words.

[1417] Input: Emotion recognition results, text data

[1418] Output: Response message

[1419] Specific behavior: The server uses the text template to generate a response message appropriate to the emotional state, such as "500 yen. Say OK. Don't worry, it will be over soon."

[1420] Step 5:

[1421] The server sends the generated response message to the terminal, and the terminal uses a speech synthesis engine to notify the user by voice.

[1422] Input: Response message

[1423] Output: Audio notification

[1424] Specific operation: When a device receives a text message sent from the server, it uses a text-to-speech engine to convert the message into audio and convey it to the user.

[1425] Step 6:

[1426] The user responds by saying "OK." This voice is also captured by the device, and new voice data is generated and sent to the server again.

[1427] Input: User's voice response

[1428] Output: Audio data (sent to server)

[1429] What happens: The user says "OK" to confirm. The device's microphone captures this sound and sends it to the server.

[1430] Step 7:

[1431] The server again uses the speech recognition means to convert the user's response into text data and confirms "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[1432] Input: Audio data

[1433] Output: Confirmation "OK" and start of payment process

[1434] What happens: The SpeechRecognition library on the server converts the voice data into text and uses natural language processing to confirm "OK." Once this confirmation is received, the payment process is carried out.

[1435] Step 8:

[1436] The server confirms that the payment has been completed, generates a message stating "Payment has been completed," and sends it to the terminal.

[1437] Input: Confirm payment completion

[1438] Output: Completion message

[1439] Specific operation: Once the payment process is complete, the server generates a confirmation message and sends it to the terminal, which is then notified to the user by the speech synthesis engine.

[1440] This series of steps provides users with an emotionally sensitive and smooth payment experience.

[1441] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1442] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1443] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1444] [Fourth embodiment]

[1445] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1446] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1447] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1448] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1449] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1450] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1451] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1452] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1453] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1454] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1455] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1456] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1457] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1458] This invention relates to a system that uses voice recognition and natural language processing to analyze the conversation between the user and the store clerk and executes payments by voice without using a QR code. This system provides smooth and efficient payments by linking the user's voice instructions, server processing, and terminal operation.

[1459] Program Overview

[1460] The system operates as follows:

[1461] User voice instructions

[1462] The user issues a voice command such as "Please use voice." The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet.

[1463] Server-based voice analysis

[1464] The server passes the received voice data to a speech recognition module, converts it into text, and uses a natural language processing module to analyze the text data and confirm the user's intent.The server then retrieves the user's account information and payment amount from the database.

[1465] Generate and notify confirmation messages

[1466] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[1467] User Acknowledgment

[1468] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[1469] Server response confirmation

[1470] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[1471] Notification of payment completion

[1472] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[1473] Specific examples

[1474] scenario

[1475] Consider the example of a user purchasing a coffee at a cafe.

[1476] User:

[1477] "Audio please."

[1478] Device:

[1479] The user's voice is captured and sent to the server.

[1480] server:

[1481] The received voice data is converted into text format and recognized as "Voice please." Next, the user account information and payment amount (for example, 500 yen) are obtained from the database. A message is generated saying "500 yen, right? Say OK," and sent to the terminal.

[1482] Device:

[1483] The user is notified by voice, "It's 500 yen. Please say OK."

[1484] User:

[1485] Respond with "OK."

[1486] Device:

[1487] Capture the user's "OK" and send it to the server.

[1488] server:

[1489] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[1490] Device:

[1491] The user is notified by voice that "Payment has been completed."

[1492] This process allows payments to be completed quickly and efficiently using only voice, without the need for a QR code. This system is particularly easy to use for people with visual impairments or those who have difficulty operating digital devices, and simplifies the payment process at stores.

[1493] The processing flow will be explained below.

[1494] Step 1:

[1495] The user issues a voice command such as "Please speak to me." The device captures this voice command and stores it as voice data.

[1496] Step 2:

[1497] The device sends the captured audio data to the server, and the server receives the audio data.

[1498] Step 3:

[1499] The server passes the voice data to the speech recognition module, converts it into text, and verifies that the text is "Please speak."

[1500] Step 4:

[1501] The server retrieves the user's account information and transaction amount from a database, then generates a text response message saying, "That's XX yen. Say OK."

[1502] Step 5:

[1503] The server generates a confirmation message and sends it to the terminal. The terminal receives the confirmation message, converts it into speech using a speech synthesis engine, and notifies the user.

[1504] Step 6:

[1505] The user responds by saying "OK." The device recaptures the user's voice data and saves it as new voice data.

[1506] Step 7:

[1507] The device sends newly captured audio data to the server, which receives it.

[1508] Step 8:

[1509] The server passes the received voice data to the voice recognition module and converts it into text format as "OK." If "OK" is confirmed, the payment process begins.

[1510] Step 9:

[1511] The server debits the user's account for the specified amount and verifies that the transaction was successful.

[1512] Step 10:

[1513] The server generates a message saying "Payment has been completed" and sends it to the terminal. The terminal converts the received message into voice and notifies the user that "Payment has been completed."

[1514] Example 1

[1515] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1516] Traditional payment systems require the use of QR codes or manual input of information, making them difficult to use, especially for the visually impaired and those with limited digital literacy. There is also a lack of voice-based systems to make payment procedures quick and efficient.

[1517] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1518] In this invention, the server includes means for capturing a user's voice command, means for transmitting the captured voice command over a network, speech recognition means for converting the voice data into text format, natural language processing means for analyzing the text data, means for confirming the user's intention and acquiring account information and the payment amount, means for notifying the user of the payment amount and requesting an "OK" confirmation response, means for capturing the user's confirmation response and transmitting it again to the server, means for confirming the converted text data and starting the payment process, and means for generating a payment completion message and notifying the user. This enables the user to complete payment quickly and efficiently using only voice, without using a QR code.

[1519] The "means for capturing user voice commands" is hardware or software for obtaining voice commands uttered by a user.

[1520] The "means for transmitting the captured voice command over a network" refers to a means for transmitting the captured voice data to a server over a communication network.

[1521] The "voice recognition means for converting voice data into text format" is a technology that can convert acquired voice data into text information.

[1522] "Natural language processing means for analyzing text data" is a technology for analyzing the meaning of converted text data and understanding the user's intentions.

[1523] "Means for confirming the user's intent and obtaining account information and payment amount" refers to means for determining the user's intent from the analyzed text and obtaining the necessary account information and payment amount from a database or the like.

[1524] The "means for notifying the user of the payment amount and requesting a confirmation response of "OK"" is a means for transmitting the acquired payment amount to the user and requesting a response of "OK" as confirmation to proceed with the payment.

[1525] The "means for capturing the user's confirmation response and sending it again to the server" refers to a means for capturing the user's voice response again and sending that data to the server.

[1526] The "means for verifying the converted text data and initiating the payment process" is a means for analyzing the user's response and initiating the payment procedure if appropriate verification is obtained.

[1527] The "means for generating a payment completion message and notifying the user" is a means for generating a message indicating that the payment procedure has been completed normally and conveying the contents of the message to the user.

[1528] This invention is a system that utilizes voice recognition and natural language processing technology to perform smooth and efficient payments between users and the system. This system is realized by coordinating user voice instructions, server processing, and terminal operation.

[1529] System configuration

[1530] Hardware and Software

[1531] 1. Terminal: A device that allows users to issue voice commands. The terminal is equipped with a high-sensitivity microphone that captures the user's voice. The terminal also has the function of transmitting voice data to a server via the Internet.

[1532] 2. Server: A central processing unit that receives voice data and performs speech recognition and natural language processing. Specifically, it uses the following modules:

[1533] Speech recognition module: Converts voice data into text using the Google Cloud Speech-to-Text API or similar.

[1534] Natural language processing module: Analyzes text data using IBM Watson NLP and other technologies to understand user intent.

[1535] Database management system: Manages user account information and payment amounts using a database such as MySQL.

[1536] 3. Speech synthesis engine: This technology uses Amazon Polly and other technologies to notify users of confirmation messages and payment completion messages sent from the server via voice.

[1537] Processing flow

[1538] The user issues a voice command to the device, saying, "Please use voice." The device captures this voice and sends the generated voice data to the server.

[1539] The server converts the received voice data into text using the Google Cloud Speech-to-Text API, which is then analyzed by IBM Watson NLP to understand the user's intent.

[1540] The server retrieves the user's account information and payment amount from the database, generates a confirmation message saying, "It's XX yen. Please say OK." This message is sent to the terminal.

[1541] The device uses the Amazon Polly speech synthesis engine to audibly notify the user of a confirmation message. If the user responds "OK," the device captures the audio again and sends the new audio data to the server.

[1542] The server performs voice recognition again, confirms the user's "OK", and then starts the payment process. The specified amount is deducted from the user's account and the payment is completed.

[1543] The server generates a message saying "Payment completed" and sends it to the terminal, which converts this message into speech using the Amazon Polly speech synthesis engine and notifies the user.

[1544] Specific examples

[1545] scenario

[1546] Take the example of a user buying coffee at a cafe.

[1547] User

[1548] Give the voice command "Voice please."

[1549] Terminal

[1550] The user's voice is captured and sent to the server.

[1551] server

[1552] The Google Cloud Speech-to-Text API is used to convert the voice data into text and recognize the phrase "Please speak." Next, IBM Watson NLP is used to analyze the text and retrieve the user account information and payment amount (e.g., 500 yen) from the database. A confirmation message saying "500 yen, right? Say OK" is sent to the terminal.

[1553] Terminal

[1554] The user is notified by voice, "It's 500 yen. Please say OK."

[1555] User

[1556] Respond with a voice saying "OK."

[1557] Terminal

[1558] Capture the user's "OK" and send it to the server.

[1559] server

[1560] After confirming "OK" with the Google Cloud Speech-to-Text API, the payment process is initiated and 500 yen is deducted from the user's account. The completion of the payment is confirmed and a message stating "Payment completed" is sent to the terminal.

[1561] Terminal

[1562] The user is notified by voice that "payment has been completed."

[1563] The system uses voice recognition and natural language processing to provide fast and efficient payments, and is particularly convenient for users with visual impairments and those who have difficulty operating digital devices.

[1564] Prompt Sentence Examples

[1565] "Please explain a system that allows users to use voice commands to make payments without using a QR code. The system uses voice recognition and natural language processing, and includes the user's voice instructions, processing by the server, and terminal operation. Please provide a specific example and explain the detailed steps."

[1566] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1567] Step 1:

[1568] The user issues a voice command at a store, saying, "Please use voice." The device captures this voice command with a high-sensitivity microphone. The input is the user's voice, and the output is audio data in WAV format.

[1569] Step 2:

[1570] The terminal sends the generated WAV format audio data to a server via the Internet. The input is WAV format audio data, and the output is audio data sent via the network.

[1571] Step 3:

[1572] The server passes the received voice data to the Google Cloud Speech-to-Text API. The input is WAV format voice data, which is converted into text format. The output is the text data "Please speak by voice."

[1573] Step 4:

[1574] The server passes the text data to the IBM Watson NLP module, which analyzes the user's intent. Specifically, it understands from the text data "Voice please" that the user's intent is to start payment. The input is the text data "Voice please," and the output is the analysis result that the user's intent is to start payment.

[1575] Step 5:

[1576] The server retrieves the user's account information and payment amount from the MySQL database. Specifically, it uses an API to retrieve the account ID and balance information, and extracts the payment amount. The input is the user's account ID, and the output is the account information and payment amount (for example, 500 yen).

[1577] Step 6:

[1578] The server generates a confirmation message saying "It's 500 yen. Please say OK" based on the acquired account information and payment amount. The input is the user's account information and payment amount, and the output is the text data of the confirmation message.

[1579] Step 7:

[1580] The server generates a confirmation message and sends it to the terminal in JSON format. The input is the text data of the confirmation message, and the output is the JSON format data sent over the network.

[1581] Step 8:

[1582] The device uses the Amazon Polly speech synthesis engine to notify the user by voice of the received confirmation message, "It's 500 yen. Please say OK." The input is the text data of the confirmation message, and the output is a voice notification.

[1583] Step 9:

[1584] The user responds verbally with "OK." Once again, the device captures this audio with a high-sensitivity microphone. The input is the user's voice, and the output is audio data in a new WAV format.

[1585] Step 10:

[1586] The device then sends the new WAV-formatted audio data to the server again via the Internet. The input is the new WAV-formatted audio data, and the output is the audio data sent over the network.

[1587] Step 11:

[1588] The server again calls the Google Cloud Speech-to-Text API to convert the received audio data into text format. The input is the new WAV audio data, and the output is the text data "OK".

[1589] Step 12:

[1590] The server verifies that the text data is "OK." The input is the text data "OK," and the output is the verification result.

[1591] Step 13:

[1592] The server initiates the payment process and debits the specified amount (500 yen) from the user's account via the API. The input is the user's account information and the payment amount, and the output is confirmation that the debit has been completed.

[1593] Step 14:

[1594] The server confirms that the payment has been completed and generates a message saying "Payment completed." The input is the confirmation of the withdrawal completion, and the output is the text data of the completion message.

[1595] Step 15:

[1596] The server sends the generated completion message to the terminal in JSON format. The input is the text data of the completion message, and the output is the JSON format data sent over the network.

[1597] Step 16:

[1598] The terminal uses the Amazon Polly speech synthesis engine to notify the user by voice of the message "Payment completed." The input is the text data of the completion message, and the output is the voice notification.

[1599] Through the above processing steps, payment can be completed quickly and efficiently using only voice, without using a QR code.

[1600] (Application example 1)

[1601] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1602] While the convenience of electronic payment services has improved in recent years, many systems require visual input, such as QR codes or touch operations. This poses a challenge for users with visual impairments and those unfamiliar with operating digital devices. Furthermore, existing voice payment systems lack sufficient speech recognition accuracy and natural language processing effectiveness, making it difficult to achieve smooth and prompt payments.

[1603] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1604] In this invention, the server includes a voice recognition unit, a natural language processing unit, and a terminal for capturing the user's voice commands. This allows the user to make payments quickly and accurately using only their voice, without the need for visual input. Furthermore, the user experience can be improved by using voice synthesis to deliver confirmation messages and payment completion notifications.

[1605] "Speech recognition means" is a technology for capturing a user's speech and converting it into text data.

[1606] "Natural language processing means" is a technology for analyzing text data and understanding the user's intent.

[1607] A "terminal for capturing user's voice commands" is a device for collecting user's voice and transmitting it to a server.

[1608] The "payment processing server" is a central control unit that processes payments based on captured voice commands.

[1609] "Means for confirming user account information and payment amount" refers to technology that allows the server to obtain and confirm the user's account information and payment amount.

[1610] "Means for notifying the user of the payment amount and requesting a confirmation response of 'OK'" refers to a technique for notifying the user of the payment amount by voice and requesting confirmation by voice.

[1611] The "means for receiving a user's confirmation response and completing the payment" refers to a technique for obtaining a user's confirmation response and completing the actual payment process.

[1612] "Means for notifying the user using voice synthesis" is a technology that converts a message from the server into voice and notifies the user.

[1613] "Means for notifying the user by voice after payment is completed" refers to a technique for notifying the user by voice that payment has been completed.

[1614] The present invention relates to a system that uses voice recognition and natural language processing to analyze a user's voice commands and execute payments by voice without using a QR code. This system is realized through cooperation between the user, terminals, and a server.

[1615] The system is configured as follows:

[1616] 1. A device that captures the user's voice command: When a user says "pay with voice," this device captures the voice and sends it over the internet to a server. For example, this could be a smartphone or smart glasses.

[1617] 2. Speech recognition method: The server uses speech recognition technology such as the Google Speech-to-Text API to convert the received speech into text data.

[1618] 3. Natural language processing means: Analyze text data using Google NLP API or similar to confirm the user's intent.

[1619] 4. Server that executes the payment process: The server obtains the user's account information and payment amount, and generates a confirmation message in the format "It's XX yen. Please say OK."

[1620] 5. Notifying the user using voice synthesis: The confirmation message from the server is converted into voice using the Google Text-to-Speech API and notified to the user.

[1621] 6. Means for receiving user confirmation response: When the user responds "OK", the terminal again captures this voice and sends it to the server.

[1622] 7. Completion of payment process: The server again uses voice recognition to confirm the user's response, then processes the payment and deducts the specified amount from the user's account.

[1623] 8. A means of notifying the user by voice after payment is completed: When payment is completed, the server generates a completion notification stating "Payment has been completed" and notifies the user by voice via the terminal.

[1624] Specific examples

[1625] Below is an example of a user purchasing a coffee at a cafe.

[1626] A user at a cafe says, "Pay by voice." The device captures this speech and sends it to the server. The server uses speech recognition to transcribe "Pay by voice" and uses natural language processing to understand the user's intent. The server then obtains the user's account information and payment amount (e.g., 500 yen), generates a confirmation message saying, "500 yen, right? Say OK," and converts this into voice to notify the user.

[1627] If the user responds "OK," the device captures the audio again and sends it to the server. After the server confirms this "OK," it starts the payment process, deducting 500 yen from the user's account. Finally, the server generates a message saying "Payment completed," converts it into audio, and notifies the user via the device.

[1628] This system allows even visually impaired people or users who have difficulty operating digital devices to make payments quickly and efficiently using only their voice.

[1629] Example prompt sentence:

[1630] "Voice command to pay"

[1631] "It's 500 yen. Just say OK."

[1632] "Payment completed"

[1633] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1634] Step 1:

[1635] The user says "Pay with voice." This voice command is captured by the terminal.

[1636] Input: The user's voice command.

[1637] Output: The captured audio data.

[1638] Step 2:

[1639] The device transmits the captured audio data to the server.

[1640] Input: The captured audio data.

[1641] Output: Sending audio data to the server.

[1642] Step 3:

[1643] The server converts the received voice data into text data using a voice recognition tool (Google Speech-to-Text API).

[1644] Input: The audio data sent to the server.

[1645] Output: Text data.

[1646] Step 4:

[1647] The server analyzes the text data using natural language processing tools (Google NLP API) and confirms the user's intent.

[1648] Input: Text data.

[1649] Output: Data that represents the user's intent.

[1650] Step 5:

[1651] The server obtains the user's account information and payment amount and generates a confirmation message saying, "It's 500 yen. Please say OK."

[1652] Input: Data representing the user's intent, and a database of account information on the server.

[1653] Output: Confirmation message (text format).

[1654] Step 6:

[1655] The server converts the confirmation message into voice data using a voice synthesis method (Google Text-to-Speech API) and sends it to the terminal.

[1656] Input: Confirmation message (in text format).

[1657] Output: Confirmation message (audio data).

[1658] Step 7:

[1659] The terminal notifies the user of the audio data.

[1660] Input: Confirmation message (audio data).

[1661] Output: Audio notification to the user.

[1662] Step 8:

[1663] The user says "OK." This speech is again captured by the device and sent to the server.

[1664] Input: User's spoken acknowledgment.

[1665] Output: The captured audio data of the acknowledgment.

[1666] Step 9:

[1667] The server converts the received voice data of the confirmation response into text data using a voice recognition method (Google Speech-to-Text API).

[1668] Input: Acknowledgment audio data.

[1669] Output: Acknowledgement text data.

[1670] Step 10:

[1671] After the server confirms "OK," it starts the payment process and deducts the specified amount from the user's account.

[1672] Input: Text data of confirmation response, and account information database in the server.

[1673] Output: Payment processing completed.

[1674] Step 11:

[1675] The server generates a payment completion message, converts it into voice data using a voice synthesis method (Google Text-to-Speech API), and sends it to the terminal.

[1676] Input: Payment completion message (text format).

[1677] Output: Payment completion message (audio data).

[1678] Step 12:

[1679] The terminal notifies the user by voice of a payment completion message.

[1680] Input: Payment completion message (audio data).

[1681] Output: Notification to the user that payment has been completed.

[1682] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1683] This invention relates to a system that combines voice recognition, natural language processing, and emotion recognition to analyze conversations between users and store clerks and execute voice payments without using QR codes. By linking the user's voice instructions, server processing, terminal operation, and emotion recognition, this system provides a smooth, efficient payment experience that takes emotions into consideration for the user.

[1684] Program Overview

[1685] The system operates as follows:

[1686] User voice instructions

[1687] The user issues a voice command such as "Please use voice." The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet.

[1688] Server-based voice analysis

[1689] The server passes the received voice data to a speech recognition module, converts it into text, and uses a natural language processing module to analyze the text data and confirm the user's intention.The server then uses an emotion engine to identify and analyze the user's emotion from the voice data.

[1690] Emotion-Based Processing

[1691] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is anxious, it will choose calming words. On the other hand, if the user is in a calm state, it will respond normally.

[1692] Generate and notify confirmation messages

[1693] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[1694] User Acknowledgment

[1695] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[1696] Server response confirmation

[1697] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[1698] Notification of payment completion

[1699] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[1700] Specific examples

[1701] scenario

[1702] Consider the example of a user purchasing a coffee at a cafe.

[1703] User:

[1704] "Audio please."

[1705] Device:

[1706] The user's voice is captured and sent to the server.

[1707] server:

[1708] The received voice data is converted into text and recognized as "Voice please." Next, the emotion engine is used to analyze the user's tone of voice and determine that the user is a little nervous. The server retrieves the user's account information and payment amount (for example, 500 yen) from the database. It generates a message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon," and sends it to the terminal.

[1709] Device:

[1710] The user is notified by voice, "It's 500 yen. Say OK. Don't worry, it will be over soon."

[1711] User:

[1712] Respond with "OK."

[1713] Device:

[1714] Capture the user's "OK" and send it to the server.

[1715] server:

[1716] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[1717] Device:

[1718] The user is notified by voice that "Payment has been completed."

[1719] This process allows users to experience an emotionally sensitive voice payment experience, particularly reducing confusion and tension. It also allows store clerks to complete payments quickly and efficiently. The system is also easy to use for people with visual impairments and those who have difficulty operating digital devices, simplifying the payment process in stores.

[1720] The processing flow will be explained below.

[1721] Step 1:

[1722] The user issues a voice command such as "Please speak to me." The device captures this voice command through the microphone and stores it as voice data.

[1723] Step 2:

[1724] The device sends the captured audio data to the server, and the server receives the audio data.

[1725] Step 3:

[1726] The server passes the received voice data to the voice recognition module, which converts the voice data into text. The server then analyzes the converted text, "Please speak to me."

[1727] Step 4:

[1728] The server retrieves the user's account information and payment amount from the database, and then uses an emotion engine to identify the user's emotion from the voice data.

[1729] Step 5:

[1730] The server analyzes the emotion engine results and determines that the user is in a tense state. Based on this result, it generates a confirmation message with additional words to reassure the user.

[1731] Step 6:

[1732] The server generates a confirmation message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon." and sends it to the device. The device passes the received confirmation message to the speech synthesis engine, converts it into speech, and notifies the user.

[1733] Step 7:

[1734] The user responds by saying "OK." The device recaptures this audio and saves it as new audio data.

[1735] Step 8:

[1736] The device sends the newly captured audio data to the server, and the server receives the audio data.

[1737] Step 9:

[1738] The server passes the received voice data to the voice recognition module again and converts it into text format. The server confirms "OK".

[1739] Step 10:

[1740] If the server receives the "OK" confirmation, it starts the payment process: the server debits the specified amount from the user's account.

[1741] Step 11:

[1742] The server confirms that the payment process was successful and generates a "Payment completed" message and sends it to the terminal.

[1743] Step 12:

[1744] The terminal passes the received payment completion message to a speech synthesis engine, which converts it into speech and notifies the user, "Payment has been completed."

[1745] Example 2

[1746] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1747] Conventional payment systems require users to use a visual or physical interface to make payments, which is inconvenient for the visually impaired and those who have difficulty operating digital devices. Furthermore, they are unable to provide services that respond to the user's emotional state, leaving room for improvement in the user experience. Furthermore, voice payment methods that do not use physical media such as QR codes are immature, preventing smooth payments.

[1748] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice recognition means, a natural language processing means, a means for capturing a user's voice instruction, a processing device that executes payment based on the captured voice instruction, a means for confirming the user's account information and payment amount, a means for notifying the user of the payment amount and requesting a confirmation response of "OK," a means for receiving the user's confirmation response and completing the payment, and a means for identifying the user's emotional state using an emotion recognition means and generating an appropriate response. This makes it easy for visually impaired people and users who have difficulty operating digital devices to use, and enables smooth voice payment that takes the user's emotions into consideration.

[1749] "Speech recognition means" is a technology that converts voice data into text data.

[1750] "Natural language processing means" is a technology that analyzes text data and understands context and intent.

[1751] A "means for capturing user voice instructions" is any device or technology that collects a user's voice and converts it into a digital form.

[1752] A "processing device that executes a payment procedure based on captured voice instructions" is an electronic device or system that receives voice instruction data and initiates a payment process.

[1753] "Means for confirming user account information and payment amount" refers to technology for obtaining and confirming user account information and payment amount from a database or the like.

[1754] "Means for notifying the user of the payment amount and requesting a confirmation response of 'OK'" refers to a technique or device that notifies the user of the payment amount and requests a verbal response of 'OK'.

[1755] The "means for receiving a user's confirmation response and completing the payment" refers to a technique or device that receives a user's confirmation response of "OK" and completes the payment process.

[1756] "Emotion recognition means" is a technology for identifying emotions from the user's voice and analyzing their state.

[1757] The "means for generating an appropriate response" refers to a technique or device that takes into account the emotional state of the user and generates an optimal response message.

[1758] MODE FOR CARRYING OUT THE INVENTION

[1759] This invention relates to a system that combines voice recognition, natural language processing, and emotion recognition to analyze conversations between users and store clerks and execute voice payments without using QR codes. By linking the user's voice instructions, server processing, terminal operation, and emotion recognition, this system provides a smooth, efficient payment experience that takes emotions into consideration for the user.

[1760] Hardware and Software Usage

[1761] User voice instructions

[1762] A user issues a voice command such as "Please speak to me" at a store. The device captures this command and obtains the voice data. The device then sends this data to a server via the Internet. A high-performance microphone is used as the hardware for capturing the voice.

[1763] Server-based voice analysis

[1764] The server passes the received voice data to a speech recognition module such as Google Cloud Speech-to-Text and converts it into text. It also uses a natural language processing module such as GPT-3 to analyze the text data and confirm the user's intent. The server then uses an emotion engine such as IBM Watson Tone Analyzer to identify and analyze the user's emotions from the voice data.

[1765] Emotion-Based Processing

[1766] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is anxious, it will choose calming words. On the other hand, if the user is in a calm state, it will respond normally.

[1767] Generate and notify confirmation messages

[1768] The server generates a confirmation message saying, "It's XX yen. Please say OK," and sends it to the device. The device receives this confirmation message and uses Microsoft Azure Text-to-Speech as a speech synthesis engine to notify the user by voice.

[1769] User Acknowledgment

[1770] The user responds by saying "OK." The device captures this voice and creates new voice data, which it then sends to the server.

[1771] Server response confirmation

[1772] The server again uses the speech recognition module to convert the user's response into text data and confirms it as "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[1773] Notification of payment completion

[1774] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." This is sent to the terminal, and the terminal notifies the user of the received message by voice.

[1775] Specific examples

[1776] scenario

[1777] Consider the example of a user purchasing a coffee at a cafe.

[1778] User:

[1779] "Audio please."

[1780] Device:

[1781] The user's voice is captured and sent to the server.

[1782] server:

[1783] The received voice data is converted into text and recognized as "Voice please." Next, the emotion engine is used to analyze the user's tone of voice and determine that the user is a little nervous. The server retrieves the user's account information and payment amount (for example, 500 yen) from the database. It generates a message saying, "It's 500 yen. Say OK. Don't worry, it will be done soon," and sends it to the terminal.

[1784] Device:

[1785] The user is notified by voice, "It's 500 yen. Say OK. Don't worry, it will be over soon."

[1786] User:

[1787] Respond with "OK."

[1788] Device:

[1789] Capture the user's "OK" and send it to the server.

[1790] server:

[1791] After confirming "OK", the payment process will begin, deducting 500 yen from the user's account. After confirming that the payment has been completed, a message stating "Payment completed" will be generated and sent to the terminal.

[1792] Device:

[1793] The user is notified by voice that "Payment has been completed."

[1794] This process allows users to experience an emotionally sensitive voice payment experience, particularly reducing confusion and tension. It also allows store clerks to complete payments quickly and efficiently. The system is also easy to use for people with visual impairments and those who have difficulty operating digital devices, simplifying the payment process in stores.

[1795] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1796] Step 1:

[1797] The user issues a voice command such as "Voice please." The device uses the user's voice instruction as input and captures voice data with a microphone. The device converts the captured voice data into a digital format and sends it to a server via the Internet. The output is the voice data sent to the server.

[1798] Step 2:

[1799] The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) and converts it into text data. In this step, the input voice data is converted into text. The output is in text format, "Please speak by voice."

[1800] Step 3:

[1801] The server passes the text data to a natural language processing module (e.g., GPT-3) to analyze the user's intent. Here, the text data is used as input to analyze and understand the user's intent and request. This process confirms that the user wishes to make a voice payment. The output is the analysis result regarding the user's intent.

[1802] Step 4:

[1803] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to identify and analyze the user's emotion from the voice data. The input is the voice data, and the voice tone and emotional state are analyzed. This process identifies the user's emotional state (e.g., nervousness). The output is the emotion recognition result.

[1804] Step 5:

[1805] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. The input is the emotion recognition results and the analysis results of the user's intention. For example, if the user is nervous, a calming message is generated. The output is a response message.

[1806] Step 6:

[1807] The server retrieves the payment amount (e.g., 500 yen) from the database and creates a confirmation message saying, "It's 500 yen. Please say OK." It also adds an additional message that takes emotion into consideration. The input is the user's account information and the payment amount. The output is a confirmation message.

[1808] Step 7:

[1809] The server sends the generated confirmation message to the terminal. The input is the confirmation message. The terminal receives this message. The output is the terminal that received the confirmation message.

[1810] Step 8:

[1811] The terminal converts the received confirmation message into speech using a speech synthesis engine (e.g., Microsoft Azure Text-to-Speech). The input is the text data of the confirmation message. The terminal notifies the user by voice, "That's 500 yen. Say OK. Don't worry, it will be done soon." The output is a voice notification.

[1812] Step 9:

[1813] The user responds with "OK" by voice. The terminal captures the voice data again using the user's confirmation response as input. The terminal converts this data into a digital format and sends it back to the server. The output is the voice data of the confirmation response sent to the server.

[1814] Step 10:

[1815] The server passes the sent confirmation response voice data to the voice recognition module again and converts it into text data. The input is the confirmation response voice data. The converted text data "OK" is output.

[1816] Step 11:

[1817] The server confirms the textual "OK" and starts the payment process. The input is "OK" as text data and the user's account information. The server deducts the specified amount from the user's account and makes the payment. The output is payment completion data.

[1818] Step 12:

[1819] The server confirms that the payment has been completed and generates a message saying "Payment has been completed." The input is the payment completion data. The output is the payment completion message.

[1820] Step 13:

[1821] The server sends the generated payment completion message to the terminal. The input is the payment completion message. The terminal receives this message. The output is the terminal that received the payment completion message.

[1822] Step 14:

[1823] The terminal converts the received payment completion message into speech using a speech synthesis engine. The input is the text data of the payment completion message. The terminal notifies the user by voice, "Payment has been completed." The output is a voice notification.

[1824] (Application example 2)

[1825] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1826] Conventional voice payment systems rely solely on voice recognition and natural language processing, and have the problem of being unable to provide appropriate responses that take into account the user's emotional state. Furthermore, the payment process often does not proceed smoothly when the user feels stressed or nervous. Furthermore, certain user groups, such as the visually impaired and elderly, have the problem of difficulty operating the system. There is a need to solve these issues and provide users with a comfortable and smooth payment experience.

[1827] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means, a natural language processing means, and an emotion recognition means. This makes it possible to accurately recognize the user's voice command, generate an appropriate response that takes their emotional state into consideration, and execute a smooth payment process that takes the user's emotions into consideration.

[1828] "Speech recognition means" is a general term for a device or program that captures a user's voice commands and converts them into digital text data.

[1829] "Natural language processing means" is a general term for technology or programs that analyze the user's intentions and meaning from text data and determine appropriate responses and actions.

[1830] "Terminal" is a general term for an electronic device that captures a user's voice and transmits that data to a server.

[1831] "Emotion recognition means" is a general term for technology or programs for identifying and analyzing a user's emotional state from voice data or other input.

[1832] "Server" is a general term for a computer system that performs speech recognition, natural language processing, and emotion recognition, and manages and executes the payment process.

[1833] "Account Information" means, collectively, your personal identification information and payment-related data.

[1834] "Payment amount" refers to the amount to be paid by the user.

[1835] "Response generation means" is a general term for a technology or program for generating an appropriate response message based on the analysis results.

[1836] "Acknowledgement" refers to the voice instructions given by the user to proceed with the payment process.

[1837] This invention is a system that combines speech recognition, natural language processing, and emotion recognition, allowing users to make payments quickly and smoothly using voice commands. The system includes a speech recognition unit, a natural language processing unit, an emotion recognition unit, a server, and a terminal that captures the user's voice.

[1838] The specific operation of the system is configured as follows.

[1839] Hardware and Software Configuration

[1840] Device: A smartphone to capture the user's voice.

[1841] Server: A server for processing voice data. A computer system required for speech recognition, natural language processing, and emotion recognition.

[1842] Speech Recognition Tool: Software that converts speech to text using the SpeechRecognition library.

[1843] Natural language processing tools: Software that uses TextBlob to analyze spoken text and understand user intent.

[1844] Emotion Recognition: Software that uses EmotionClassifier (emopy library) to identify user emotions.

[1845] Program processing overview

[1846] 1. User issues a voice command

[1847] The user issues a voice command to the device, saying, "Please use voice." The device captures this voice and sends the voice data to the server.

[1848] 2. Audio analysis by the server

[1849] The server converts the received voice data into text format using a voice recognition unit, then analyzes the text data using a natural language processing unit to confirm the user's intention, and further identifies the user's emotion from the voice data using an emotion recognition unit.

[1850] 3. Emotion-based response generation

[1851] The server generates an appropriate response based on the user's emotional state based on the emotion recognition results. For example, if the user is nervous, it will choose calming words, and if the user is in a relaxed state, it will respond normally.

[1852] 4. Generating and notifying confirmation messages

[1853] The server generates a confirmation message saying "It's XX yen. Please say OK" and sends it to the terminal. The terminal receives the confirmation message and uses a speech synthesis engine to notify the user by voice.

[1854] 5. User Acknowledgment

[1855] The user responds by saying "OK." This voice is also captured by the device, and new voice data is created and sent to the server.

[1856] 6. The server confirms the response and starts the payment process

[1857] The server again uses the speech recognition means to convert the user's response into text data and confirms "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[1858] 7. Notification of payment completion

[1859] The server confirms that the payment has been completed, generates a completion message, and sends it to the terminal. The terminal then notifies the user of the received message by voice.

[1860] Specific examples

[1861] As a concrete example, let's consider the process when a user purchases coffee at a cafe. In this case, the process is as follows:

[1862] 1. User: "Voice please."

[1863] 2. Device: Captures the user's voice and sends it to the server.

[1864] 3. Server: Converts the received voice data into text and recognizes "Voice please." Analyzes the user's tone of voice using an emotion recognition engine and determines that the user is a little nervous. Retrieves the user's account information and payment amount (e.g., 500 yen) from the database. Generates a message saying, "500 yen, right? Say OK. Don't worry, it will be done soon," and sends it to the device.

[1865] 4. Terminal: A voice message will say, "It's 500 yen. Say OK. Don't worry, it will be done soon."

[1866] 5. User: Responds "OK."

[1867] 6. Terminal: Captures the user's "OK" and sends it to the server.

[1868] 7. Server: After confirming "OK", it deducts 500 yen and starts the payment process. It confirms that the payment has been completed, generates a completion message, and sends it to the terminal.

[1869] 8. Terminal: The user is notified by voice that "Payment has been completed."

[1870] This allows users to experience a smooth payment experience that takes their emotions into consideration.

[1871] Example prompts for generative AI models

[1872] "Generate a calming response message for users who are a little nervous when paying by voice."

[1873] Such a system will enable voice payments that are easy to use, especially for those with visual impairments or who have difficulty operating digital devices.

[1874] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1875] Step 1:

[1876] The user issues a voice command to the device, saying, "Please speak to me." The device captures this voice and generates voice data, which is then sent directly to the server.

[1877] Input: User's voice command

[1878] Output: Audio data (sent to server)

[1879] How it works: When a user speaks into the smartphone application, the microphone captures the voice, and the voice data is transmitted to the server in real time.

[1880] Step 2:

[1881] The server converts the received voice data into text using speech recognition means, which is then analyzed by natural language processing means to ascertain the user's intent.

[1882] Input: Audio data

[1883] Output: Text data

[1884] How it works: The SpeechRecognition library on the server converts the audio data into text, then uses the TextBlob library to analyze the text and extract the user's intent.

[1885] Step 3:

[1886] The server analyzes the user's emotional state using the text data and the voice data with an emotion recognition means, and the analyzed emotional information is further processed.

[1887] Input: Text and audio data

[1888] Output: Emotion recognition result

[1889] Specific operation: The emopy library is used to analyze the user's emotional state from audio and text data, and the results are obtained. For example, states such as "tense" or "calm" are identified.

[1890] Step 4:

[1891] The server generates an appropriate response message based on the emotion recognition results, depending on the user's emotional state. For example, if the user is nervous, it generates a message containing calming words.

[1892] Input: Emotion recognition results, text data

[1893] Output: Response message

[1894] Specific behavior: The server uses the text template to generate a response message appropriate to the emotional state, such as "500 yen. Say OK. Don't worry, it will be over soon."

[1895] Step 5:

[1896] The server sends the generated response message to the terminal, and the terminal uses a speech synthesis engine to notify the user by voice.

[1897] Input: Response message

[1898] Output: Audio notification

[1899] Specific operation: When a device receives a text message sent from the server, it uses a text-to-speech engine to convert the message into audio and convey it to the user.

[1900] Step 6:

[1901] The user responds by saying "OK." This voice is also captured by the device, and new voice data is generated and sent to the server again.

[1902] Input: User's voice response

[1903] Output: Audio data (sent to server)

[1904] What happens: The user says "OK" to confirm. The device's microphone captures this sound and sends it to the server.

[1905] Step 7:

[1906] The server again uses the speech recognition means to convert the user's response into text data and confirms "OK." After confirmation, the server starts the payment process and debits the specified amount from the user's account.

[1907] Input: Audio data

[1908] Output: Confirmation "OK" and start of payment process

[1909] What happens: The SpeechRecognition library on the server converts the voice data into text and uses natural language processing to confirm "OK." Once this confirmation is received, the payment process is carried out.

[1910] Step 8:

[1911] The server confirms that the payment has been completed, generates a message stating "Payment has been completed," and sends it to the terminal.

[1912] Input: Confirm payment completion

[1913] Output: Completion message

[1914] Specific operation: Once the payment process is complete, the server generates a confirmation message and sends it to the terminal, which is then notified to the user by the speech synthesis engine.

[1915] This series of steps provides users with an emotionally sensitive and smooth payment experience.

[1916] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1917] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1918] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1919] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1920] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1921] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1922] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1923] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1924] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1925] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1926] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1927] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1928] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1929] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1930] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1931] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1932] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1933] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1934] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1935] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1936] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1937] The following is further disclosed regarding the above embodiment.

[1938] (Claim 1)

[1939] a voice recognition means;

[1940] natural language processing means;

[1941] a terminal for capturing voice commands of a user;

[1942] a server that executes a payment process based on the captured voice commands;

[1943] A means for verifying the user's account information and payment amount;

[1944] means for notifying the user of the payment amount and requesting an "OK" confirmation response;

[1945] means for receiving a user confirmation response and completing the payment;

[1946] A system including:

[1947] (Claim 2)

[1948] 10. The system of claim 1, further comprising a voice recognition means for converting a user's voice commands and acknowledgments into text data.

[1949] (Claim 3)

[1950] 10. The system of claim 1, further comprising means for analyzing a user's voice commands using natural language processing means.

[1951] "Example 1"

[1952] (Claim 1)

[1953] means for capturing a user's voice commands;

[1954] means for transmitting the captured voice commands over a network;

[1955] a speech recognition means for converting speech data into text form;

[1956] natural language processing means for analyzing text data;

[1957] A means for confirming the user's intent and obtaining account information and payment amount;

[1958] means for notifying the user of the payment amount and requesting an "OK" confirmation response;

[1959] means for capturing the user's confirmation response and sending it back to the server;

[1960] A means for verifying the converted text data and initiating a payment process;

[1961] means for generating a payment completion message and notifying the user;

[1962] A system including:

[1963] (Claim 2)

[1964] 10. The system of claim 1, further comprising a voice recognition means for converting the voice data and the acknowledgement into text data.

[1965] (Claim 3)

[1966] 10. The system of claim 1, further comprising means for analyzing a user's voice command using natural language processing means.

[1967] "Application Example 1"

[1968] (Claim 1)

[1969] a voice recognition means;

[1970] natural language processing means;

[1971] a terminal for capturing voice commands of a user;

[1972] a server that executes a payment process based on the captured voice commands;

[1973] A means for verifying the user's account information and payment amount;

[1974] means for notifying the user of the payment amount and requesting an "OK" confirmation response;

[1975] means for receiving a user confirmation response and completing the payment;

[1976] means for notifying a user using speech synthesis;

[1977] A means for notifying the user by voice after the payment is completed;

[1978] A system including:

[1979] (Claim 2)

[1980] 10. The system of claim 1, further comprising a voice recognition means for converting a user's voice commands and acknowledgments into text data.

[1981] (Claim 3)

[1982] 10. The system of claim 1, further comprising means for analyzing a user's voice command using natural language processing means.

[1983] "Example 2: Combining Emotion Engines"

[1984] Rewritten claims:

[1985] (Claim 1)

[1986] a voice recognition means;

[1987] natural language processing means;

[1988] a device for capturing voice instructions of a user;

[1989] a processor that executes a payment transaction based on the captured voice instructions;

[1990] A means for verifying the user's account information and payment amount;

[1991] means for notifying the user of the payment amount and requesting an "OK" confirmation response;

[1992] means for receiving a user confirmation response and completing the payment;

[1993] means for identifying the user's emotional state using emotion recognition means and generating an appropriate response;

[1994] A system including:

[1995] (Claim 2)

[1996] 10. The system of claim 1, further comprising a voice recognition means for converting a user's voice instructions and acknowledgments into text data.

[1997] (Claim 3)

[1998] 10. The system of claim 1, further comprising means for analyzing a user's voice instructions using natural language processing means to ascertain the user's intent.

[1999] "Application example 2 when combining emotion engines"

[2000] (Claim 1)

[2001] a voice recognition means;

[2002] natural language processing means;

[2003] a terminal for capturing voice commands of a user;

[2004] An emotion recognition means;

[2005] a server that executes a payment process based on the captured voice commands;

[2006] A means for verifying the user's account information and payment amount;

[2007] means for generating an appropriate response based on the emotion analysis result and notifying the user;

[2008] means for notifying the user of the payment amount and requesting an "OK" confirmation response;

[2009] means for receiving a user confirmation response and completing the payment;

[2010] A system including:

[2011] (Claim 2)

[2012] 10. The system of claim 1, further comprising a voice recognition means for converting a user's voice commands and acknowledgments into text data.

[2013] (Claim 3)

[2014] 10. The system of claim 1, further comprising means for analyzing a user's voice command using natural language processing means. [Explanation of symbols]

[2015] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a voice recognition means; natural language processing means; a terminal for capturing voice commands of a user; a server that executes a payment process based on the captured voice commands; A means for verifying the user's account information and payment amount; means for notifying the user of the payment amount and requesting an "OK" confirmation response; means for receiving a user confirmation response and completing the payment; A system including:

2. 2. The system of claim 1, further comprising speech recognition means for converting a user's voice commands and acknowledgments into text data.

3. 10. The system of claim 1, further comprising means for analyzing a user's voice commands using natural language processing means.

Images