System

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring a AI input from a user, a speech recognition engine means for primarily converting the acquired speech input into a text, a generation speech model means for analyzing the context of the primarily converted text and complementing and correcting the text, and a means for transmitting the complemented and corrected final text to a user device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current speech recognition systems have a poor ability to understand subtle differences in pronunciation and context, resulting in the output of words or sentences that differ from what the user intended. Such inaccurate recognition results cause significant stress and inconvenience to users. This problem is particularly pronounced when inputting important conversations or formal documents. The present invention aims to solve this problem in current speech recognition systems and achieve more accurate and natural speech input. [Means for solving the problem]

[0005] The present invention provides a speech input system including a means for acquiring speech input from a user, a speech recognition engine means for primarily converting the acquired speech input into text, a generative AI model means for analyzing the context surrounding the primarily converted text and completing and correcting it, and a means for transmitting the completed and corrected final text to a user terminal. This system makes it possible to provide accurate and natural-sounding text to the user by appropriately correcting words and sentences inaccurately converted by the primary speech recognition using the generative AI model. Furthermore, this means significantly improves the accuracy of the speech recognition system and enhances user convenience.

[0006] "User" refers to a person using the system to provide voice input.

[0007] "Voice input" refers to the process of a user uttering words aloud and the system capturing that speech as digital data.

[0008] "Speech recognition engine" refers to a software or hardware component that analyzes captured voice data and converts it into text.

[0009] "Initially converted to text" refers to the text conversion result that the speech recognition engine initially produces, which may be subsequently modified.

[0010] A "generative AI model" refers to an artificial intelligence model that analyzes the context of primary converted text and complements and corrects inaccurate parts.

[0011] "Context" refers to the words and phrases that exist before and after the primary converted text, and includes information that is important for understanding the meaning and semantics of the sentence.

[0012] "Completion and correction" refers to the process in which the generative AI model appropriately corrects inaccurate or missing parts of the primary converted text.

[0013] "Final text" refers to the accurate, natural-sounding text after the generative AI model has completed and corrected it.

[0014] "User terminal" refers to a device (e.g., smartphone, tablet, computer) through which a user makes voice input.

[0015] "Transmission" refers to the process in which the server transmits data (text resulting from voice input) to the user terminal.

[0016] "System" refers to a set of hardware and software combinations that executes voice input and converts, complements, corrects, and transmits the results to text. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention relates to a system and method for improving the accuracy of a speech recognition system. This system receives speech input from a user, converts it into text, and then uses a generative AI model to complement and correct inaccuracies based on the context surrounding the text. The system's program and processing are described in detail below.

[0039] Program processing

[0040] 1. Acquiring voice input:

[0041] Terminal: The user speaks into the terminal, which uses a microphone to capture the voice and stores the data digitally.

[0042] Device: Once the audio capture is complete, the digital audio data is sent over the internet to a server.

[0043] 2. Primary speech recognition:

[0044] Server: Passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[0045] Server: Temporarily stores the converted text. This becomes the primary speech recognition result. This result is further supplemented and corrected in subsequent processing steps.

[0046] 3. Contextual understanding and completion:

[0047] Server: Inputs the primary speech recognition results into a generative AI model, which analyzes the context of the text and identifies inaccuracies or inaudible parts.

[0048] Server: The generative AI model uses contextual analysis to correct inaccuracies in the primary speech recognition results. For example, "The meeting is from Sanji" is corrected to "The meeting starts at 3 o'clock."

[0049] 4. Generate and send the final result:

[0050] Server: Formats the completed and corrected text and prepares it as the final text.

[0051] Server: Sends the prepared final text to the user's terminal.

[0052] On the device, the final text received is displayed to the user, allowing the user to see accurate and natural-looking results of their voice input.

[0053] Specific examples

[0054] Example 1:

[0055] User: Says, "Let's put this on hold."

[0056] Terminal: Captures audio and sends digital audio data to the server.

[0057] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[0058] Server: A generative AI model analyzes this primary text in context and modifies "fencing" to "pending."

[0059] Server: Sends the final text "Let's put this on hold" to the user terminal.

[0060] Terminal: Show the user "Let's put this on hold."

[0061] Example 2:

[0062] User: Say, "Today's meeting starts at 3:00."

[0063] Terminal: Captures audio and sends digital audio data to the server.

[0064] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[0065] Server: The generative AI model analyzes this primary text taking into account the context and corrects "Sanji" to "Miji".

[0066] Server: Sends the final text "Today's meeting starts at 3:00" to the user terminal.

[0067] Terminal: Display to the user "Today's meeting starts at 3:00."

[0068] In this way, the present invention significantly improves the accuracy of speech recognition and provides users with a more natural and accurate speech input experience. The system of the present invention enables users to achieve stress-free communication.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] Device: Captures the user's voice. The user speaks into the device, and the device's microphone converts the voice into a digital format and stores it in an internal buffer.

[0072] Step 2:

[0073] Terminal: After the audio capture is complete, the digital audio data is sent to the server via the Internet using HTTP requests or WebSocket communication.

[0074] Step 3:

[0075] Server: Passes the received voice data to a voice recognition engine (e.g., a commercial voice recognition API), which analyzes the input voice data and converts it into temporary text.

[0076] Step 4:

[0077] Server: Receives the text output from the speech recognition engine and stores it in temporary memory (e.g., a database or cache memory). The text at this stage is the primary speech recognition result.

[0078] Step 5:

[0079] Server: Passes the primary speech recognition results to a generative AI model for context analysis. The generative AI model (e.g., a model based on natural language processing) analyzes the input text and its context to identify inaccurate or unclear parts.

[0080] Step 6:

[0081] Generative AI model (on server):

[0082] The AI ​​model corrects any identified inaccuracies based on the primary speech recognition results.

[0083] For example, the incorrectly recognized part "Sanji" is corrected to "three o'clock."

[0084] Step 7:

[0085] Server: Takes the corrected text data, reformats it as final text, and passes it on to other applications or services as needed.

[0086] Step 8:

[0087] Server: Sends the final text to the user's device, using JSON or another data exchange format.

[0088] Step 9:

[0089] Terminal: Parses the data received from the server and extracts the final text. An appropriate UI component (e.g., a text view or popup) is used to display the extracted text to the user.

[0090] Step 10:

[0091] Terminal: Displays the final text to the user, who can review it and take further action if necessary.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] Current speech recognition systems are prone to misrecognition during the initial conversion of speech input to text, resulting in inaccurate recognition of the user's intent, especially in noisy environments and complex contexts.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for primarily converting voice into text, a generative AI model means for analyzing the context of the primarily converted text and supplementing and correcting it, and a means for transmitting the supplemented and corrected final text to a user terminal, thereby improving the accuracy of voice recognition, accurately reflecting the user's intention, and enabling a more natural and accurate voice input experience.

[0097] "Voice input" is a series of voice data uttered by a user into a terminal.

[0098] "Speech recognition engine means" means software or hardware that utilizes machine learning algorithms to analyze voice input and convert digital voice data into text.

[0099] The "generative AI model means" is an artificial intelligence model that uses natural language processing technology to analyze the context of the primarily converted text and complement and correct inaccurate parts.

[0100] A "user terminal" is a device that accepts voice input and displays the resulting text. Examples include smartphones, tablets, and PCs.

[0101] "Primary speech recognition result" is the text data initially generated from speech input by the speech recognition engine means.

[0102] "Completion and correction" is the process by which the generative AI model detects inaccuracies in the primary speech recognition results and corrects them based on appropriate context.

[0103] "Final text" refers to text data that has been completed through supplementation and correction by the generative AI model means.

[0104] The present invention relates to a system and method for improving the accuracy of a voice input system. The system receives voice input from a user, converts it to text, and then uses a generative AI model to complement and correct inaccuracies based on the context of the text. Specific embodiments of the system are described in detail below.

[0105] First, the user speaks into their device (e.g., smartphone, tablet, PC, etc.). The device has a built-in microphone that captures the user's voice and converts it into digital voice data. This digital voice data is sent to a server via the Internet.

[0106] The server uses a speech recognition engine to convert the received digital voice data into text. Examples of the speech recognition engine that can be used include Google® Cloud Speech-to-Text and DeepSpeech. The speech recognition engine analyzes the voice data and generates a primary speech recognition result. This primary speech recognition result is text data in the form of, for example, "The meeting is from Sanji."

[0107] The server then inputs the primary speech recognition results into a generative AI model. This generative AI model uses natural language processing technologies such as GPT-4 (registered trademark) and BERT. The generative AI model analyzes the context to identify and correct inaccuracies in the primary speech recognition results. For example, "Sanji" may be corrected to "Miji." Through this correction and completion process, the generative AI model generates the final text.

[0108] The final text after completion and correction is sent from the server to the user's device, which has the function of displaying the received final text, allowing the user to check the results of their voice input accurately and naturally.

[0109] A specific example is given below.

[0110] Specific examples

[0111] Example 1

[0112] User: Says, "Let's put this on hold."

[0113] Terminal: Captures audio and sends digital audio data to the server.

[0114] Server: A speech recognition engine (e.g., Google Cloud Speech-to-Text) converts the speech data into the initial text "Let's fence this."

[0115] Server: A generative AI model (e.g., GPT-4) analyzes the primary text based on context and modifies "fencing" to "pending."

[0116] Server: Sends the final text "Let's put this on hold" to the user terminal.

[0117] Terminal: Show the user "Let's put this on hold."

[0118] Example 2

[0119] User: Say, "Today's meeting starts at 3:00."

[0120] Terminal: Captures audio and sends digital audio data to the server.

[0121] Server: A speech recognition engine (e.g., DeepSpeech) converts the speech data into the text "Today's meeting is with Sanji."

[0122] Server: A generative AI model (e.g., BERT) analyzes the primary text based on context and modifies "Sanji" to "Miji."

[0123] Server: Sends the final text "Today's meeting starts at 3:00" to the user terminal.

[0124] Terminal: Display to the user "Today's meeting starts at 3:00."

[0125] An example of a prompt is as follows:

[0126] User Input: Let's put this on hold

[0127] Initial speech recognition result: Let's fence this one.

[0128] Completion and correction results by generative AI model: Let's put this on hold.

[0129] This system significantly improves the accuracy of voice recognition, providing users with a more natural and accurate voice input experience, enabling stress-free communication.

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1:

[0132] Acquiring voice input

[0133] User: The user speaks into the terminal. For example, the user says, "The meeting starts at 3 o'clock."

[0134] Device: The device's microphone captures the user's voice and converts it into digital audio data (e.g., WAV format). A microphone icon appears to indicate that recording is in progress.

[0135] Input: User's voice input.

[0136] Output: Digital audio data.

[0137] What it does: The device captures the audio data, stores it internally, and sends it to a server over the Internet (using the HTTPS protocol).

[0138] Step 2:

[0139] Primary Speech Recognition

[0140] Server: Receives digital audio data sent from the device. At this time, the audio data is temporarily stored in the server's memory.

[0141] Server: Using a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech), the voice data is converted into text, e.g., "The meeting is with Sanji."

[0142] Input: Digital audio data.

[0143] Output: Primary speech recognition result (text data).

[0144] Specific operation: The speech recognition engine starts working, analyzing and converting the voice data to generate text data. The user is notified that "Speech recognition in progress. Please wait a moment."

[0145] Step 3:

[0146] Contextual understanding and completion

[0147] Server: The primary speech recognition results are fed into a generative AI model (e.g., GPT-4 or BERT). The generative AI model analyzes the context of the text and complements or corrects inaccuracies. For example, correcting "Sanji" to "Miji."

[0148] Input: Primary speech recognition result (text data).

[0149] Output: The final completed and corrected text.

[0150] How it works: The generative AI model analyzes the context based on the primary speech recognition results and makes appropriate corrections. The progress of the analysis process is displayed in real time on the server console, and the user is informed that "text is being analyzed."

[0151] Step 4:

[0152] Generate and send the final result

[0153] Server: The completed and corrected text is formatted and prepared for final use, e.g., adding appropriate punctuation and formatting.

[0154] Server: Sends the final text to the user's device. Data is transferred securely using the HTTPS protocol.

[0155] Input: The completed and corrected final text.

[0156] Output: The final text that is displayed on the user's terminal.

[0157] Specific behavior: The server sends the final text to the user's device, and the device displays the received text to the user. A notification sound is played and a pop-up message appears saying "Processing completed. Please check the results."

[0158] (Application example 1)

[0159] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] Conventional speech recognition systems often fail to accurately capture user intent due to their low accuracy in converting speech to text. This problem is particularly severe in practical applications such as food delivery, where incorrectly recognized orders can lead to user dissatisfaction and a decline in service quality. Therefore, there is a need for systems that can more accurately convert speech input to text and provide context-based completion and correction.

[0161] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0162] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for converting the acquired voice input into text, a generative AI model means for analyzing the context of the converted text and completing and correcting it, a means for transmitting the completed and corrected final text to a user terminal, and a means for acquiring information using the voice input and confirming the user's order details, thereby improving the accuracy of voice recognition and enabling users to accurately place food delivery orders.

[0163] The "means for acquiring voice input from the user" is a function for acquiring information input by voice by the user as digital data using a device such as a microphone.

[0164] The "speech recognition engine means for primarily converting acquired voice input into text" is a system that executes a process of converting acquired voice data into text format using a machine learning algorithm.

[0165] "Generative AI model means for analyzing the context before and after primary converted text and completing and correcting it" is a function that uses generative AI technology to analyze the context of speech data that has been converted into text for the first time and to complete or correct inaccurate or missing parts.

[0166] "Means for sending the completed and corrected final text to the user's terminal" refers to the process of sending the text data completed and corrected by the generative AI model to the user's terminal.

[0167] "Means for obtaining information using voice input and confirming the user's order details" refers to a function that captures the order information entered by the user through voice and identifies and confirms the exact order details based on that information.

[0168] This invention is a system that accurately converts user speech input into text and enhances the user experience by complementing and correcting recognition errors, particularly in the context of food delivery orders.

[0169] System configuration and program processing

[0170] Hardware and Software

[0171] Hardware

[0172] User terminal: A smartphone or microphone is included for users to input voice.

[0173] Server: Equipped with high-performance CPU / GPU (e.g. NVIDIA A100).

[0174] software

[0175] Speech recognition engine: DeepSpeech (open source speech recognition engine)

[0176] Generative AI model: GPT-4 (generative AI model)

[0177] Food delivery application: Flutter (registered trademark) (UI framework)

[0178] Processing flow

[0179] 1. Acquiring voice input

[0180] Users enter their food delivery order details by voice into their smartphone.

[0181] The smartphone's microphone captures the sound and generates digital audio data.

[0182] 2. Primary speech recognition

[0183] The smartphone sends the captured audio data to the server.

[0184] DeepSpeech, located on the server, converts the voice data into text.

[0185] 3. Contextual understanding and completion

[0186] The converted text is input into the GPT-4 model on the server.

[0187] The GPT-4 model analyzes the context of the text to identify inaccuracies or parts that were not heard.

[0188] The GPT-4 model complements and corrects the primary text based on the analysis results.

[0189] 4. Generate and send the final result

[0190] Format the completed and corrected text and prepare it as the final order text.

[0191] The final text is sent to the user's smartphone, and the order details are displayed.

[0192] Specific operation examples

[0193] Example 1:

[0194] User: Say "I'd like a fried chicken set and a Coke, please."

[0195] Smartphone: Captures audio and sends digital audio data to a server.

[0196] Server: DeepSpeech converts the primary text "I'd like a fried chicken sed and one curried please."

[0197] GPT-4 model: Analyzes the context and corrects "kare" to "cola" and "sed" to "set."

[0198] Server: "I'd like a fried chicken set and a bottle of coke, please" is sent to the smartphone and displayed.

[0199] Example 2:

[0200] User: Say "I'd like a Margherita pizza and orange juice, please."

[0201] Smartphone: Captures audio and sends digital audio data to a server.

[0202] Server: DeepSpeech converts the primary text "I'd like a Margherita pizza and an Olathe juice, please."

[0203] GPT-4 model: Corrects "Olathe" to "Orange Juice."

[0204] Server: "Margherita pizza and orange juice please" is sent to the smartphone and displayed.

[0205] Example of input prompt for generative AI model

[0206] Prompt statement:

[0207] Based on the context of a food delivery order, please correct the following speech recognition output to the correct order:

[0208] Initial speech recognition results: Margherita pizza and Ol'ze juice

[0209] Such systems and methods enable users to place more accurate food delivery orders by voice, improving the user experience.

[0210] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0211] Step 1:

[0212] Users speak into their smartphone to order food delivery, and the smartphone's microphone captures the voice and generates digital audio data, which is then transmitted over the Internet to a server.

[0213] Input: Audio input

[0214] Output: Digital audio data

[0215] How it works: The smartphone's microphone captures the user's voice and stores it as digital audio data, which is then sent over the internet to a server.

[0216] Step 2:

[0217] The server passes the received digital voice data to DeepSpeech, which converts the voice data into text. This primary text data is temporarily stored on the server.

[0218] Input: Digital audio data

[0219] Output: Primary text data

[0220] How it works: DeepSpeech on the server analyzes the audio data and converts it into text. The converted primary text is stored on the server.

[0221] Step 3:

[0222] The server inputs the primary text data into the GPT-4 model, which analyzes the context of the primary text to identify inaccuracies or unclear parts, and then appropriately complements and corrects the inaccuracies based on the context.

[0223] Input: Primary text data

[0224] Output: Completion and correction of text data

[0225] How it works: The GPT-4 model on the server analyzes text and context to identify inaccuracies, then provides contextual completion and correction.

[0226] Step 4:

[0227] The server formats the completed and corrected text and prepares it as the final text data, which is then sent to the user's smartphone via the Internet.

[0228] Input: Completed and corrected text data

[0229] Output: Final text data

[0230] Operation: The text data is formatted on the server to generate the final text data, which is then sent to the user's smartphone via the Internet.

[0231] Step 5:

[0232] The user's smartphone receives the final text data and displays it to the user in an appropriate format, allowing the user to confirm the exact food delivery order details.

[0233] Input: Final text data

[0234] Output: The order displayed to the user

[0235] Operation: The smartphone analyzes the final text data received from the server and displays it on the screen in a format that the user can read.

[0236] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0237] The present invention relates to a system and method for improving the accuracy of speech recognition systems, particularly by combining an emotion engine that recognizes user emotions to achieve more natural and context-aware speech recognition. This system acquires speech input from a user, converts it into text, and then uses a generative AI model to analyze the context surrounding the text, complementing and correcting inaccuracies, while also incorporating the user's emotional information recognized by the emotion engine.

[0238] Program processing

[0239] 1. Acquiring voice input:

[0240] Terminal: The user speaks into the terminal, which uses a microphone to capture the voice and stores the data digitally.

[0241] Device: Once the audio capture is complete, the digital audio data is sent over the internet to a server.

[0242] 2. Primary speech recognition:

[0243] Server: Passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[0244] Server: Temporarily stores the converted text. This is the primary speech recognition result. This result is the first step in the completion and correction process.

[0245] 3. Emotion Recognition:

[0246] Server: Inputs the voice data into the emotion engine, which analyzes the user's tone and speaking style.

[0247] Emotion engine (on the server):

[0248] It recognizes the user's emotions (e.g., anger, joy, sadness) from voice data and passes that emotional information to a generative AI model.

[0249] 4. Contextual understanding and completion:

[0250] Server: Passes the primary speech recognition results and recognized emotion information to the generative AI model for context analysis. The generative AI model analyzes the input text, its context, and the emotion information to identify inaccuracies.

[0251] Server: The generative AI model complements and corrects inaccurate parts of the primary speech recognition results based on contextual analysis and emotional information. For example, "The meeting is from Sanji" is corrected to "The meeting starts at 3 o'clock."

[0252] 5. Generate and send the final result:

[0253] Server: Formats the completed and corrected text and prepares it as the final text.

[0254] Server: Sends the prepared final text to the user's device, along with the recognized emotion information.

[0255] Terminal: Displays the final text and emotion information received to the user.

[0256] Specific examples

[0257] Example 1:

[0258] User: Says, "Let's put this on hold."

[0259] Terminal: Captures audio and sends digital audio data to the server.

[0260] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[0261] Server: The emotion engine analyzes the voice data and recognizes the user's calmness.

[0262] Server: The generative AI model analyzes the primary text and sentiment information and modifies "fencing" to "pending."

[0263] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[0264] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[0265] Example 2:

[0266] User: "The meeting starts at 3 o'clock today," says angrily.

[0267] Terminal: Captures audio and sends digital audio data to the server.

[0268] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[0269] Server: The emotion engine analyzes the voice data and recognizes the user's anger.

[0270] Server: The generative AI model analyzes the primary text and emotional information and corrects "Sanji" to "Miji."

[0271] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[0272] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[0273] In this way, the present invention further improves the accuracy and naturalness of speech recognition by incorporating emotion information, allowing users to take more appropriate action based on the emotion recognized by the system.

[0274] The processing flow will be explained below.

[0275] Step 1:

[0276] Device: The user speaks into the device, the device's microphone captures the audio, converts it into a digital format, and stores it in an internal buffer.

[0277] Step 2:

[0278] Device: After the audio capture is complete, the digital audio data is sent to the server over the internet using an HTTP POST request or WebSocket communication.

[0279] Step 3:

[0280] Server: The server passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[0281] Step 4:

[0282] Server: Stores the primary text obtained from the speech recognition engine in temporary memory. This is the primary speech recognition result, which is supplemented and corrected in subsequent processing steps.

[0283] Step 5:

[0284] Server: Inputs the voice data into the emotion engine, which analyzes the user's tone and speaking style. Based on this analysis, the user's emotion (e.g., anger, joy, sadness) is recognized.

[0285] Step 6:

[0286] Server: Obtains the emotion information obtained from the emotion engine and passes it to the generative AI model, which receives the primary speech recognition result, context, and emotion information as input.

[0287] Step 7:

[0288] Generative AI model (on server):

[0289] Analysis is performed based on the text and emotional information of the primary speech recognition results, as well as contextual information.

[0290] Identify and correct inaccuracies or nonsensical parts of the text.

[0291] For example, in the sentence "Today's meeting is with Sanji," "Sanji" is changed to "three o'clock" based on context and emotional information.

[0292] Step 8:

[0293] Server: Formats the completed and corrected final text and prepares it as the final text. It also converts it, including emotion information, into a format (e.g., a JSON file) to send to the user's device.

[0294] Step 9:

[0295] Server: Sends the final text and emotion information to the user's device, often via HTTPS.

[0296] Step 10:

[0297] Device: Analyzes the data received from the server, extracts the final text and emotion information, and processes the extracted text and emotion information to display them in the appropriate UI components.

[0298] Step 11:

[0299] Terminal: The final text and the recognized emotion information are displayed to the user, who can then take further action based on the displayed information.

[0300] Specific examples

[0301] Example 1:

[0302] User: Says, "Let's put this on hold."

[0303] Terminal: Captures audio and sends digital audio data to the server.

[0304] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[0305] Server: The emotion engine analyzes the voice data and recognizes the user's calm emotion.

[0306] Server: The generative AI model modifies "fencing" to "pending" based on the primary text and sentiment information.

[0307] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[0308] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[0309] Example 2:

[0310] User: "The meeting starts at 3 o'clock today," says angrily.

[0311] Terminal: Captures audio and sends digital audio data to the server.

[0312] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[0313] Server: The emotion engine analyzes the voice data and recognizes the user's anger emotion.

[0314] Server: The generative AI model modifies "Sanji" to "Miji" based on the primary text and emotional information.

[0315] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[0316] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[0317] In this way, the present invention further improves the accuracy and naturalness of speech recognition by incorporating emotion information, allowing users to take more appropriate actions based on the emotion recognized by the system.

[0318] Example 2

[0319] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0320] Conventional speech recognition systems often misrecognize users' speech input when converting it into text, resulting in inaccurate text. Furthermore, because they do not take the user's emotions into account, they have poor contextual understanding, making it difficult to achieve natural conversations. Therefore, there is a need to improve the accuracy and naturalness of speech recognition.

[0321] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0322] In this invention, the server includes means for acquiring voice input from a user, speech recognition engine means for primarily converting the acquired voice input into text, emotion engine means for recognizing the user's emotion based on the context surrounding the primarily converted text and the speech data, generative AI model means for analyzing the context based on the primarily converted text and the recognized emotion information and for supplementing and correcting the text, and means for transmitting the supplemented and corrected final text and the recognized emotion information to a user terminal, thereby enabling improvements in the accuracy and naturalness of speech recognition.

[0323] A "means for obtaining audio input" is a device or method that captures audio produced by a user.

[0324] The "voice recognition engine means" is a device or system that analyzes acquired voice data and converts it primarily into text.

[0325] The "emotion engine means" is a device or system that analyzes voice data and identifies the user's emotion from the tone, tempo, etc. of the voice.

[0326] The "generative AI model means" is an artificial intelligence model that analyzes the context based on the primary converted text and emotional information, and complements and corrects inaccurate parts.

[0327] "Completion and correction methods" are methods for correcting errors in text based on context and sentiment information.

[0328] The "means for transmitting the final text to the user terminal" refers to a device or system that transmits the completed text and emotion information to the user's device.

[0329] This invention is a system that acquires user voice input and performs highly accurate text conversion and emotion recognition based on it. This system aims to solve the problems of conventional voice recognition systems and improve the accuracy and naturalness of voice recognition by combining a voice recognition engine, an emotion recognition engine, and a generative AI model.

[0330] Hardware and Software Configuration

[0331] 1. A way to capture user voice input

[0332] Device: The user speaks into the device (PC, smartphone, tablet, etc.). The device is equipped with a high-sensitivity microphone (e.g., Shure MV5) to capture the voice with high accuracy. The device uses a digital signal processing chip to convert the analog voice signal into a digital signal.

[0333] 2. Speech Recognition Engine Means

[0334] Server: The voice data is sent to the server via the Internet, and the server uses a voice recognition engine (e.g., a general voice recognition API) to convert the voice into text.

[0335] 3. Emotion Recognition Engine Means

[0336] Server: Input the voice data into an emotion recognition engine (e.g., a general emotion analysis API) to analyze the tone, speed, and tempo of the voice to identify the user's emotion. For example, generate emotion labels such as "anger," "joy," or "sadness."

[0337] 4. Generative AI Model Means

[0338] Server: The primary text obtained from the speech recognition engine and the emotion information obtained from the emotion recognition engine are input into a generative AI model (e.g., a general GPT model). The generative AI model analyzes the primary text based on the context and emotion information, and corrects and complements inaccurate parts.

[0339] 5. Means of sending the final text to the user terminal

[0340] Server: Sends the final text and emotion information corrected by the generative AI model to the user device, which then displays the received text and emotion information to the user.

[0341] Specific examples

[0342] Example 1:

[0343] User: Says, "Let's put this on hold."

[0344] Device: Captures audio and converts it into digital audio data using a highly sensitive microphone, then transmits the digital audio data to a server over the Internet.

[0345] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[0346] Server: The emotion engine analyzes the voice data and recognizes the user's calmness.

[0347] Server: The generative AI model analyzes the primary text and sentiment information and modifies "fencing" to "pending."

[0348] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[0349] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[0350] Example 2:

[0351] User: "The meeting starts at 3 o'clock today," says angrily.

[0352] Device: Captures audio and converts it into digital audio data using a highly sensitive microphone, then transmits the digital audio data to a server over the Internet.

[0353] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[0354] Server: The emotion engine analyzes the voice data and recognizes the user's anger.

[0355] Server: The generative AI model analyzes the primary text and emotional information and corrects "Sanji" to "Miji."

[0356] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[0357] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[0358] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0359] Step 1:

[0360] Acquiring voice input

[0361] User: The user speaks into the device.

[0362] Input: User's voice.

[0363] Output: The audio signal captured on the device.

[0364] Terminal: Uses a sensitive microphone to capture audio and converts analog audio signals into digital signals.

[0365] Data processing: Converting analog audio signals into digital format.

[0366] What it does: A digital signal processing chip converts analog audio signals into digital signals.

[0367] Step 2:

[0368] Data transmission

[0369] Terminal: Digitized voice data is stored in temporary storage and transmitted to the server using a security protocol.

[0370] Input: Digital audio data.

[0371] Output: Digital audio data sent to the server.

[0372] Specific operation: Digital audio data is transmitted using TLS (Transport Layer Security).

[0373] Step 3:

[0374] Primary Speech Recognition

[0375] Server: Analyzes the received audio data.

[0376] Input: Digital audio data.

[0377] Output: Primary speech recognition results (text format).

[0378] Server: Uses a speech recognition engine (e.g., a general speech recognition API) to convert speech to text.

[0379] Data processing: Converting voice data into text.

[0380] What it does: A speech recognition algorithm analyzes the audio data and converts it into text.

[0381] Step 4:

[0382] emotion recognition

[0383] Server: Input the voice data into an emotion recognition engine (e.g., a common emotion analysis API) to identify the user's emotion.

[0384] Input: Digital audio data.

[0385] Output: Emotion information (e.g., anger, joy, sadness).

[0386] Data calculations: Analyze the tone, speed, and tempo of the speech and generate emotion labels.

[0387] Specific operation: The emotion recognition engine extracts voice features and determines the emotion.

[0388] Step 5:

[0389] Contextual understanding and completion

[0390] Server: Inputs the primary speech recognition results and emotion information into the generative AI model, analyzes the context, and identifies inaccuracies.

[0391] Input: Primary speech recognition results (text) and emotion information.

[0392] Output: The final corrected text.

[0393] Generative AI models: Completing and correcting inaccuracies in the primary text based on context and sentiment information.

[0394] Data arithmetic: Contextual analysis and correction.

[0395] How it works: The generative AI model corrects errors based on the context and sentiment of the text, correcting them to make them more appropriate.

[0396] Step 6:

[0397] Generate and send the final result

[0398] Server: Formats the completed and corrected final text and sends it to the user's terminal.

[0399] Input: Final corrected text and sentiment information.

[0400] Output: The final text and sentiment information sent to the user device.

[0401] Terminal: Receives the final text and emotion information and displays it to the user.

[0402] Data computation: Representing text and emotional information.

[0403] Specific operation: Converts the received content into a display format and displays it on the screen.

[0404] (Application example 2)

[0405] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0406] Conventional speech recognition systems simply convert speech to text without considering the user's emotions, which often results in inaccurate conversion and insufficient understanding of context. Furthermore, the inability to respond appropriately based on emotions has led to problems with lower satisfaction, especially when communicating with users on devices such as self-driving vehicles.

[0407] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0408] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for converting the acquired voice input into text, an emotion recognition engine means for recognizing the user's emotion from the acquired voice data, and a generative AI model means for analyzing the context before and after the converted text and supplementing and correcting it. This not only improves the accuracy of voice recognition, but also enables a more natural response that understands the context and includes the user's emotion.

[0409] "Voice input" refers to voice data uttered by a user through a microphone.

[0410] A "speech recognition engine" is software and hardware used to convert captured voice input into text form.

[0411] A "generative AI model" is an artificial intelligence model that analyzes the context of text and completes or corrects it.

[0412] An "emotion recognition engine" is software and hardware for extracting and recognizing a user's emotions from voice input.

[0413] The "context" refers to the content immediately before and after the speech input in the text data.

[0414] "Completion and correction" refers to changing inaccurate parts to appropriate content and adding missing information.

[0415] "Final text" refers to text data that has been primarily converted by a speech recognition engine, and then supplemented and corrected through an emotion recognition engine and a generative AI model.

[0416] A "user terminal" is a terminal device that a user uses to perform operations and display information.

[0417] To realize this invention, we use a system that combines a speech recognition engine, an emotion recognition engine, and a generative AI model. This enables highly accurate and natural conversion of user voice input into text. This system includes the following specific procedures, hardware, and software.

[0418] The server first receives voice input from the user. The voice recognition engine converts this voice input into text, which is then temporarily saved. The voice data is then input into the emotion recognition engine, which analyzes the user's emotions from the voice data and passes them to the generative AI model.

[0419] The generative AI model analyzes the converted text and its context, as well as the emotional information recognized by the emotion recognition engine. This analysis identifies inaccuracies in the converted text and corrects them appropriately. The final text and emotional information are sent to the user's device and displayed.

[0420] The hardware used includes a microphone for voice capture, a user terminal, and a server. Software includes a speech recognition engine, an emotion recognition engine, and a generative AI model. For example, the speech recognition engine can use the Python speech_recognition library, and the emotion recognition engine and generative AI model can use models connected via a REST API.

[0421] As a concrete example, consider the case where a user says, "I want to go to Shinjuku, but not right now." First, voice data is captured by a microphone and sent to a server. A speech recognition engine converts this voice data into text, "I want to go to Shinjuku, but not right now." Next, an emotion recognition engine recognizes the user's calmness from the voice. A generative AI model analyzes this text and emotional information to generate appropriate navigation advice. In this way, the system can provide a more natural and appropriate response.

[0422] An example prompt for a generative AI model is:

[0423] "The user says, 'I want to go to Shinjuku, but not right now.' The user is emotionally calm. Use this information to correct inaccuracies and generate appropriate navigation advice that fits the context."

[0424] This system improves the accuracy and naturalness of speech recognition, providing a more useful and reliable user experience.

[0425] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0426] Step 1:

[0427] Users speak into a microphone inside the autonomous vehicle, which captures the voice data, converts it into a digital format, and transmits it to a server.

[0428] Input: Voice input from the user

[0429] Output: Digital audio data sent to the server

[0430] Specific operation: When a user says, "I'd like to go to Shinjuku, but not right now," the microphone captures the voice, digitizes the voice data, and sends it to the server.

[0431] Step 2:

[0432] The server passes the received digital voice data to a speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[0433] Input: Digital audio data

[0434] Output: The converted text

[0435] Specific operation: The speech recognition engine converts the text into "I would like to go to Shinjuku, but not right now."

[0436] Step 3:

[0437] The server temporarily stores the converted text, which is then used for later analysis.

[0438] Input: The primary converted text

[0439] Output: Saved text data

[0440] Specific operation: The server temporarily stores the text data "I want to go to Shinjuku, but not right now" in storage.

[0441] Step 4:

[0442] The server inputs the voice data into an emotion recognition engine, which analyzes the user's emotions from the voice and passes the emotional information to a generative AI model.

[0443] Input: Digital audio data

[0444] Output: Recognized emotion information

[0445] Specific operation: The emotion recognition engine recognizes the user's calm emotions and provides that information to the generative AI model.

[0446] Step 5:

[0447] The server inputs the converted text and the recognized emotion information into a generative AI model, which analyzes the text based on the context and emotion information, identifying inaccurate parts and correcting them.

[0448] Input: Primary converted text, recognized emotion information

[0449] Output: Final completed and corrected text

[0450] Specific operation: The generative AI model analyzes the sentence, "I want to go to Shinjuku, but not right now," and complements and corrects it to suggest appropriate navigation.

[0451] Step 6:

[0452] The server sends the completed and corrected final text and emotion information to the user's device.

[0453] Input: Completion / correction final text, sentiment information

[0454] Output: Final text and emotion information sent to the user's device

[0455] Specific operation: The server sends the final text "I want to go to Shinjuku, but not right now" and the emotional information "calm" to the user terminal, which then displays it.

[0456] This series of steps improves the accuracy and naturalness of speech recognition, resulting in a system that can respond appropriately to the user's emotions.

[0457] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0458] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0459] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0460] [Second embodiment]

[0461] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0462] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0463] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0464] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0465] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0466] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0467] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0468] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0469] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0470] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0471] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0472] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0473] This invention relates to a system and method for improving the accuracy of a speech recognition system. This system receives speech input from a user, converts it into text, and then uses a generative AI model to complement and correct inaccuracies based on the context surrounding the text. The system's program and processing are described in detail below.

[0474] Program processing

[0475] 1. Acquiring voice input:

[0476] Terminal: The user speaks into the terminal, which uses a microphone to capture the voice and stores the data digitally.

[0477] Device: Once the audio capture is complete, the digital audio data is sent over the internet to a server.

[0478] 2. Primary speech recognition:

[0479] Server: Passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[0480] Server: Temporarily stores the converted text. This becomes the primary speech recognition result. This result is further supplemented and corrected in subsequent processing steps.

[0481] 3. Contextual understanding and completion:

[0482] Server: Inputs the primary speech recognition results into a generative AI model, which analyzes the context of the text and identifies inaccuracies or inaudible parts.

[0483] Server: The generative AI model uses contextual analysis to correct inaccuracies in the primary speech recognition results. For example, "The meeting is from Sanji" is corrected to "The meeting starts at 3 o'clock."

[0484] 4. Generate and send the final result:

[0485] Server: Formats the completed and corrected text and prepares it as the final text.

[0486] Server: Sends the prepared final text to the user's terminal.

[0487] On the device, the final text received is displayed to the user, allowing the user to see accurate and natural-looking results of their voice input.

[0488] Specific examples

[0489] Example 1:

[0490] User: Says, "Let's put this on hold."

[0491] Terminal: Captures audio and sends digital audio data to the server.

[0492] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[0493] Server: A generative AI model analyzes this primary text in context and modifies "fencing" to "pending."

[0494] Server: Sends the final text "Let's put this on hold" to the user terminal.

[0495] Terminal: Show the user "Let's put this on hold."

[0496] Example 2:

[0497] User: Say, "Today's meeting starts at 3:00."

[0498] Terminal: Captures audio and sends digital audio data to the server.

[0499] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[0500] Server: The generative AI model analyzes this primary text taking into account the context and corrects "Sanji" to "Miji".

[0501] Server: Sends the final text "Today's meeting starts at 3:00" to the user terminal.

[0502] Terminal: Display to the user "Today's meeting starts at 3:00."

[0503] In this way, the present invention significantly improves the accuracy of speech recognition and provides users with a more natural and accurate speech input experience. The system of the present invention enables users to achieve stress-free communication.

[0504] The processing flow will be explained below.

[0505] Step 1:

[0506] Device: Captures the user's voice. The user speaks into the device, and the device's microphone converts the voice into a digital format and stores it in an internal buffer.

[0507] Step 2:

[0508] Terminal: After the audio capture is complete, the digital audio data is sent to the server via the Internet using HTTP requests or WebSocket communication.

[0509] Step 3:

[0510] Server: Passes the received voice data to a voice recognition engine (e.g., a commercial voice recognition API), which analyzes the input voice data and converts it into temporary text.

[0511] Step 4:

[0512] Server: Receives the text output from the speech recognition engine and stores it in temporary memory (e.g., a database or cache memory). The text at this stage is the primary speech recognition result.

[0513] Step 5:

[0514] Server: Passes the primary speech recognition results to a generative AI model for context analysis. The generative AI model (e.g., a model based on natural language processing) analyzes the input text and its context to identify inaccurate or unclear parts.

[0515] Step 6:

[0516] Generative AI model (on server):

[0517] The AI ​​model corrects any identified inaccuracies based on the primary speech recognition results.

[0518] For example, the incorrectly recognized part "Sanji" is corrected to "three o'clock."

[0519] Step 7:

[0520] Server: Takes the corrected text data, reformats it as final text, and passes it on to other applications or services as needed.

[0521] Step 8:

[0522] Server: Sends the final text to the user's device, using JSON or another data exchange format.

[0523] Step 9:

[0524] Terminal: Parses the data received from the server and extracts the final text. An appropriate UI component (e.g., a text view or popup) is used to display the extracted text to the user.

[0525] Step 10:

[0526] Terminal: Displays the final text to the user, who can review it and take further action if necessary.

[0527] Example 1

[0528] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0529] Current speech recognition systems are prone to misrecognition during the initial conversion of speech input to text, resulting in inaccurate recognition of the user's intent, especially in noisy environments and complex contexts.

[0530] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0531] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for primarily converting voice into text, a generative AI model means for analyzing the context of the primarily converted text and supplementing and correcting it, and a means for transmitting the supplemented and corrected final text to a user terminal, thereby improving the accuracy of voice recognition, accurately reflecting the user's intention, and enabling a more natural and accurate voice input experience.

[0532] "Voice input" is a series of voice data uttered by a user into a terminal.

[0533] "Speech recognition engine means" means software or hardware that utilizes machine learning algorithms to analyze voice input and convert digital voice data into text.

[0534] The "generative AI model means" is an artificial intelligence model that uses natural language processing technology to analyze the context of the primarily converted text and complement and correct inaccurate parts.

[0535] A "user terminal" is a device that accepts voice input and displays the resulting text. Examples include smartphones, tablets, and PCs.

[0536] "Primary speech recognition result" is the text data initially generated from speech input by the speech recognition engine means.

[0537] "Completion and correction" is the process by which the generative AI model detects inaccuracies in the primary speech recognition results and corrects them based on appropriate context.

[0538] "Final text" refers to text data that has been completed through supplementation and correction by the generative AI model means.

[0539] The present invention relates to a system and method for improving the accuracy of a voice input system. The system receives voice input from a user, converts it to text, and then uses a generative AI model to complement and correct inaccuracies based on the context of the text. Specific embodiments of the system are described in detail below.

[0540] First, the user speaks into their device (e.g., smartphone, tablet, PC, etc.). The device has a built-in microphone that captures the user's voice and converts it into digital voice data. This digital voice data is sent to a server via the Internet.

[0541] The server uses a speech recognition engine to convert the received digital voice data into text. This speech recognition engine can be Google Cloud Speech-to-Text or DeepSpeech. The speech recognition engine analyzes the voice data and generates a primary speech recognition result. This primary speech recognition result is text data in the form of, for example, "The meeting is from Sanji."

[0542] The server then inputs the primary speech recognition results into a generative AI model. This generative AI model uses natural language processing technologies such as GPT-4 and BERT. The generative AI model analyzes the context to identify and correct inaccuracies in the primary speech recognition results. For example, it might correct "Sanji" to "Miji." Through this correction and completion process, the generative AI model generates the final text.

[0543] The final text after completion and correction is sent from the server to the user's device, which has the function of displaying the received final text, allowing the user to check the results of their voice input accurately and naturally.

[0544] A specific example is given below.

[0545] Specific examples

[0546] Example 1

[0547] User: Says, "Let's put this on hold."

[0548] Terminal: Captures audio and sends digital audio data to the server.

[0549] Server: A speech recognition engine (e.g., Google Cloud Speech-to-Text) converts the speech data into the initial text "Let's fence this."

[0550] Server: A generative AI model (e.g., GPT-4) analyzes the primary text based on context and modifies "fencing" to "pending."

[0551] Server: Sends the final text "Let's put this on hold" to the user terminal.

[0552] Terminal: Show the user "Let's put this on hold."

[0553] Example 2

[0554] User: Say, "Today's meeting starts at 3:00."

[0555] Terminal: Captures audio and sends digital audio data to the server.

[0556] Server: A speech recognition engine (e.g., DeepSpeech) converts the speech data into the text "Today's meeting is with Sanji."

[0557] Server: A generative AI model (e.g., BERT) analyzes the primary text based on context and modifies "Sanji" to "Miji."

[0558] Server: Sends the final text "Today's meeting starts at 3:00" to the user terminal.

[0559] Terminal: Display to the user "Today's meeting starts at 3:00."

[0560] An example of a prompt is as follows:

[0561] User Input: Let's put this on hold

[0562] Initial speech recognition result: Let's fence this one.

[0563] Completion and correction results by generative AI model: Let's put this on hold.

[0564] This system significantly improves the accuracy of voice recognition, providing users with a more natural and accurate voice input experience, enabling stress-free communication.

[0565] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0566] Step 1:

[0567] Acquiring voice input

[0568] User: The user speaks into the terminal. For example, the user says, "The meeting starts at 3 o'clock."

[0569] Device: The device's microphone captures the user's voice and converts it into digital audio data (e.g., WAV format). A microphone icon appears to indicate that recording is in progress.

[0570] Input: User's voice input.

[0571] Output: Digital audio data.

[0572] What it does: The device captures the audio data, stores it internally, and sends it to a server over the Internet (using the HTTPS protocol).

[0573] Step 2:

[0574] Primary Speech Recognition

[0575] Server: Receives digital audio data sent from the device. At this time, the audio data is temporarily stored in the server's memory.

[0576] Server: Using a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech), the voice data is converted into text, e.g., "The meeting is with Sanji."

[0577] Input: Digital audio data.

[0578] Output: Primary speech recognition result (text data).

[0579] Specific operation: The speech recognition engine starts working, analyzing and converting the voice data to generate text data. The user is notified that "Speech recognition in progress. Please wait a moment."

[0580] Step 3:

[0581] Contextual understanding and completion

[0582] Server: The primary speech recognition results are fed into a generative AI model (e.g., GPT-4 or BERT). The generative AI model analyzes the context of the text and complements or corrects inaccuracies. For example, correcting "Sanji" to "Miji."

[0583] Input: Primary speech recognition result (text data).

[0584] Output: The final completed and corrected text.

[0585] How it works: The generative AI model analyzes the context based on the primary speech recognition results and makes appropriate corrections. The progress of the analysis process is displayed in real time on the server console, and the user is informed that "text is being analyzed."

[0586] Step 4:

[0587] Generate and send the final result

[0588] Server: The completed and corrected text is formatted and prepared for final use, e.g., adding appropriate punctuation and formatting.

[0589] Server: Sends the final text to the user's device. Data is transferred securely using the HTTPS protocol.

[0590] Input: The completed and corrected final text.

[0591] Output: The final text that is displayed on the user's terminal.

[0592] Specific behavior: The server sends the final text to the user's device, and the device displays the received text to the user. A notification sound is played and a pop-up message appears saying "Processing completed. Please check the results."

[0593] (Application example 1)

[0594] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0595] Conventional speech recognition systems often fail to accurately capture user intent due to their low accuracy in converting speech to text. This problem is particularly severe in practical applications such as food delivery, where incorrectly recognized orders can lead to user dissatisfaction and a decline in service quality. Therefore, there is a need for systems that can more accurately convert speech input to text and provide context-based completion and correction.

[0596] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0597] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for converting the acquired voice input into text, a generative AI model means for analyzing the context of the converted text and completing and correcting it, a means for transmitting the completed and corrected final text to a user terminal, and a means for acquiring information using the voice input and confirming the user's order details, thereby improving the accuracy of voice recognition and enabling users to accurately place food delivery orders.

[0598] The "means for acquiring voice input from the user" is a function for acquiring information input by voice by the user as digital data using a device such as a microphone.

[0599] The "speech recognition engine means for primarily converting acquired voice input into text" is a system that executes a process of converting acquired voice data into text format using a machine learning algorithm.

[0600] "Generative AI model means for analyzing the context before and after primary converted text and completing and correcting it" is a function that uses generative AI technology to analyze the context of speech data that has been converted into text for the first time and to complete or correct inaccurate or missing parts.

[0601] "Means for sending the completed and corrected final text to the user's terminal" refers to the process of sending the text data completed and corrected by the generative AI model to the user's terminal.

[0602] "Means for obtaining information using voice input and confirming the user's order details" refers to a function that captures the order information entered by the user through voice and identifies and confirms the exact order details based on that information.

[0603] This invention is a system that accurately converts user speech input into text and enhances the user experience by complementing and correcting recognition errors, particularly in the context of food delivery orders.

[0604] System configuration and program processing

[0605] Hardware and Software

[0606] Hardware

[0607] User terminal: A smartphone or microphone is included for users to input voice.

[0608] Server: Equipped with high-performance CPU / GPU (e.g. NVIDIA A100).

[0609] software

[0610] Speech recognition engine: DeepSpeech (open source speech recognition engine)

[0611] Generative AI model: GPT-4 (generative AI model)

[0612] Food delivery application: Flutter (UI framework)

[0613] Processing flow

[0614] 1. Acquiring voice input

[0615] Users enter their food delivery order details by voice into their smartphone.

[0616] The smartphone's microphone captures the sound and generates digital audio data.

[0617] 2. Primary speech recognition

[0618] The smartphone sends the captured audio data to the server.

[0619] DeepSpeech, located on the server, converts the voice data into text.

[0620] 3. Contextual understanding and completion

[0621] The converted text is input into the GPT-4 model on the server.

[0622] The GPT-4 model analyzes the context of the text to identify inaccuracies or parts that were not heard.

[0623] The GPT-4 model complements and corrects the primary text based on the analysis results.

[0624] 4. Generate and send the final result

[0625] Format the completed and corrected text and prepare it as the final order text.

[0626] The final text is sent to the user's smartphone, and the order details are displayed.

[0627] Specific operation examples

[0628] Example 1:

[0629] User: Say "I'd like a fried chicken set and a Coke, please."

[0630] Smartphone: Captures audio and sends digital audio data to a server.

[0631] Server: DeepSpeech converts the primary text "I'd like a fried chicken sed and one curried please."

[0632] GPT-4 model: Analyzes the context and corrects "kare" to "cola" and "sed" to "set."

[0633] Server: "I'd like a fried chicken set and a bottle of coke, please" is sent to the smartphone and displayed.

[0634] Example 2:

[0635] User: Say "I'd like a Margherita pizza and orange juice, please."

[0636] Smartphone: Captures audio and sends digital audio data to a server.

[0637] Server: DeepSpeech converts the primary text "I'd like a Margherita pizza and an Olathe juice, please."

[0638] GPT-4 model: Corrects "Olathe" to "Orange Juice."

[0639] Server: "Margherita pizza and orange juice please" is sent to the smartphone and displayed.

[0640] Example of input prompt for generative AI model

[0641] Prompt statement:

[0642] Based on the context of a food delivery order, please correct the following speech recognition output to the correct order:

[0643] Initial speech recognition results: Margherita pizza and Ol'ze juice

[0644] Such systems and methods enable users to place more accurate food delivery orders by voice, improving the user experience.

[0645] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0646] Step 1:

[0647] Users speak into their smartphone to order food delivery, and the smartphone's microphone captures the voice and generates digital audio data, which is then transmitted over the Internet to a server.

[0648] Input: Audio input

[0649] Output: Digital audio data

[0650] How it works: The smartphone's microphone captures the user's voice and stores it as digital audio data, which is then sent over the internet to a server.

[0651] Step 2:

[0652] The server passes the received digital voice data to DeepSpeech, which converts the voice data into text. This primary text data is temporarily stored on the server.

[0653] Input: Digital audio data

[0654] Output: Primary text data

[0655] How it works: DeepSpeech on the server analyzes the audio data and converts it into text. The converted primary text is stored on the server.

[0656] Step 3:

[0657] The server inputs the primary text data into the GPT-4 model, which analyzes the context of the primary text to identify inaccuracies or unclear parts, and then appropriately complements and corrects the inaccuracies based on the context.

[0658] Input: Primary text data

[0659] Output: Completion and correction of text data

[0660] How it works: The GPT-4 model on the server analyzes text and context to identify inaccuracies, then provides contextual completion and correction.

[0661] Step 4:

[0662] The server formats the completed and corrected text and prepares it as the final text data, which is then sent to the user's smartphone via the Internet.

[0663] Input: Completed and corrected text data

[0664] Output: Final text data

[0665] Operation: The text data is formatted on the server to generate the final text data, which is then sent to the user's smartphone via the Internet.

[0666] Step 5:

[0667] The user's smartphone receives the final text data and displays it to the user in an appropriate format, allowing the user to confirm the exact food delivery order details.

[0668] Input: Final text data

[0669] Output: The order displayed to the user

[0670] Operation: The smartphone analyzes the final text data received from the server and displays it on the screen in a format that the user can read.

[0671] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0672] The present invention relates to a system and method for improving the accuracy of speech recognition systems, particularly by combining an emotion engine that recognizes user emotions to achieve more natural and context-aware speech recognition. This system acquires speech input from a user, converts it into text, and then uses a generative AI model to analyze the context surrounding the text, complementing and correcting inaccuracies, while also incorporating the user's emotional information recognized by the emotion engine.

[0673] Program processing

[0674] 1. Acquiring voice input:

[0675] Terminal: The user speaks into the terminal, which uses a microphone to capture the voice and stores the data digitally.

[0676] Device: Once the audio capture is complete, the digital audio data is sent over the internet to a server.

[0677] 2. Primary speech recognition:

[0678] Server: Passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[0679] Server: Temporarily stores the converted text. This is the primary speech recognition result. This result is the first step in the completion and correction process.

[0680] 3. Emotion Recognition:

[0681] Server: Inputs the voice data into the emotion engine, which analyzes the user's tone and speaking style.

[0682] Emotion engine (on the server):

[0683] It recognizes the user's emotions (e.g., anger, joy, sadness) from voice data and passes that emotional information to a generative AI model.

[0684] 4. Contextual understanding and completion:

[0685] Server: Passes the primary speech recognition results and recognized emotion information to the generative AI model for context analysis. The generative AI model analyzes the input text, its context, and the emotion information to identify inaccuracies.

[0686] Server: The generative AI model complements and corrects inaccurate parts of the primary speech recognition results based on contextual analysis and emotional information. For example, "The meeting is from Sanji" is corrected to "The meeting starts at 3 o'clock."

[0687] 5. Generate and send the final result:

[0688] Server: Formats the completed and corrected text and prepares it as the final text.

[0689] Server: Sends the prepared final text to the user's device, along with the recognized emotion information.

[0690] Terminal: Displays the final text and emotion information received to the user.

[0691] Specific examples

[0692] Example 1:

[0693] User: Says, "Let's put this on hold."

[0694] Terminal: Captures audio and sends digital audio data to the server.

[0695] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[0696] Server: The emotion engine analyzes the voice data and recognizes the user's calmness.

[0697] Server: The generative AI model analyzes the primary text and sentiment information and modifies "fencing" to "pending."

[0698] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[0699] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[0700] Example 2:

[0701] User: "The meeting starts at 3 o'clock today," says angrily.

[0702] Terminal: Captures audio and sends digital audio data to the server.

[0703] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[0704] Server: The emotion engine analyzes the voice data and recognizes the user's anger.

[0705] Server: The generative AI model analyzes the primary text and emotional information and corrects "Sanji" to "Miji."

[0706] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[0707] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[0708] In this way, the present invention further improves the accuracy and naturalness of speech recognition by incorporating emotion information, allowing users to take more appropriate action based on the emotion recognized by the system.

[0709] The processing flow will be explained below.

[0710] Step 1:

[0711] Device: The user speaks into the device, the device's microphone captures the audio, converts it into a digital format, and stores it in an internal buffer.

[0712] Step 2:

[0713] Device: After the audio capture is complete, the digital audio data is sent to the server over the internet using an HTTP POST request or WebSocket communication.

[0714] Step 3:

[0715] Server: The server passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[0716] Step 4:

[0717] Server: Stores the primary text obtained from the speech recognition engine in temporary memory. This is the primary speech recognition result, which is supplemented and corrected in subsequent processing steps.

[0718] Step 5:

[0719] Server: Inputs the voice data into the emotion engine, which analyzes the user's tone and speaking style. Based on this analysis, the user's emotion (e.g., anger, joy, sadness) is recognized.

[0720] Step 6:

[0721] Server: Obtains the emotion information obtained from the emotion engine and passes it to the generative AI model, which receives the primary speech recognition result, context, and emotion information as input.

[0722] Step 7:

[0723] Generative AI model (on server):

[0724] Analysis is performed based on the text and emotional information of the primary speech recognition results, as well as contextual information.

[0725] Identify and correct inaccuracies or nonsensical parts of the text.

[0726] For example, in the sentence "Today's meeting is with Sanji," "Sanji" is changed to "three o'clock" based on context and emotional information.

[0727] Step 8:

[0728] Server: Formats the completed and corrected final text and prepares it as the final text. It also converts it, including emotion information, into a format (e.g., a JSON file) to send to the user's device.

[0729] Step 9:

[0730] Server: Sends the final text and emotion information to the user's device, often via HTTPS.

[0731] Step 10:

[0732] Device: Analyzes the data received from the server, extracts the final text and emotion information, and processes the extracted text and emotion information to display them in the appropriate UI components.

[0733] Step 11:

[0734] Terminal: The final text and the recognized emotion information are displayed to the user, who can then take further action based on the displayed information.

[0735] Specific examples

[0736] Example 1:

[0737] User: Says, "Let's put this on hold."

[0738] Terminal: Captures audio and sends digital audio data to the server.

[0739] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[0740] Server: The emotion engine analyzes the voice data and recognizes the user's calm emotion.

[0741] Server: The generative AI model modifies "fencing" to "pending" based on the primary text and sentiment information.

[0742] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[0743] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[0744] Example 2:

[0745] User: "The meeting starts at 3 o'clock today," says angrily.

[0746] Terminal: Captures audio and sends digital audio data to the server.

[0747] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[0748] Server: The emotion engine analyzes the voice data and recognizes the user's anger emotion.

[0749] Server: The generative AI model modifies "Sanji" to "Miji" based on the primary text and emotional information.

[0750] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[0751] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[0752] In this way, the present invention further improves the accuracy and naturalness of speech recognition by incorporating emotion information, allowing users to take more appropriate actions based on the emotion recognized by the system.

[0753] Example 2

[0754] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0755] Conventional speech recognition systems often misrecognize users' speech input when converting it into text, resulting in inaccurate text. Furthermore, because they do not take the user's emotions into account, they have poor contextual understanding, making it difficult to achieve natural conversations. Therefore, there is a need to improve the accuracy and naturalness of speech recognition.

[0756] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0757] In this invention, the server includes means for acquiring voice input from a user, speech recognition engine means for primarily converting the acquired voice input into text, emotion engine means for recognizing the user's emotion based on the context surrounding the primarily converted text and the speech data, generative AI model means for analyzing the context based on the primarily converted text and the recognized emotion information and for supplementing and correcting the text, and means for transmitting the supplemented and corrected final text and the recognized emotion information to a user terminal, thereby enabling improvements in the accuracy and naturalness of speech recognition.

[0758] A "means for obtaining audio input" is a device or method that captures audio produced by a user.

[0759] The "voice recognition engine means" is a device or system that analyzes acquired voice data and converts it primarily into text.

[0760] The "emotion engine means" is a device or system that analyzes voice data and identifies the user's emotion from the tone, tempo, etc. of the voice.

[0761] The "generative AI model means" is an artificial intelligence model that analyzes the context based on the primary converted text and emotional information, and complements and corrects inaccurate parts.

[0762] "Completion and correction methods" are methods for correcting errors in text based on context and sentiment information.

[0763] The "means for transmitting the final text to the user terminal" refers to a device or system that transmits the completed text and emotion information to the user's device.

[0764] This invention is a system that acquires user voice input and performs highly accurate text conversion and emotion recognition based on it. This system aims to solve the problems of conventional voice recognition systems and improve the accuracy and naturalness of voice recognition by combining a voice recognition engine, an emotion recognition engine, and a generative AI model.

[0765] Hardware and Software Configuration

[0766] 1. A way to capture user voice input

[0767] Device: The user speaks into the device (PC, smartphone, tablet, etc.). The device is equipped with a high-sensitivity microphone (e.g., Shure MV5) to capture the voice with high accuracy. The device uses a digital signal processing chip to convert the analog voice signal into a digital signal.

[0768] 2. Speech Recognition Engine Means

[0769] Server: The voice data is sent to the server via the Internet, and the server uses a voice recognition engine (e.g., a general voice recognition API) to convert the voice into text.

[0770] 3. Emotion Recognition Engine Means

[0771] Server: Input the voice data into an emotion recognition engine (e.g., a general emotion analysis API) to analyze the tone, speed, and tempo of the voice to identify the user's emotion. For example, generate emotion labels such as "anger," "joy," or "sadness."

[0772] 4. Generative AI Model Means

[0773] Server: The primary text obtained from the speech recognition engine and the emotion information obtained from the emotion recognition engine are input into a generative AI model (e.g., a general GPT model). The generative AI model analyzes the primary text based on the context and emotion information, and corrects and complements inaccurate parts.

[0774] 5. Means of sending the final text to the user terminal

[0775] Server: Sends the final text and emotion information corrected by the generative AI model to the user device, which then displays the received text and emotion information to the user.

[0776] Specific examples

[0777] Example 1:

[0778] User: Says, "Let's put this on hold."

[0779] Device: Captures audio and converts it into digital audio data using a highly sensitive microphone, then transmits the digital audio data to a server over the Internet.

[0780] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[0781] Server: The emotion engine analyzes the voice data and recognizes the user's calmness.

[0782] Server: The generative AI model analyzes the primary text and sentiment information and modifies "fencing" to "pending."

[0783] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[0784] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[0785] Example 2:

[0786] User: "The meeting starts at 3 o'clock today," says angrily.

[0787] Device: Captures audio and converts it into digital audio data using a highly sensitive microphone, then transmits the digital audio data to a server over the Internet.

[0788] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[0789] Server: The emotion engine analyzes the voice data and recognizes the user's anger.

[0790] Server: The generative AI model analyzes the primary text and emotional information and corrects "Sanji" to "Miji."

[0791] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[0792] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[0793] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0794] Step 1:

[0795] Acquiring voice input

[0796] User: The user speaks into the device.

[0797] Input: User's voice.

[0798] Output: The audio signal captured on the device.

[0799] Terminal: Uses a sensitive microphone to capture audio and converts analog audio signals into digital signals.

[0800] Data processing: Converting analog audio signals into digital format.

[0801] What it does: A digital signal processing chip converts analog audio signals into digital signals.

[0802] Step 2:

[0803] Data transmission

[0804] Terminal: Digitized voice data is stored in temporary storage and transmitted to the server using a security protocol.

[0805] Input: Digital audio data.

[0806] Output: Digital audio data sent to the server.

[0807] Specific operation: Digital audio data is transmitted using TLS (Transport Layer Security).

[0808] Step 3:

[0809] Primary Speech Recognition

[0810] Server: Analyzes the received audio data.

[0811] Input: Digital audio data.

[0812] Output: Primary speech recognition results (text format).

[0813] Server: Uses a speech recognition engine (e.g., a general speech recognition API) to convert speech to text.

[0814] Data processing: Converting voice data into text.

[0815] What it does: A speech recognition algorithm analyzes the audio data and converts it into text.

[0816] Step 4:

[0817] emotion recognition

[0818] Server: Input the voice data into an emotion recognition engine (e.g., a common emotion analysis API) to identify the user's emotion.

[0819] Input: Digital audio data.

[0820] Output: Emotion information (e.g., anger, joy, sadness).

[0821] Data calculations: Analyze the tone, speed, and tempo of the speech and generate emotion labels.

[0822] Specific operation: The emotion recognition engine extracts voice features and determines the emotion.

[0823] Step 5:

[0824] Contextual understanding and completion

[0825] Server: Inputs the primary speech recognition results and emotion information into the generative AI model, analyzes the context, and identifies inaccuracies.

[0826] Input: Primary speech recognition results (text) and emotion information.

[0827] Output: The final corrected text.

[0828] Generative AI models: Completing and correcting inaccuracies in the primary text based on context and sentiment information.

[0829] Data arithmetic: Contextual analysis and correction.

[0830] How it works: The generative AI model corrects errors based on the context and sentiment of the text, correcting them to make them more appropriate.

[0831] Step 6:

[0832] Generate and send the final result

[0833] Server: Formats the completed and corrected final text and sends it to the user's terminal.

[0834] Input: Final corrected text and sentiment information.

[0835] Output: The final text and sentiment information sent to the user device.

[0836] Terminal: Receives the final text and emotion information and displays it to the user.

[0837] Data computation: Representing text and emotional information.

[0838] Specific operation: Converts the received content into a display format and displays it on the screen.

[0839] (Application example 2)

[0840] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0841] Conventional speech recognition systems simply convert speech to text without considering the user's emotions, which often results in inaccurate conversion and insufficient understanding of context. Furthermore, the inability to respond appropriately based on emotions has led to problems with lower satisfaction, especially when communicating with users on devices such as self-driving vehicles.

[0842] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0843] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for converting the acquired voice input into text, an emotion recognition engine means for recognizing the user's emotion from the acquired voice data, and a generative AI model means for analyzing the context before and after the converted text and supplementing and correcting it. This not only improves the accuracy of voice recognition, but also enables a more natural response that understands the context and includes the user's emotion.

[0844] "Voice input" refers to voice data uttered by a user through a microphone.

[0845] A "speech recognition engine" is software and hardware used to convert captured voice input into text form.

[0846] A "generative AI model" is an artificial intelligence model that analyzes the context of text and completes or corrects it.

[0847] An "emotion recognition engine" is software and hardware for extracting and recognizing a user's emotions from voice input.

[0848] The "context" refers to the content immediately before and after the speech input in the text data.

[0849] "Completion and correction" refers to changing inaccurate parts to appropriate content and adding missing information.

[0850] "Final text" refers to text data that has been primarily converted by a speech recognition engine, and then supplemented and corrected through an emotion recognition engine and a generative AI model.

[0851] A "user terminal" is a terminal device that a user uses to perform operations and display information.

[0852] To realize this invention, we use a system that combines a speech recognition engine, an emotion recognition engine, and a generative AI model. This enables highly accurate and natural conversion of user voice input into text. This system includes the following specific procedures, hardware, and software.

[0853] The server first receives voice input from the user. The voice recognition engine converts this voice input into text, which is then temporarily saved. The voice data is then input into the emotion recognition engine, which analyzes the user's emotions from the voice data and passes them to the generative AI model.

[0854] The generative AI model analyzes the converted text and its context, as well as the emotional information recognized by the emotion recognition engine. This analysis identifies inaccuracies in the converted text and corrects them appropriately. The final text and emotional information are sent to the user's device and displayed.

[0855] The hardware used includes a microphone for voice capture, a user terminal, and a server. Software includes a speech recognition engine, an emotion recognition engine, and a generative AI model. For example, the speech recognition engine can use the Python speech_recognition library, and the emotion recognition engine and generative AI model can use models connected via a REST API.

[0856] As a concrete example, consider the case where a user says, "I want to go to Shinjuku, but not right now." First, voice data is captured by a microphone and sent to a server. A speech recognition engine converts this voice data into text, "I want to go to Shinjuku, but not right now." Next, an emotion recognition engine recognizes the user's calmness from the voice. A generative AI model analyzes this text and emotional information to generate appropriate navigation advice. In this way, the system can provide a more natural and appropriate response.

[0857] An example prompt for a generative AI model is:

[0858] "The user says, 'I want to go to Shinjuku, but not right now.' The user is emotionally calm. Use this information to correct inaccuracies and generate appropriate navigation advice that fits the context."

[0859] This system improves the accuracy and naturalness of speech recognition, providing a more useful and reliable user experience.

[0860] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0861] Step 1:

[0862] Users speak into a microphone inside the autonomous vehicle, which captures the voice data, converts it into a digital format, and transmits it to a server.

[0863] Input: Voice input from the user

[0864] Output: Digital audio data sent to the server

[0865] Specific operation: When a user says, "I'd like to go to Shinjuku, but not right now," the microphone captures the voice, digitizes the voice data, and sends it to the server.

[0866] Step 2:

[0867] The server passes the received digital voice data to a speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[0868] Input: Digital audio data

[0869] Output: The converted text

[0870] Specific operation: The speech recognition engine converts the text into "I would like to go to Shinjuku, but not right now."

[0871] Step 3:

[0872] The server temporarily stores the converted text, which is then used for later analysis.

[0873] Input: The primary converted text

[0874] Output: Saved text data

[0875] Specific operation: The server temporarily stores the text data "I want to go to Shinjuku, but not right now" in storage.

[0876] Step 4:

[0877] The server inputs the voice data into an emotion recognition engine, which analyzes the user's emotions from the voice and passes the emotional information to a generative AI model.

[0878] Input: Digital audio data

[0879] Output: Recognized emotion information

[0880] Specific operation: The emotion recognition engine recognizes the user's calm emotions and provides that information to the generative AI model.

[0881] Step 5:

[0882] The server inputs the converted text and the recognized emotion information into a generative AI model, which analyzes the text based on the context and emotion information, identifying inaccurate parts and correcting them.

[0883] Input: Primary converted text, recognized emotion information

[0884] Output: Final completed and corrected text

[0885] Specific operation: The generative AI model analyzes the sentence, "I want to go to Shinjuku, but not right now," and complements and corrects it to suggest appropriate navigation.

[0886] Step 6:

[0887] The server sends the completed and corrected final text and emotion information to the user's device.

[0888] Input: Completion / correction final text, sentiment information

[0889] Output: Final text and emotion information sent to the user's device

[0890] Specific operation: The server sends the final text "I want to go to Shinjuku, but not right now" and the emotional information "calm" to the user terminal, which then displays it.

[0891] This series of steps improves the accuracy and naturalness of speech recognition, resulting in a system that can respond appropriately to the user's emotions.

[0892] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0893] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0894] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0895] [Third embodiment]

[0896] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0897] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0898] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0899] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0900] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0901] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0902] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0903] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0904] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0905] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0906] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0907] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0908] This invention relates to a system and method for improving the accuracy of a speech recognition system. This system receives speech input from a user, converts it into text, and then uses a generative AI model to complement and correct inaccuracies based on the context surrounding the text. The system's program and processing are described in detail below.

[0909] Program processing

[0910] 1. Acquiring voice input:

[0911] Terminal: The user speaks into the terminal, which uses a microphone to capture the voice and stores the data digitally.

[0912] Device: Once the audio capture is complete, the digital audio data is sent over the internet to a server.

[0913] 2. Primary speech recognition:

[0914] Server: Passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[0915] Server: Temporarily stores the converted text. This becomes the primary speech recognition result. This result is further supplemented and corrected in subsequent processing steps.

[0916] 3. Contextual understanding and completion:

[0917] Server: Inputs the primary speech recognition results into a generative AI model, which analyzes the context of the text and identifies inaccuracies or inaudible parts.

[0918] Server: The generative AI model uses contextual analysis to correct inaccuracies in the primary speech recognition results. For example, "The meeting is from Sanji" is corrected to "The meeting starts at 3 o'clock."

[0919] 4. Generate and send the final result:

[0920] Server: Formats the completed and corrected text and prepares it as the final text.

[0921] Server: Sends the prepared final text to the user's terminal.

[0922] On the device, the final text received is displayed to the user, allowing the user to see accurate and natural-looking results of their voice input.

[0923] Specific examples

[0924] Example 1:

[0925] User: Says, "Let's put this on hold."

[0926] Terminal: Captures audio and sends digital audio data to the server.

[0927] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[0928] Server: A generative AI model analyzes this primary text in context and modifies "fencing" to "pending."

[0929] Server: Sends the final text "Let's put this on hold" to the user terminal.

[0930] Terminal: Show the user "Let's put this on hold."

[0931] Example 2:

[0932] User: Say, "Today's meeting starts at 3:00."

[0933] Terminal: Captures audio and sends digital audio data to the server.

[0934] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[0935] Server: The generative AI model analyzes this primary text taking into account the context and corrects "Sanji" to "Miji".

[0936] Server: Sends the final text "Today's meeting starts at 3:00" to the user terminal.

[0937] Terminal: Display to the user "Today's meeting starts at 3:00."

[0938] In this way, the present invention significantly improves the accuracy of speech recognition and provides users with a more natural and accurate speech input experience. The system of the present invention enables users to achieve stress-free communication.

[0939] The processing flow will be explained below.

[0940] Step 1:

[0941] Device: Captures the user's voice. The user speaks into the device, and the device's microphone converts the voice into a digital format and stores it in an internal buffer.

[0942] Step 2:

[0943] Terminal: After the audio capture is complete, the digital audio data is sent to the server via the Internet using HTTP requests or WebSocket communication.

[0944] Step 3:

[0945] Server: Passes the received voice data to a voice recognition engine (e.g., a commercial voice recognition API), which analyzes the input voice data and converts it into temporary text.

[0946] Step 4:

[0947] Server: Receives the text output from the speech recognition engine and stores it in temporary memory (e.g., a database or cache memory). The text at this stage is the primary speech recognition result.

[0948] Step 5:

[0949] Server: Passes the primary speech recognition results to a generative AI model for context analysis. The generative AI model (e.g., a model based on natural language processing) analyzes the input text and its context to identify inaccurate or unclear parts.

[0950] Step 6:

[0951] Generative AI model (on server):

[0952] The AI ​​model corrects any identified inaccuracies based on the primary speech recognition results.

[0953] For example, the incorrectly recognized part "Sanji" is corrected to "three o'clock."

[0954] Step 7:

[0955] Server: Takes the corrected text data, reformats it as final text, and passes it on to other applications or services as needed.

[0956] Step 8:

[0957] Server: Sends the final text to the user's device, using JSON or another data exchange format.

[0958] Step 9:

[0959] Terminal: Parses the data received from the server and extracts the final text. An appropriate UI component (e.g., a text view or popup) is used to display the extracted text to the user.

[0960] Step 10:

[0961] Terminal: Displays the final text to the user, who can review it and take further action if necessary.

[0962] Example 1

[0963] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0964] Current speech recognition systems are prone to misrecognition during the initial conversion of speech input to text, resulting in inaccurate recognition of the user's intent, especially in noisy environments and complex contexts.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0966] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for primarily converting voice into text, a generative AI model means for analyzing the context of the primarily converted text and supplementing and correcting it, and a means for transmitting the supplemented and corrected final text to a user terminal, thereby improving the accuracy of voice recognition, accurately reflecting the user's intention, and enabling a more natural and accurate voice input experience.

[0967] "Voice input" is a series of voice data uttered by a user into a terminal.

[0968] "Speech recognition engine means" means software or hardware that utilizes machine learning algorithms to analyze voice input and convert digital voice data into text.

[0969] The "generative AI model means" is an artificial intelligence model that uses natural language processing technology to analyze the context of the primarily converted text and complement and correct inaccurate parts.

[0970] A "user terminal" is a device that accepts voice input and displays the resulting text. Examples include smartphones, tablets, and PCs.

[0971] "Primary speech recognition result" is the text data initially generated from speech input by the speech recognition engine means.

[0972] "Completion and correction" is the process by which the generative AI model detects inaccuracies in the primary speech recognition results and corrects them based on appropriate context.

[0973] "Final text" refers to text data that has been completed through supplementation and correction by the generative AI model means.

[0974] The present invention relates to a system and method for improving the accuracy of a voice input system. The system receives voice input from a user, converts it to text, and then uses a generative AI model to complement and correct inaccuracies based on the context of the text. Specific embodiments of the system are described in detail below.

[0975] First, the user speaks into their device (e.g., smartphone, tablet, PC, etc.). The device has a built-in microphone that captures the user's voice and converts it into digital voice data. This digital voice data is sent to a server via the Internet.

[0976] The server uses a speech recognition engine to convert the received digital voice data into text. This speech recognition engine can be Google Cloud Speech-to-Text or DeepSpeech. The speech recognition engine analyzes the voice data and generates a primary speech recognition result. This primary speech recognition result is text data in the form of, for example, "The meeting is from Sanji."

[0977] The server then inputs the primary speech recognition results into a generative AI model. This generative AI model uses natural language processing technologies such as GPT-4 and BERT. The generative AI model analyzes the context to identify and correct inaccuracies in the primary speech recognition results. For example, it might correct "Sanji" to "Miji." Through this correction and completion process, the generative AI model generates the final text.

[0978] The final text after completion and correction is sent from the server to the user's device, which has the function of displaying the received final text, allowing the user to check the results of their voice input accurately and naturally.

[0979] A specific example is given below.

[0980] Specific examples

[0981] Example 1

[0982] User: Says, "Let's put this on hold."

[0983] Terminal: Captures audio and sends digital audio data to the server.

[0984] Server: A speech recognition engine (e.g., Google Cloud Speech-to-Text) converts the speech data into the initial text "Let's fence this."

[0985] Server: A generative AI model (e.g., GPT-4) analyzes the primary text based on context and modifies "fencing" to "pending."

[0986] Server: Sends the final text "Let's put this on hold" to the user terminal.

[0987] Terminal: Show the user "Let's put this on hold."

[0988] Example 2

[0989] User: Say, "Today's meeting starts at 3:00."

[0990] Terminal: Captures audio and sends digital audio data to the server.

[0991] Server: A speech recognition engine (e.g., DeepSpeech) converts the speech data into the text "Today's meeting is with Sanji."

[0992] Server: A generative AI model (e.g., BERT) analyzes the primary text based on context and modifies "Sanji" to "Miji."

[0993] Server: Sends the final text "Today's meeting starts at 3:00" to the user terminal.

[0994] Terminal: Display to the user "Today's meeting starts at 3:00."

[0995] An example of a prompt is as follows:

[0996] User Input: Let's put this on hold

[0997] Initial speech recognition result: Let's fence this one.

[0998] Completion and correction results by generative AI model: Let's put this on hold.

[0999] This system significantly improves the accuracy of voice recognition, providing users with a more natural and accurate voice input experience, enabling stress-free communication.

[1000] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1001] Step 1:

[1002] Acquiring voice input

[1003] User: The user speaks into the terminal. For example, the user says, "The meeting starts at 3 o'clock."

[1004] Device: The device's microphone captures the user's voice and converts it into digital audio data (e.g., WAV format). A microphone icon appears to indicate that recording is in progress.

[1005] Input: User's voice input.

[1006] Output: Digital audio data.

[1007] What it does: The device captures the audio data, stores it internally, and sends it to a server over the Internet (using the HTTPS protocol).

[1008] Step 2:

[1009] Primary Speech Recognition

[1010] Server: Receives digital audio data sent from the device. At this time, the audio data is temporarily stored in the server's memory.

[1011] Server: Using a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech), the voice data is converted into text, e.g., "The meeting is with Sanji."

[1012] Input: Digital audio data.

[1013] Output: Primary speech recognition result (text data).

[1014] Specific operation: The speech recognition engine starts working, analyzing and converting the voice data to generate text data. The user is notified that "Speech recognition in progress. Please wait a moment."

[1015] Step 3:

[1016] Contextual understanding and completion

[1017] Server: The primary speech recognition results are fed into a generative AI model (e.g., GPT-4 or BERT). The generative AI model analyzes the context of the text and complements or corrects inaccuracies. For example, correcting "Sanji" to "Miji."

[1018] Input: Primary speech recognition result (text data).

[1019] Output: The final completed and corrected text.

[1020] How it works: The generative AI model analyzes the context based on the primary speech recognition results and makes appropriate corrections. The progress of the analysis process is displayed in real time on the server console, and the user is informed that "text is being analyzed."

[1021] Step 4:

[1022] Generate and send the final result

[1023] Server: The completed and corrected text is formatted and prepared for final use, e.g., adding appropriate punctuation and formatting.

[1024] Server: Sends the final text to the user's device. Data is transferred securely using the HTTPS protocol.

[1025] Input: The completed and corrected final text.

[1026] Output: The final text that is displayed on the user's terminal.

[1027] Specific behavior: The server sends the final text to the user's device, and the device displays the received text to the user. A notification sound is played and a pop-up message appears saying "Processing completed. Please check the results."

[1028] (Application example 1)

[1029] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1030] Conventional speech recognition systems often fail to accurately capture user intent due to their low accuracy in converting speech to text. This problem is particularly severe in practical applications such as food delivery, where incorrectly recognized orders can lead to user dissatisfaction and a decline in service quality. Therefore, there is a need for systems that can more accurately convert speech input to text and provide context-based completion and correction.

[1031] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1032] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for converting the acquired voice input into text, a generative AI model means for analyzing the context of the converted text and completing and correcting it, a means for transmitting the completed and corrected final text to a user terminal, and a means for acquiring information using the voice input and confirming the user's order details, thereby improving the accuracy of voice recognition and enabling users to accurately place food delivery orders.

[1033] The "means for acquiring voice input from the user" is a function for acquiring information input by voice by the user as digital data using a device such as a microphone.

[1034] The "speech recognition engine means for primarily converting acquired voice input into text" is a system that executes a process of converting acquired voice data into text format using a machine learning algorithm.

[1035] "Generative AI model means for analyzing the context before and after primary converted text and completing and correcting it" is a function that uses generative AI technology to analyze the context of speech data that has been converted into text for the first time and to complete or correct inaccurate or missing parts.

[1036] "Means for sending the completed and corrected final text to the user's terminal" refers to the process of sending the text data completed and corrected by the generative AI model to the user's terminal.

[1037] "Means for obtaining information using voice input and confirming the user's order details" refers to a function that captures the order information entered by the user through voice and identifies and confirms the exact order details based on that information.

[1038] This invention is a system that accurately converts user speech input into text and enhances the user experience by complementing and correcting recognition errors, particularly in the context of food delivery orders.

[1039] System configuration and program processing

[1040] Hardware and Software

[1041] Hardware

[1042] User terminal: A smartphone or microphone is included for users to input voice.

[1043] Server: Equipped with high-performance CPU / GPU (e.g. NVIDIA A100).

[1044] software

[1045] Speech recognition engine: DeepSpeech (open source speech recognition engine)

[1046] Generative AI model: GPT-4 (generative AI model)

[1047] Food delivery application: Flutter (UI framework)

[1048] Processing flow

[1049] 1. Acquiring voice input

[1050] Users enter their food delivery order details by voice into their smartphone.

[1051] The smartphone's microphone captures the sound and generates digital audio data.

[1052] 2. Primary speech recognition

[1053] The smartphone sends the captured audio data to the server.

[1054] DeepSpeech, located on the server, converts the voice data into text.

[1055] 3. Contextual understanding and completion

[1056] The converted text is input into the GPT-4 model on the server.

[1057] The GPT-4 model analyzes the context of the text to identify inaccuracies or parts that were not heard.

[1058] The GPT-4 model complements and corrects the primary text based on the analysis results.

[1059] 4. Generate and send the final result

[1060] Format the completed and corrected text and prepare it as the final order text.

[1061] The final text is sent to the user's smartphone, and the order details are displayed.

[1062] Specific operation examples

[1063] Example 1:

[1064] User: Say "I'd like a fried chicken set and a Coke, please."

[1065] Smartphone: Captures audio and sends digital audio data to a server.

[1066] Server: DeepSpeech converts the primary text "I'd like a fried chicken sed and one curried please."

[1067] GPT-4 model: Analyzes the context and corrects "kare" to "cola" and "sed" to "set."

[1068] Server: "I'd like a fried chicken set and a bottle of coke, please" is sent to the smartphone and displayed.

[1069] Example 2:

[1070] User: Say "I'd like a Margherita pizza and orange juice, please."

[1071] Smartphone: Captures audio and sends digital audio data to a server.

[1072] Server: DeepSpeech converts the primary text "I'd like a Margherita pizza and an Olathe juice, please."

[1073] GPT-4 model: Corrects "Olathe" to "Orange Juice."

[1074] Server: "Margherita pizza and orange juice please" is sent to the smartphone and displayed.

[1075] Example of input prompt for generative AI model

[1076] Prompt statement:

[1077] Based on the context of a food delivery order, please correct the following speech recognition output to the correct order:

[1078] Initial speech recognition results: Margherita pizza and Ol'ze juice

[1079] Such systems and methods enable users to place more accurate food delivery orders by voice, improving the user experience.

[1080] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1081] Step 1:

[1082] Users speak into their smartphone to order food delivery, and the smartphone's microphone captures the voice and generates digital audio data, which is then transmitted over the Internet to a server.

[1083] Input: Audio input

[1084] Output: Digital audio data

[1085] How it works: The smartphone's microphone captures the user's voice and stores it as digital audio data, which is then sent over the internet to a server.

[1086] Step 2:

[1087] The server passes the received digital voice data to DeepSpeech, which converts the voice data into text. This primary text data is temporarily stored on the server.

[1088] Input: Digital audio data

[1089] Output: Primary text data

[1090] How it works: DeepSpeech on the server analyzes the audio data and converts it into text. The converted primary text is stored on the server.

[1091] Step 3:

[1092] The server inputs the primary text data into the GPT-4 model, which analyzes the context of the primary text to identify inaccuracies or unclear parts, and then appropriately complements and corrects the inaccuracies based on the context.

[1093] Input: Primary text data

[1094] Output: Completion and correction of text data

[1095] How it works: The GPT-4 model on the server analyzes text and context to identify inaccuracies, then provides contextual completion and correction.

[1096] Step 4:

[1097] The server formats the completed and corrected text and prepares it as the final text data, which is then sent to the user's smartphone via the Internet.

[1098] Input: Completed and corrected text data

[1099] Output: Final text data

[1100] Operation: The text data is formatted on the server to generate the final text data, which is then sent to the user's smartphone via the Internet.

[1101] Step 5:

[1102] The user's smartphone receives the final text data and displays it to the user in an appropriate format, allowing the user to confirm the exact food delivery order details.

[1103] Input: Final text data

[1104] Output: The order displayed to the user

[1105] Operation: The smartphone analyzes the final text data received from the server and displays it on the screen in a format that the user can read.

[1106] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1107] The present invention relates to a system and method for improving the accuracy of speech recognition systems, particularly by combining an emotion engine that recognizes user emotions to achieve more natural and context-aware speech recognition. This system acquires speech input from a user, converts it into text, and then uses a generative AI model to analyze the context surrounding the text, complementing and correcting inaccuracies, while also incorporating the user's emotional information recognized by the emotion engine.

[1108] Program processing

[1109] 1. Acquiring voice input:

[1110] Terminal: The user speaks into the terminal, which uses a microphone to capture the voice and stores the data digitally.

[1111] Device: Once the audio capture is complete, the digital audio data is sent over the internet to a server.

[1112] 2. Primary speech recognition:

[1113] Server: Passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[1114] Server: Temporarily stores the converted text. This is the primary speech recognition result. This result is the first step in the completion and correction process.

[1115] 3. Emotion Recognition:

[1116] Server: Inputs the voice data into the emotion engine, which analyzes the user's tone and speaking style.

[1117] Emotion engine (on the server):

[1118] It recognizes the user's emotions (e.g., anger, joy, sadness) from voice data and passes that emotional information to a generative AI model.

[1119] 4. Contextual understanding and completion:

[1120] Server: Passes the primary speech recognition results and recognized emotion information to the generative AI model for context analysis. The generative AI model analyzes the input text, its context, and the emotion information to identify inaccuracies.

[1121] Server: The generative AI model complements and corrects inaccurate parts of the primary speech recognition results based on contextual analysis and emotional information. For example, "The meeting is from Sanji" is corrected to "The meeting starts at 3 o'clock."

[1122] 5. Generate and send the final result:

[1123] Server: Formats the completed and corrected text and prepares it as the final text.

[1124] Server: Sends the prepared final text to the user's device, along with the recognized emotion information.

[1125] Terminal: Displays the final text and emotion information received to the user.

[1126] Specific examples

[1127] Example 1:

[1128] User: Says, "Let's put this on hold."

[1129] Terminal: Captures audio and sends digital audio data to the server.

[1130] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[1131] Server: The emotion engine analyzes the voice data and recognizes the user's calmness.

[1132] Server: The generative AI model analyzes the primary text and sentiment information and modifies "fencing" to "pending."

[1133] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[1134] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[1135] Example 2:

[1136] User: "The meeting starts at 3 o'clock today," says angrily.

[1137] Terminal: Captures audio and sends digital audio data to the server.

[1138] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[1139] Server: The emotion engine analyzes the voice data and recognizes the user's anger.

[1140] Server: The generative AI model analyzes the primary text and emotional information and corrects "Sanji" to "Miji."

[1141] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[1142] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[1143] In this way, the present invention further improves the accuracy and naturalness of speech recognition by incorporating emotion information, allowing users to take more appropriate action based on the emotion recognized by the system.

[1144] The processing flow will be explained below.

[1145] Step 1:

[1146] Device: The user speaks into the device, the device's microphone captures the audio, converts it into a digital format, and stores it in an internal buffer.

[1147] Step 2:

[1148] Device: After the audio capture is complete, the digital audio data is sent to the server over the internet using an HTTP POST request or WebSocket communication.

[1149] Step 3:

[1150] Server: The server passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[1151] Step 4:

[1152] Server: Stores the primary text obtained from the speech recognition engine in temporary memory. This is the primary speech recognition result, which is supplemented and corrected in subsequent processing steps.

[1153] Step 5:

[1154] Server: Inputs the voice data into the emotion engine, which analyzes the user's tone and speaking style. Based on this analysis, the user's emotion (e.g., anger, joy, sadness) is recognized.

[1155] Step 6:

[1156] Server: Obtains the emotion information obtained from the emotion engine and passes it to the generative AI model, which receives the primary speech recognition result, context, and emotion information as input.

[1157] Step 7:

[1158] Generative AI model (on server):

[1159] Analysis is performed based on the text and emotional information of the primary speech recognition results, as well as contextual information.

[1160] Identify and correct inaccuracies or nonsensical parts of the text.

[1161] For example, in the sentence "Today's meeting is with Sanji," "Sanji" is changed to "three o'clock" based on context and emotional information.

[1162] Step 8:

[1163] Server: Formats the completed and corrected final text and prepares it as the final text. It also converts it, including emotion information, into a format (e.g., a JSON file) to send to the user's device.

[1164] Step 9:

[1165] Server: Sends the final text and emotion information to the user's device, often via HTTPS.

[1166] Step 10:

[1167] Device: Analyzes the data received from the server, extracts the final text and emotion information, and processes the extracted text and emotion information to display them in the appropriate UI components.

[1168] Step 11:

[1169] Terminal: The final text and the recognized emotion information are displayed to the user, who can then take further action based on the displayed information.

[1170] Specific examples

[1171] Example 1:

[1172] User: Says, "Let's put this on hold."

[1173] Terminal: Captures audio and sends digital audio data to the server.

[1174] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[1175] Server: The emotion engine analyzes the voice data and recognizes the user's calm emotion.

[1176] Server: The generative AI model modifies "fencing" to "pending" based on the primary text and sentiment information.

[1177] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[1178] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[1179] Example 2:

[1180] User: "The meeting starts at 3 o'clock today," says angrily.

[1181] Terminal: Captures audio and sends digital audio data to the server.

[1182] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[1183] Server: The emotion engine analyzes the voice data and recognizes the user's anger emotion.

[1184] Server: The generative AI model modifies "Sanji" to "Miji" based on the primary text and emotional information.

[1185] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[1186] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[1187] In this way, the present invention further improves the accuracy and naturalness of speech recognition by incorporating emotion information, allowing users to take more appropriate actions based on the emotion recognized by the system.

[1188] Example 2

[1189] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1190] Conventional speech recognition systems often misrecognize users' speech input when converting it into text, resulting in inaccurate text. Furthermore, because they do not take the user's emotions into account, they have poor contextual understanding, making it difficult to achieve natural conversations. Therefore, there is a need to improve the accuracy and naturalness of speech recognition.

[1191] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1192] In this invention, the server includes means for acquiring voice input from a user, speech recognition engine means for primarily converting the acquired voice input into text, emotion engine means for recognizing the user's emotion based on the context surrounding the primarily converted text and the speech data, generative AI model means for analyzing the context based on the primarily converted text and the recognized emotion information and for supplementing and correcting the text, and means for transmitting the supplemented and corrected final text and the recognized emotion information to a user terminal, thereby enabling improvements in the accuracy and naturalness of speech recognition.

[1193] A "means for obtaining audio input" is a device or method that captures audio produced by a user.

[1194] The "voice recognition engine means" is a device or system that analyzes acquired voice data and converts it primarily into text.

[1195] The "emotion engine means" is a device or system that analyzes voice data and identifies the user's emotion from the tone, tempo, etc. of the voice.

[1196] The "generative AI model means" is an artificial intelligence model that analyzes the context based on the primary converted text and emotional information, and complements and corrects inaccurate parts.

[1197] "Completion and correction methods" are methods for correcting errors in text based on context and sentiment information.

[1198] The "means for transmitting the final text to the user terminal" refers to a device or system that transmits the completed text and emotion information to the user's device.

[1199] This invention is a system that acquires user voice input and performs highly accurate text conversion and emotion recognition based on it. This system aims to solve the problems of conventional voice recognition systems and improve the accuracy and naturalness of voice recognition by combining a voice recognition engine, an emotion recognition engine, and a generative AI model.

[1200] Hardware and Software Configuration

[1201] 1. A way to capture user voice input

[1202] Device: The user speaks into the device (PC, smartphone, tablet, etc.). The device is equipped with a high-sensitivity microphone (e.g., Shure MV5) to capture the voice with high accuracy. The device uses a digital signal processing chip to convert the analog voice signal into a digital signal.

[1203] 2. Speech Recognition Engine Means

[1204] Server: The voice data is sent to the server via the Internet, and the server uses a voice recognition engine (e.g., a general voice recognition API) to convert the voice into text.

[1205] 3. Emotion Recognition Engine Means

[1206] Server: Input the voice data into an emotion recognition engine (e.g., a general emotion analysis API) to analyze the tone, speed, and tempo of the voice to identify the user's emotion. For example, generate emotion labels such as "anger," "joy," or "sadness."

[1207] 4. Generative AI Model Means

[1208] Server: The primary text obtained from the speech recognition engine and the emotion information obtained from the emotion recognition engine are input into a generative AI model (e.g., a general GPT model). The generative AI model analyzes the primary text based on the context and emotion information, and corrects and complements inaccurate parts.

[1209] 5. Means of sending the final text to the user terminal

[1210] Server: Sends the final text and emotion information corrected by the generative AI model to the user device, which then displays the received text and emotion information to the user.

[1211] Specific examples

[1212] Example 1:

[1213] User: Says, "Let's put this on hold."

[1214] Device: Captures audio and converts it into digital audio data using a highly sensitive microphone, then transmits the digital audio data to a server over the Internet.

[1215] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[1216] Server: The emotion engine analyzes the voice data and recognizes the user's calmness.

[1217] Server: The generative AI model analyzes the primary text and sentiment information and modifies "fencing" to "pending."

[1218] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[1219] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[1220] Example 2:

[1221] User: "The meeting starts at 3 o'clock today," says angrily.

[1222] Device: Captures audio and converts it into digital audio data using a highly sensitive microphone, then transmits the digital audio data to a server over the Internet.

[1223] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[1224] Server: The emotion engine analyzes the voice data and recognizes the user's anger.

[1225] Server: The generative AI model analyzes the primary text and emotional information and corrects "Sanji" to "Miji."

[1226] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[1227] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[1228] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1229] Step 1:

[1230] Acquiring voice input

[1231] User: The user speaks into the device.

[1232] Input: User's voice.

[1233] Output: The audio signal captured on the device.

[1234] Terminal: Uses a sensitive microphone to capture audio and converts analog audio signals into digital signals.

[1235] Data processing: Converting analog audio signals into digital format.

[1236] What it does: A digital signal processing chip converts analog audio signals into digital signals.

[1237] Step 2:

[1238] Data transmission

[1239] Terminal: Digitized voice data is stored in temporary storage and transmitted to the server using a security protocol.

[1240] Input: Digital audio data.

[1241] Output: Digital audio data sent to the server.

[1242] Specific operation: Digital audio data is transmitted using TLS (Transport Layer Security).

[1243] Step 3:

[1244] Primary Speech Recognition

[1245] Server: Analyzes the received audio data.

[1246] Input: Digital audio data.

[1247] Output: Primary speech recognition results (text format).

[1248] Server: Uses a speech recognition engine (e.g., a general speech recognition API) to convert speech to text.

[1249] Data processing: Converting voice data into text.

[1250] What it does: A speech recognition algorithm analyzes the audio data and converts it into text.

[1251] Step 4:

[1252] emotion recognition

[1253] Server: Input the voice data into an emotion recognition engine (e.g., a common emotion analysis API) to identify the user's emotion.

[1254] Input: Digital audio data.

[1255] Output: Emotion information (e.g., anger, joy, sadness).

[1256] Data calculations: Analyze the tone, speed, and tempo of the speech and generate emotion labels.

[1257] Specific operation: The emotion recognition engine extracts voice features and determines the emotion.

[1258] Step 5:

[1259] Contextual understanding and completion

[1260] Server: Inputs the primary speech recognition results and emotion information into the generative AI model, analyzes the context, and identifies inaccuracies.

[1261] Input: Primary speech recognition results (text) and emotion information.

[1262] Output: The final corrected text.

[1263] Generative AI models: Completing and correcting inaccuracies in the primary text based on context and sentiment information.

[1264] Data arithmetic: Contextual analysis and correction.

[1265] How it works: The generative AI model corrects errors based on the context and sentiment of the text, correcting them to make them more appropriate.

[1266] Step 6:

[1267] Generate and send the final result

[1268] Server: Formats the completed and corrected final text and sends it to the user's terminal.

[1269] Input: Final corrected text and sentiment information.

[1270] Output: The final text and sentiment information sent to the user device.

[1271] Terminal: Receives the final text and emotion information and displays it to the user.

[1272] Data computation: Representing text and emotional information.

[1273] Specific operation: Converts the received content into a display format and displays it on the screen.

[1274] (Application example 2)

[1275] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1276] Conventional speech recognition systems simply convert speech to text without considering the user's emotions, which often results in inaccurate conversion and insufficient understanding of context. Furthermore, the inability to respond appropriately based on emotions has led to problems with lower satisfaction, especially when communicating with users on devices such as self-driving vehicles.

[1277] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1278] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for converting the acquired voice input into text, an emotion recognition engine means for recognizing the user's emotion from the acquired voice data, and a generative AI model means for analyzing the context before and after the converted text and supplementing and correcting it. This not only improves the accuracy of voice recognition, but also enables a more natural response that understands the context and includes the user's emotion.

[1279] "Voice input" refers to voice data uttered by a user through a microphone.

[1280] A "speech recognition engine" is software and hardware used to convert captured voice input into text form.

[1281] A "generative AI model" is an artificial intelligence model that analyzes the context of text and completes or corrects it.

[1282] An "emotion recognition engine" is software and hardware for extracting and recognizing a user's emotions from voice input.

[1283] The "context" refers to the content immediately before and after the speech input in the text data.

[1284] "Completion and correction" refers to changing inaccurate parts to appropriate content and adding missing information.

[1285] "Final text" refers to text data that has been primarily converted by a speech recognition engine, and then supplemented and corrected through an emotion recognition engine and a generative AI model.

[1286] A "user terminal" is a terminal device that a user uses to perform operations and display information.

[1287] To realize this invention, we use a system that combines a speech recognition engine, an emotion recognition engine, and a generative AI model. This enables highly accurate and natural conversion of user voice input into text. This system includes the following specific procedures, hardware, and software.

[1288] The server first receives voice input from the user. The voice recognition engine converts this voice input into text, which is then temporarily saved. The voice data is then input into the emotion recognition engine, which analyzes the user's emotions from the voice data and passes them to the generative AI model.

[1289] The generative AI model analyzes the converted text and its context, as well as the emotional information recognized by the emotion recognition engine. This analysis identifies inaccuracies in the converted text and corrects them appropriately. The final text and emotional information are sent to the user's device and displayed.

[1290] The hardware used includes a microphone for voice capture, a user terminal, and a server. Software includes a speech recognition engine, an emotion recognition engine, and a generative AI model. For example, the speech recognition engine can use the Python speech_recognition library, and the emotion recognition engine and generative AI model can use models connected via a REST API.

[1291] As a concrete example, consider the case where a user says, "I want to go to Shinjuku, but not right now." First, voice data is captured by a microphone and sent to a server. A speech recognition engine converts this voice data into text, "I want to go to Shinjuku, but not right now." Next, an emotion recognition engine recognizes the user's calmness from the voice. A generative AI model analyzes this text and emotional information to generate appropriate navigation advice. In this way, the system can provide a more natural and appropriate response.

[1292] An example prompt for a generative AI model is:

[1293] "The user says, 'I want to go to Shinjuku, but not right now.' The user is emotionally calm. Use this information to correct inaccuracies and generate appropriate navigation advice that fits the context."

[1294] This system improves the accuracy and naturalness of speech recognition, providing a more useful and reliable user experience.

[1295] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1296] Step 1:

[1297] Users speak into a microphone inside the autonomous vehicle, which captures the voice data, converts it into a digital format, and transmits it to a server.

[1298] Input: Voice input from the user

[1299] Output: Digital audio data sent to the server

[1300] Specific operation: When a user says, "I'd like to go to Shinjuku, but not right now," the microphone captures the voice, digitizes the voice data, and sends it to the server.

[1301] Step 2:

[1302] The server passes the received digital voice data to a speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[1303] Input: Digital audio data

[1304] Output: The converted text

[1305] Specific operation: The speech recognition engine converts the text into "I would like to go to Shinjuku, but not right now."

[1306] Step 3:

[1307] The server temporarily stores the converted text, which is then used for later analysis.

[1308] Input: The primary converted text

[1309] Output: Saved text data

[1310] Specific operation: The server temporarily stores the text data "I want to go to Shinjuku, but not right now" in storage.

[1311] Step 4:

[1312] The server inputs the voice data into an emotion recognition engine, which analyzes the user's emotions from the voice and passes the emotional information to a generative AI model.

[1313] Input: Digital audio data

[1314] Output: Recognized emotion information

[1315] Specific operation: The emotion recognition engine recognizes the user's calm emotions and provides that information to the generative AI model.

[1316] Step 5:

[1317] The server inputs the converted text and the recognized emotion information into a generative AI model, which analyzes the text based on the context and emotion information, identifying inaccurate parts and correcting them.

[1318] Input: Primary converted text, recognized emotion information

[1319] Output: Final completed and corrected text

[1320] Specific operation: The generative AI model analyzes the sentence, "I want to go to Shinjuku, but not right now," and complements and corrects it to suggest appropriate navigation.

[1321] Step 6:

[1322] The server sends the completed and corrected final text and emotion information to the user's device.

[1323] Input: Completion / correction final text, sentiment information

[1324] Output: Final text and emotion information sent to the user's device

[1325] Specific operation: The server sends the final text "I want to go to Shinjuku, but not right now" and the emotional information "calm" to the user terminal, which then displays it.

[1326] This series of steps improves the accuracy and naturalness of speech recognition, resulting in a system that can respond appropriately to the user's emotions.

[1327] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1328] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1329] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1330] [Fourth embodiment]

[1331] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1332] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1333] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1334] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1335] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1336] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1337] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1338] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1339] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1340] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1341] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1342] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1343] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1344] This invention relates to a system and method for improving the accuracy of a speech recognition system. This system receives speech input from a user, converts it into text, and then uses a generative AI model to complement and correct inaccuracies based on the context surrounding the text. The system's program and processing are described in detail below.

[1345] Program processing

[1346] 1. Acquiring voice input:

[1347] Terminal: The user speaks into the terminal, which uses a microphone to capture the voice and stores the data digitally.

[1348] Device: Once the audio capture is complete, the digital audio data is sent over the internet to a server.

[1349] 2. Primary speech recognition:

[1350] Server: Passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[1351] Server: Temporarily stores the converted text. This becomes the primary speech recognition result. This result is further supplemented and corrected in subsequent processing steps.

[1352] 3. Contextual understanding and completion:

[1353] Server: Inputs the primary speech recognition results into a generative AI model, which analyzes the context of the text and identifies inaccuracies or inaudible parts.

[1354] Server: The generative AI model uses contextual analysis to correct inaccuracies in the primary speech recognition results. For example, "The meeting is from Sanji" is corrected to "The meeting starts at 3 o'clock."

[1355] 4. Generate and send the final result:

[1356] Server: Formats the completed and corrected text and prepares it as the final text.

[1357] Server: Sends the prepared final text to the user's terminal.

[1358] On the device, the final text received is displayed to the user, allowing the user to see accurate and natural-looking results of their voice input.

[1359] Specific examples

[1360] Example 1:

[1361] User: Says, "Let's put this on hold."

[1362] Terminal: Captures audio and sends digital audio data to the server.

[1363] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[1364] Server: A generative AI model analyzes this primary text in context and modifies "fencing" to "pending."

[1365] Server: Sends the final text "Let's put this on hold" to the user terminal.

[1366] Terminal: Show the user "Let's put this on hold."

[1367] Example 2:

[1368] User: Say, "Today's meeting starts at 3:00."

[1369] Terminal: Captures audio and sends digital audio data to the server.

[1370] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[1371] Server: The generative AI model analyzes this primary text taking into account the context and corrects "Sanji" to "Miji".

[1372] Server: Sends the final text "Today's meeting starts at 3:00" to the user terminal.

[1373] Terminal: Display to the user "Today's meeting starts at 3:00."

[1374] In this way, the present invention significantly improves the accuracy of speech recognition and provides users with a more natural and accurate speech input experience. The system of the present invention enables users to achieve stress-free communication.

[1375] The processing flow will be explained below.

[1376] Step 1:

[1377] Device: Captures the user's voice. The user speaks into the device, and the device's microphone converts the voice into a digital format and stores it in an internal buffer.

[1378] Step 2:

[1379] Terminal: After the audio capture is complete, the digital audio data is sent to the server via the Internet using HTTP requests or WebSocket communication.

[1380] Step 3:

[1381] Server: Passes the received voice data to a voice recognition engine (e.g., a commercial voice recognition API), which analyzes the input voice data and converts it into temporary text.

[1382] Step 4:

[1383] Server: Receives the text output from the speech recognition engine and stores it in temporary memory (e.g., a database or cache memory). The text at this stage is the primary speech recognition result.

[1384] Step 5:

[1385] Server: Passes the primary speech recognition results to a generative AI model for context analysis. The generative AI model (e.g., a model based on natural language processing) analyzes the input text and its context to identify inaccurate or unclear parts.

[1386] Step 6:

[1387] Generative AI model (on server):

[1388] The AI ​​model corrects any identified inaccuracies based on the primary speech recognition results.

[1389] For example, the incorrectly recognized part "Sanji" is corrected to "three o'clock."

[1390] Step 7:

[1391] Server: Takes the corrected text data, reformats it as final text, and passes it on to other applications or services as needed.

[1392] Step 8:

[1393] Server: Sends the final text to the user's device, using JSON or another data exchange format.

[1394] Step 9:

[1395] Terminal: Parses the data received from the server and extracts the final text. An appropriate UI component (e.g., a text view or popup) is used to display the extracted text to the user.

[1396] Step 10:

[1397] Terminal: Displays the final text to the user, who can review it and take further action if necessary.

[1398] Example 1

[1399] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1400] Current speech recognition systems are prone to misrecognition during the initial conversion of speech input to text, resulting in inaccurate recognition of the user's intent, especially in noisy environments and complex contexts.

[1401] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1402] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for primarily converting voice into text, a generative AI model means for analyzing the context of the primarily converted text and supplementing and correcting it, and a means for transmitting the supplemented and corrected final text to a user terminal, thereby improving the accuracy of voice recognition, accurately reflecting the user's intention, and enabling a more natural and accurate voice input experience.

[1403] "Voice input" is a series of voice data uttered by a user into a terminal.

[1404] "Speech recognition engine means" means software or hardware that utilizes machine learning algorithms to analyze voice input and convert digital voice data into text.

[1405] The "generative AI model means" is an artificial intelligence model that uses natural language processing technology to analyze the context of the primarily converted text and complement and correct inaccurate parts.

[1406] A "user terminal" is a device that accepts voice input and displays the resulting text. Examples include smartphones, tablets, and PCs.

[1407] "Primary speech recognition result" is the text data initially generated from speech input by the speech recognition engine means.

[1408] "Completion and correction" is the process by which the generative AI model detects inaccuracies in the primary speech recognition results and corrects them based on appropriate context.

[1409] "Final text" refers to text data that has been completed through supplementation and correction by the generative AI model means.

[1410] The present invention relates to a system and method for improving the accuracy of a voice input system. The system receives voice input from a user, converts it to text, and then uses a generative AI model to complement and correct inaccuracies based on the context of the text. Specific embodiments of the system are described in detail below.

[1411] First, the user speaks into their device (e.g., smartphone, tablet, PC, etc.). The device has a built-in microphone that captures the user's voice and converts it into digital voice data. This digital voice data is sent to a server via the Internet.

[1412] The server uses a speech recognition engine to convert the received digital voice data into text. This speech recognition engine can be Google Cloud Speech-to-Text or DeepSpeech. The speech recognition engine analyzes the voice data and generates a primary speech recognition result. This primary speech recognition result is text data in the form of, for example, "The meeting is from Sanji."

[1413] The server then inputs the primary speech recognition results into a generative AI model. This generative AI model uses natural language processing technologies such as GPT-4 and BERT. The generative AI model analyzes the context to identify and correct inaccuracies in the primary speech recognition results. For example, it might correct "Sanji" to "Miji." Through this correction and completion process, the generative AI model generates the final text.

[1414] The final text after completion and correction is sent from the server to the user's device, which has the function of displaying the received final text, allowing the user to check the results of their voice input accurately and naturally.

[1415] A specific example is given below.

[1416] Specific examples

[1417] Example 1

[1418] User: Says, "Let's put this on hold."

[1419] Terminal: Captures audio and sends digital audio data to the server.

[1420] Server: A speech recognition engine (e.g., Google Cloud Speech-to-Text) converts the speech data into the initial text "Let's fence this."

[1421] Server: A generative AI model (e.g., GPT-4) analyzes the primary text based on context and modifies "fencing" to "pending."

[1422] Server: Sends the final text "Let's put this on hold" to the user terminal.

[1423] Terminal: Show the user "Let's put this on hold."

[1424] Example 2

[1425] User: Say, "Today's meeting starts at 3:00."

[1426] Terminal: Captures audio and sends digital audio data to the server.

[1427] Server: A speech recognition engine (e.g., DeepSpeech) converts the speech data into the text "Today's meeting is with Sanji."

[1428] Server: A generative AI model (e.g., BERT) analyzes the primary text based on context and modifies "Sanji" to "Miji."

[1429] Server: Sends the final text "Today's meeting starts at 3:00" to the user terminal.

[1430] Terminal: Display to the user "Today's meeting starts at 3:00."

[1431] An example of a prompt is as follows:

[1432] User Input: Let's put this on hold

[1433] Initial speech recognition result: Let's fence this one.

[1434] Completion and correction results by generative AI model: Let's put this on hold.

[1435] This system significantly improves the accuracy of voice recognition, providing users with a more natural and accurate voice input experience, enabling stress-free communication.

[1436] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1437] Step 1:

[1438] Acquiring voice input

[1439] User: The user speaks into the terminal. For example, the user says, "The meeting starts at 3 o'clock."

[1440] Device: The device's microphone captures the user's voice and converts it into digital audio data (e.g., WAV format). A microphone icon appears to indicate that recording is in progress.

[1441] Input: User's voice input.

[1442] Output: Digital audio data.

[1443] What it does: The device captures the audio data, stores it internally, and sends it to a server over the Internet (using the HTTPS protocol).

[1444] Step 2:

[1445] Primary Speech Recognition

[1446] Server: Receives digital audio data sent from the device. At this time, the audio data is temporarily stored in the server's memory.

[1447] Server: Using a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech), the voice data is converted into text, e.g., "The meeting is with Sanji."

[1448] Input: Digital audio data.

[1449] Output: Primary speech recognition result (text data).

[1450] Specific operation: The speech recognition engine starts working, analyzing and converting the voice data to generate text data. The user is notified that "Speech recognition in progress. Please wait a moment."

[1451] Step 3:

[1452] Contextual understanding and completion

[1453] Server: The primary speech recognition results are fed into a generative AI model (e.g., GPT-4 or BERT). The generative AI model analyzes the context of the text and complements or corrects inaccuracies. For example, correcting "Sanji" to "Miji."

[1454] Input: Primary speech recognition result (text data).

[1455] Output: The final completed and corrected text.

[1456] How it works: The generative AI model analyzes the context based on the primary speech recognition results and makes appropriate corrections. The progress of the analysis process is displayed in real time on the server console, and the user is informed that "text is being analyzed."

[1457] Step 4:

[1458] Generate and send the final result

[1459] Server: The completed and corrected text is formatted and prepared for final use, e.g., adding appropriate punctuation and formatting.

[1460] Server: Sends the final text to the user's device. Data is transferred securely using the HTTPS protocol.

[1461] Input: The completed and corrected final text.

[1462] Output: The final text that is displayed on the user's terminal.

[1463] Specific behavior: The server sends the final text to the user's device, and the device displays the received text to the user. A notification sound is played and a pop-up message appears saying "Processing completed. Please check the results."

[1464] (Application example 1)

[1465] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1466] Conventional speech recognition systems often fail to accurately capture user intent due to their low accuracy in converting speech to text. This problem is particularly severe in practical applications such as food delivery, where incorrectly recognized orders can lead to user dissatisfaction and a decline in service quality. Therefore, there is a need for systems that can more accurately convert speech input to text and provide context-based completion and correction.

[1467] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1468] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for converting the acquired voice input into text, a generative AI model means for analyzing the context of the converted text and completing and correcting it, a means for transmitting the completed and corrected final text to a user terminal, and a means for acquiring information using the voice input and confirming the user's order details, thereby improving the accuracy of voice recognition and enabling users to accurately place food delivery orders.

[1469] The "means for acquiring voice input from the user" is a function for acquiring information input by voice by the user as digital data using a device such as a microphone.

[1470] The "speech recognition engine means for primarily converting acquired voice input into text" is a system that executes a process of converting acquired voice data into text format using a machine learning algorithm.

[1471] "Generative AI model means for analyzing the context before and after primary converted text and completing and correcting it" is a function that uses generative AI technology to analyze the context of speech data that has been converted into text for the first time and to complete or correct inaccurate or missing parts.

[1472] "Means for sending the completed and corrected final text to the user's terminal" refers to the process of sending the text data completed and corrected by the generative AI model to the user's terminal.

[1473] "Means for obtaining information using voice input and confirming the user's order details" refers to a function that captures the order information entered by the user through voice and identifies and confirms the exact order details based on that information.

[1474] This invention is a system that accurately converts user speech input into text and enhances the user experience by complementing and correcting recognition errors, particularly in the context of food delivery orders.

[1475] System configuration and program processing

[1476] Hardware and Software

[1477] Hardware

[1478] User terminal: A smartphone or microphone is included for users to input voice.

[1479] Server: Equipped with high-performance CPU / GPU (e.g. NVIDIA A100).

[1480] software

[1481] Speech recognition engine: DeepSpeech (open source speech recognition engine)

[1482] Generative AI model: GPT-4 (generative AI model)

[1483] Food delivery application: Flutter (UI framework)

[1484] Processing flow

[1485] 1. Acquiring voice input

[1486] Users enter their food delivery order details by voice into their smartphone.

[1487] The smartphone's microphone captures the sound and generates digital audio data.

[1488] 2. Primary speech recognition

[1489] The smartphone sends the captured audio data to the server.

[1490] DeepSpeech, located on the server, converts the voice data into text.

[1491] 3. Contextual understanding and completion

[1492] The converted text is input into the GPT-4 model on the server.

[1493] The GPT-4 model analyzes the context of the text to identify inaccuracies or parts that were not heard.

[1494] The GPT-4 model complements and corrects the primary text based on the analysis results.

[1495] 4. Generate and send the final result

[1496] Format the completed and corrected text and prepare it as the final order text.

[1497] The final text is sent to the user's smartphone, and the order details are displayed.

[1498] Specific operation examples

[1499] Example 1:

[1500] User: Say "I'd like a fried chicken set and a Coke, please."

[1501] Smartphone: Captures audio and sends digital audio data to a server.

[1502] Server: DeepSpeech converts the primary text "I'd like a fried chicken sed and one curried please."

[1503] GPT-4 model: Analyzes the context and corrects "kare" to "cola" and "sed" to "set."

[1504] Server: "I'd like a fried chicken set and a bottle of coke, please" is sent to the smartphone and displayed.

[1505] Example 2:

[1506] User: Say "I'd like a Margherita pizza and orange juice, please."

[1507] Smartphone: Captures audio and sends digital audio data to a server.

[1508] Server: DeepSpeech converts the primary text "I'd like a Margherita pizza and an Olathe juice, please."

[1509] GPT-4 model: Corrects "Olathe" to "Orange Juice."

[1510] Server: "Margherita pizza and orange juice please" is sent to the smartphone and displayed.

[1511] Example of input prompt for generative AI model

[1512] Prompt statement:

[1513] Based on the context of a food delivery order, please correct the following speech recognition output to the correct order:

[1514] Initial speech recognition results: Margherita pizza and Ol'ze juice

[1515] Such systems and methods enable users to place more accurate food delivery orders by voice, improving the user experience.

[1516] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1517] Step 1:

[1518] Users speak into their smartphone to order food delivery, and the smartphone's microphone captures the voice and generates digital audio data, which is then transmitted over the Internet to a server.

[1519] Input: Audio input

[1520] Output: Digital audio data

[1521] How it works: The smartphone's microphone captures the user's voice and stores it as digital audio data, which is then sent over the internet to a server.

[1522] Step 2:

[1523] The server passes the received digital voice data to DeepSpeech, which converts the voice data into text. This primary text data is temporarily stored on the server.

[1524] Input: Digital audio data

[1525] Output: Primary text data

[1526] How it works: DeepSpeech on the server analyzes the audio data and converts it into text. The converted primary text is stored on the server.

[1527] Step 3:

[1528] The server inputs the primary text data into the GPT-4 model, which analyzes the context of the primary text to identify inaccuracies or unclear parts, and then appropriately complements and corrects the inaccuracies based on the context.

[1529] Input: Primary text data

[1530] Output: Completion and correction of text data

[1531] How it works: The GPT-4 model on the server analyzes text and context to identify inaccuracies, then provides contextual completion and correction.

[1532] Step 4:

[1533] The server formats the completed and corrected text and prepares it as the final text data, which is then sent to the user's smartphone via the Internet.

[1534] Input: Completed and corrected text data

[1535] Output: Final text data

[1536] Operation: The text data is formatted on the server to generate the final text data, which is then sent to the user's smartphone via the Internet.

[1537] Step 5:

[1538] The user's smartphone receives the final text data and displays it to the user in an appropriate format, allowing the user to confirm the exact food delivery order details.

[1539] Input: Final text data

[1540] Output: The order displayed to the user

[1541] Operation: The smartphone analyzes the final text data received from the server and displays it on the screen in a format that the user can read.

[1542] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1543] The present invention relates to a system and method for improving the accuracy of speech recognition systems, particularly by combining an emotion engine that recognizes user emotions to achieve more natural and context-aware speech recognition. This system acquires speech input from a user, converts it into text, and then uses a generative AI model to analyze the context surrounding the text, complementing and correcting inaccuracies, while also incorporating the user's emotional information recognized by the emotion engine.

[1544] Program processing

[1545] 1. Acquiring voice input:

[1546] Terminal: The user speaks into the terminal, which uses a microphone to capture the voice and stores the data digitally.

[1547] Device: Once the audio capture is complete, the digital audio data is sent over the internet to a server.

[1548] 2. Primary speech recognition:

[1549] Server: Passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[1550] Server: Temporarily stores the converted text. This is the primary speech recognition result. This result is the first step in the completion and correction process.

[1551] 3. Emotion Recognition:

[1552] Server: Inputs the voice data into the emotion engine, which analyzes the user's tone and speaking style.

[1553] Emotion engine (on the server):

[1554] It recognizes the user's emotions (e.g., anger, joy, sadness) from voice data and passes that emotional information to a generative AI model.

[1555] 4. Contextual understanding and completion:

[1556] Server: Passes the primary speech recognition results and recognized emotion information to the generative AI model for context analysis. The generative AI model analyzes the input text, its context, and the emotion information to identify inaccuracies.

[1557] Server: The generative AI model complements and corrects inaccurate parts of the primary speech recognition results based on contextual analysis and emotional information. For example, "The meeting is from Sanji" is corrected to "The meeting starts at 3 o'clock."

[1558] 5. Generate and send the final result:

[1559] Server: Formats the completed and corrected text and prepares it as the final text.

[1560] Server: Sends the prepared final text to the user's device, along with the recognized emotion information.

[1561] Terminal: Displays the final text and emotion information received to the user.

[1562] Specific examples

[1563] Example 1:

[1564] User: Says, "Let's put this on hold."

[1565] Terminal: Captures audio and sends digital audio data to the server.

[1566] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[1567] Server: The emotion engine analyzes the voice data and recognizes the user's calmness.

[1568] Server: The generative AI model analyzes the primary text and sentiment information and modifies "fencing" to "pending."

[1569] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[1570] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[1571] Example 2:

[1572] User: "The meeting starts at 3 o'clock today," says angrily.

[1573] Terminal: Captures audio and sends digital audio data to the server.

[1574] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[1575] Server: The emotion engine analyzes the voice data and recognizes the user's anger.

[1576] Server: The generative AI model analyzes the primary text and emotional information and corrects "Sanji" to "Miji."

[1577] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[1578] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[1579] In this way, the present invention further improves the accuracy and naturalness of speech recognition by incorporating emotion information, allowing users to take more appropriate action based on the emotion recognized by the system.

[1580] The processing flow will be explained below.

[1581] Step 1:

[1582] Device: The user speaks into the device, the device's microphone captures the audio, converts it into a digital format, and stores it in an internal buffer.

[1583] Step 2:

[1584] Device: After the audio capture is complete, the digital audio data is sent to the server over the internet using an HTTP POST request or WebSocket communication.

[1585] Step 3:

[1586] Server: The server passes the received voice data to the speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[1587] Step 4:

[1588] Server: Stores the primary text obtained from the speech recognition engine in temporary memory. This is the primary speech recognition result, which is supplemented and corrected in subsequent processing steps.

[1589] Step 5:

[1590] Server: Inputs the voice data into the emotion engine, which analyzes the user's tone and speaking style. Based on this analysis, the user's emotion (e.g., anger, joy, sadness) is recognized.

[1591] Step 6:

[1592] Server: Obtains the emotion information obtained from the emotion engine and passes it to the generative AI model, which receives the primary speech recognition result, context, and emotion information as input.

[1593] Step 7:

[1594] Generative AI model (on server):

[1595] Analysis is performed based on the text and emotional information of the primary speech recognition results, as well as contextual information.

[1596] Identify and correct inaccuracies or nonsensical parts of the text.

[1597] For example, in the sentence "Today's meeting is with Sanji," "Sanji" is changed to "three o'clock" based on context and emotional information.

[1598] Step 8:

[1599] Server: Formats the completed and corrected final text and prepares it as the final text. It also converts it, including emotion information, into a format (e.g., a JSON file) to send to the user's device.

[1600] Step 9:

[1601] Server: Sends the final text and emotion information to the user's device, often via HTTPS.

[1602] Step 10:

[1603] Device: Analyzes the data received from the server, extracts the final text and emotion information, and processes the extracted text and emotion information to display them in the appropriate UI components.

[1604] Step 11:

[1605] Terminal: The final text and the recognized emotion information are displayed to the user, who can then take further action based on the displayed information.

[1606] Specific examples

[1607] Example 1:

[1608] User: Says, "Let's put this on hold."

[1609] Terminal: Captures audio and sends digital audio data to the server.

[1610] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[1611] Server: The emotion engine analyzes the voice data and recognizes the user's calm emotion.

[1612] Server: The generative AI model modifies "fencing" to "pending" based on the primary text and sentiment information.

[1613] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[1614] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[1615] Example 2:

[1616] User: "The meeting starts at 3 o'clock today," says angrily.

[1617] Terminal: Captures audio and sends digital audio data to the server.

[1618] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[1619] Server: The emotion engine analyzes the voice data and recognizes the user's anger emotion.

[1620] Server: The generative AI model modifies "Sanji" to "Miji" based on the primary text and emotional information.

[1621] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[1622] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[1623] In this way, the present invention further improves the accuracy and naturalness of speech recognition by incorporating emotion information, allowing users to take more appropriate actions based on the emotion recognized by the system.

[1624] Example 2

[1625] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1626] Conventional speech recognition systems often misrecognize users' speech input when converting it into text, resulting in inaccurate text. Furthermore, because they do not take the user's emotions into account, they have poor contextual understanding, making it difficult to achieve natural conversations. Therefore, there is a need to improve the accuracy and naturalness of speech recognition.

[1627] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1628] In this invention, the server includes means for acquiring voice input from a user, speech recognition engine means for primarily converting the acquired voice input into text, emotion engine means for recognizing the user's emotion based on the context surrounding the primarily converted text and the speech data, generative AI model means for analyzing the context based on the primarily converted text and the recognized emotion information and for supplementing and correcting the text, and means for transmitting the supplemented and corrected final text and the recognized emotion information to a user terminal, thereby enabling improvements in the accuracy and naturalness of speech recognition.

[1629] A "means for obtaining audio input" is a device or method that captures audio produced by a user.

[1630] The "voice recognition engine means" is a device or system that analyzes acquired voice data and converts it primarily into text.

[1631] The "emotion engine means" is a device or system that analyzes voice data and identifies the user's emotion from the tone, tempo, etc. of the voice.

[1632] The "generative AI model means" is an artificial intelligence model that analyzes the context based on the primary converted text and emotional information, and complements and corrects inaccurate parts.

[1633] "Completion and correction methods" are methods for correcting errors in text based on context and sentiment information.

[1634] The "means for transmitting the final text to the user terminal" refers to a device or system that transmits the completed text and emotion information to the user's device.

[1635] This invention is a system that acquires user voice input and performs highly accurate text conversion and emotion recognition based on it. This system aims to solve the problems of conventional voice recognition systems and improve the accuracy and naturalness of voice recognition by combining a voice recognition engine, an emotion recognition engine, and a generative AI model.

[1636] Hardware and Software Configuration

[1637] 1. A way to capture user voice input

[1638] Device: The user speaks into the device (PC, smartphone, tablet, etc.). The device is equipped with a high-sensitivity microphone (e.g., Shure MV5) to capture the voice with high accuracy. The device uses a digital signal processing chip to convert the analog voice signal into a digital signal.

[1639] 2. Speech Recognition Engine Means

[1640] Server: The voice data is sent to the server via the Internet, and the server uses a voice recognition engine (e.g., a general voice recognition API) to convert the voice into text.

[1641] 3. Emotion Recognition Engine Means

[1642] Server: Input the voice data into an emotion recognition engine (e.g., a general emotion analysis API) to analyze the tone, speed, and tempo of the voice to identify the user's emotion. For example, generate emotion labels such as "anger," "joy," or "sadness."

[1643] 4. Generative AI Model Means

[1644] Server: The primary text obtained from the speech recognition engine and the emotion information obtained from the emotion recognition engine are input into a generative AI model (e.g., a general GPT model). The generative AI model analyzes the primary text based on the context and emotion information, and corrects and complements inaccurate parts.

[1645] 5. Means of sending the final text to the user terminal

[1646] Server: Sends the final text and emotion information corrected by the generative AI model to the user device, which then displays the received text and emotion information to the user.

[1647] Specific examples

[1648] Example 1:

[1649] User: Says, "Let's put this on hold."

[1650] Device: Captures audio and converts it into digital audio data using a highly sensitive microphone, then transmits the digital audio data to a server over the Internet.

[1651] Server: The speech recognition engine converts the speech data into the text "Let's fence this."

[1652] Server: The emotion engine analyzes the voice data and recognizes the user's calmness.

[1653] Server: The generative AI model analyzes the primary text and sentiment information and modifies "fencing" to "pending."

[1654] Server: Sends the final text "Let's put this on hold" and the emotional information "Calm" to the user terminal.

[1655] Terminal: Show the user "Let's put this on hold" and "Keep calm."

[1656] Example 2:

[1657] User: "The meeting starts at 3 o'clock today," says angrily.

[1658] Device: Captures audio and converts it into digital audio data using a highly sensitive microphone, then transmits the digital audio data to a server over the Internet.

[1659] Server: The speech recognition engine converts the speech data into text, "Today's meeting is with Sanji."

[1660] Server: The emotion engine analyzes the voice data and recognizes the user's anger.

[1661] Server: The generative AI model analyzes the primary text and emotional information and corrects "Sanji" to "Miji."

[1662] Server: Sends the final text "Today's meeting starts at 3 o'clock" and the emotion information "anger" to the user terminal.

[1663] Terminal: Display "Today's meeting starts at 3 o'clock" and "Anger" to the user.

[1664] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1665] Step 1:

[1666] Acquiring voice input

[1667] User: The user speaks into the device.

[1668] Input: User's voice.

[1669] Output: The audio signal captured on the device.

[1670] Terminal: Uses a sensitive microphone to capture audio and converts analog audio signals into digital signals.

[1671] Data processing: Converting analog audio signals into digital format.

[1672] What it does: A digital signal processing chip converts analog audio signals into digital signals.

[1673] Step 2:

[1674] Data transmission

[1675] Terminal: Digitized voice data is stored in temporary storage and transmitted to the server using a security protocol.

[1676] Input: Digital audio data.

[1677] Output: Digital audio data sent to the server.

[1678] Specific operation: Digital audio data is transmitted using TLS (Transport Layer Security).

[1679] Step 3:

[1680] Primary Speech Recognition

[1681] Server: Analyzes the received audio data.

[1682] Input: Digital audio data.

[1683] Output: Primary speech recognition results (text format).

[1684] Server: Uses a speech recognition engine (e.g., a general speech recognition API) to convert speech to text.

[1685] Data processing: Converting voice data into text.

[1686] What it does: A speech recognition algorithm analyzes the audio data and converts it into text.

[1687] Step 4:

[1688] emotion recognition

[1689] Server: Input the voice data into an emotion recognition engine (e.g., a common emotion analysis API) to identify the user's emotion.

[1690] Input: Digital audio data.

[1691] Output: Emotion information (e.g., anger, joy, sadness).

[1692] Data calculations: Analyze the tone, speed, and tempo of the speech and generate emotion labels.

[1693] Specific operation: The emotion recognition engine extracts voice features and determines the emotion.

[1694] Step 5:

[1695] Contextual understanding and completion

[1696] Server: Inputs the primary speech recognition results and emotion information into the generative AI model, analyzes the context, and identifies inaccuracies.

[1697] Input: Primary speech recognition results (text) and emotion information.

[1698] Output: The final corrected text.

[1699] Generative AI models: Completing and correcting inaccuracies in the primary text based on context and sentiment information.

[1700] Data arithmetic: Contextual analysis and correction.

[1701] How it works: The generative AI model corrects errors based on the context and sentiment of the text, correcting them to make them more appropriate.

[1702] Step 6:

[1703] Generate and send the final result

[1704] Server: Formats the completed and corrected final text and sends it to the user's terminal.

[1705] Input: Final corrected text and sentiment information.

[1706] Output: The final text and sentiment information sent to the user device.

[1707] Terminal: Receives the final text and emotion information and displays it to the user.

[1708] Data computation: Representing text and emotional information.

[1709] Specific operation: Converts the received content into a display format and displays it on the screen.

[1710] (Application example 2)

[1711] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1712] Conventional speech recognition systems simply convert speech to text without considering the user's emotions, which often results in inaccurate conversion and insufficient understanding of context. Furthermore, the inability to respond appropriately based on emotions has led to problems with lower satisfaction, especially when communicating with users on devices such as self-driving vehicles.

[1713] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1714] In this invention, the server includes means for acquiring voice input from a user, a voice recognition engine means for converting the acquired voice input into text, an emotion recognition engine means for recognizing the user's emotion from the acquired voice data, and a generative AI model means for analyzing the context before and after the converted text and supplementing and correcting it. This not only improves the accuracy of voice recognition, but also enables a more natural response that understands the context and includes the user's emotion.

[1715] "Voice input" refers to voice data uttered by a user through a microphone.

[1716] A "speech recognition engine" is software and hardware used to convert captured voice input into text form.

[1717] A "generative AI model" is an artificial intelligence model that analyzes the context of text and completes or corrects it.

[1718] An "emotion recognition engine" is software and hardware for extracting and recognizing a user's emotions from voice input.

[1719] The "context" refers to the content immediately before and after the speech input in the text data.

[1720] "Completion and correction" refers to changing inaccurate parts to appropriate content and adding missing information.

[1721] "Final text" refers to text data that has been primarily converted by a speech recognition engine, and then supplemented and corrected through an emotion recognition engine and a generative AI model.

[1722] A "user terminal" is a terminal device that a user uses to perform operations and display information.

[1723] To realize this invention, we use a system that combines a speech recognition engine, an emotion recognition engine, and a generative AI model. This enables highly accurate and natural conversion of user voice input into text. This system includes the following specific procedures, hardware, and software.

[1724] The server first receives voice input from the user. The voice recognition engine converts this voice input into text, which is then temporarily saved. The voice data is then input into the emotion recognition engine, which analyzes the user's emotions from the voice data and passes them to the generative AI model.

[1725] The generative AI model analyzes the converted text and its context, as well as the emotional information recognized by the emotion recognition engine. This analysis identifies inaccuracies in the converted text and corrects them appropriately. The final text and emotional information are sent to the user's device and displayed.

[1726] The hardware used includes a microphone for voice capture, a user terminal, and a server. Software includes a speech recognition engine, an emotion recognition engine, and a generative AI model. For example, the speech recognition engine can use the Python speech_recognition library, and the emotion recognition engine and generative AI model can use models connected via a REST API.

[1727] As a concrete example, consider the case where a user says, "I want to go to Shinjuku, but not right now." First, voice data is captured by a microphone and sent to a server. A speech recognition engine converts this voice data into text, "I want to go to Shinjuku, but not right now." Next, an emotion recognition engine recognizes the user's calmness from the voice. A generative AI model analyzes this text and emotional information to generate appropriate navigation advice. In this way, the system can provide a more natural and appropriate response.

[1728] An example prompt for a generative AI model is:

[1729] "The user says, 'I want to go to Shinjuku, but not right now.' The user is emotionally calm. Use this information to correct inaccuracies and generate appropriate navigation advice that fits the context."

[1730] This system improves the accuracy and naturalness of speech recognition, providing a more useful and reliable user experience.

[1731] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1732] Step 1:

[1733] Users speak into a microphone inside the autonomous vehicle, which captures the voice data, converts it into a digital format, and transmits it to a server.

[1734] Input: Voice input from the user

[1735] Output: Digital audio data sent to the server

[1736] Specific operation: When a user says, "I'd like to go to Shinjuku, but not right now," the microphone captures the voice, digitizes the voice data, and sends it to the server.

[1737] Step 2:

[1738] The server passes the received digital voice data to a speech recognition engine, which uses machine learning algorithms to convert the voice data into text.

[1739] Input: Digital audio data

[1740] Output: The converted text

[1741] Specific operation: The speech recognition engine converts the text into "I would like to go to Shinjuku, but not right now."

[1742] Step 3:

[1743] The server temporarily stores the converted text, which is then used for later analysis.

[1744] Input: The primary converted text

[1745] Output: Saved text data

[1746] Specific operation: The server temporarily stores the text data "I want to go to Shinjuku, but not right now" in storage.

[1747] Step 4:

[1748] The server inputs the voice data into an emotion recognition engine, which analyzes the user's emotions from the voice and passes the emotional information to a generative AI model.

[1749] Input: Digital audio data

[1750] Output: Recognized emotion information

[1751] Specific operation: The emotion recognition engine recognizes the user's calm emotions and provides that information to the generative AI model.

[1752] Step 5:

[1753] The server inputs the converted text and the recognized emotion information into a generative AI model, which analyzes the text based on the context and emotion information, identifying inaccurate parts and correcting them.

[1754] Input: Primary converted text, recognized emotion information

[1755] Output: Final completed and corrected text

[1756] Specific operation: The generative AI model analyzes the sentence, "I want to go to Shinjuku, but not right now," and complements and corrects it to suggest appropriate navigation.

[1757] Step 6:

[1758] The server sends the completed and corrected final text and emotion information to the user's device.

[1759] Input: Completion / correction final text, sentiment information

[1760] Output: Final text and emotion information sent to the user's device

[1761] Specific operation: The server sends the final text "I want to go to Shinjuku, but not right now" and the emotional information "calm" to the user terminal, which then displays it.

[1762] This series of steps improves the accuracy and naturalness of speech recognition, resulting in a system that can respond appropriately to the user's emotions.

[1763] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1764] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1765] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1766] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1767] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1768] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1769] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1770] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1771] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1772] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1773] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1774] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1775] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1776] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1777] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1778] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1779] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1780] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1781] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1782] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1783] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1784] The following is further disclosed regarding the above embodiment.

[1785] (Claim 1)

[1786] means for obtaining speech input from a user;

[1787] a speech recognition engine means for converting the acquired speech input into text;

[1788] A generative AI model means for analyzing the context of the primary converted text and completing and correcting it;

[1789] means for transmitting the completed and corrected final text to the user terminal;

[1790] A system including:

[1791] (Claim 2)

[1792] 10. The system of claim 1, further comprising means for identifying and correcting the inaccurate portion of the primary converted text based on appropriate context if the primary converted text is inaccurate.

[1793] (Claim 3)

[1794] The system of claim 1, further comprising means for inputting the primary converted text and surrounding context information into the generative AI model means, and providing a method for the generative AI model means to correct inaccuracies based on the context.

[1795] "Example 1"

[1796] (Claim 1)

[1797] means for obtaining speech input from a user;

[1798] a speech recognition engine means for converting the acquired speech input into text;

[1799] A generative AI model means for analyzing the context of the primary converted text and completing and correcting it;

[1800] means for transmitting the completed and corrected final text to the user terminal;

[1801] A system including:

[1802] (Claim 2)

[1803] 10. The system of claim 1, further comprising means for identifying and correcting the inaccurate portion of the primary converted text based on appropriate context if the primary converted text is inaccurate.

[1804] (Claim 3)

[1805] The system of claim 1, further comprising means for inputting the primary converted text and surrounding context information into the generative AI model means, and providing a method for the generative AI model means to correct inaccuracies based on the context.

[1806] "Application Example 1"

[1807] (Claim 1)

[1808] means for obtaining speech input from a user;

[1809] a speech recognition engine means for converting the acquired speech input into text;

[1810] A generative AI model means for analyzing the context of the primary converted text and completing and correcting it;

[1811] means for transmitting the completed and corrected final text to the user terminal;

[1812] means for obtaining information using voice input to confirm the user's order;

[1813] A system including:

[1814] (Claim 2)

[1815] 10. The system of claim 1, further comprising means for identifying and correcting the inaccurate portion of the primary converted text based on appropriate context if the primary converted text is inaccurate.

[1816] (Claim 3)

[1817] The system of claim 1, further comprising means for inputting the primary converted text and surrounding context information into the generative AI model means, and providing a method for the generative AI model means to correct inaccuracies based on the context.

[1818] "Example 2: Combining Emotion Engines"

[1819] (Claim 1)

[1820] means for obtaining speech input from a user;

[1821] a speech recognition engine means for converting the acquired speech input into text;

[1822] emotion engine means for recognizing a user's emotion based on the context of the primary converted text and the speech data;

[1823] A generative AI model means for analyzing, complementing, and correcting the context based on the primary converted text and the recognized emotion information;

[1824] means for transmitting the completed and corrected final text and the recognized emotion information to a user terminal;

[1825] A system including:

[1826] (Claim 2)

[1827] 10. The system of claim 1, further comprising means for identifying and correcting inaccurate portions of the primary converted text based on appropriate context and recognized sentiment information, if any.

[1828] (Claim 3)

[1829] The system of claim 1 further comprising means for inputting the primary converted text, surrounding contextual information, and recognized emotion information into a generative AI model means, and providing a method for the generative AI model means to correct inaccuracies based on the contextual and emotion information.

[1830] "Application example 2 when combining emotion engines"

[1831] (Claim 1)

[1832] means for obtaining speech input from a user;

[1833] a speech recognition engine means for converting the acquired speech input into text;

[1834] A generative AI model means for analyzing the context of the primary converted text and completing and correcting it;

[1835] an emotion recognition engine means for recognizing the emotion of a user from the acquired voice data;

[1836] A means for inputting the emotion information recognized by the emotion recognition engine means into a generation AI model means, analyzing the information together with the context before and after the emotion information, and performing supplementation and correction;

[1837] a means for transmitting the completed and corrected final text and emotion information to a user terminal;

[1838] A system including:

[1839] (Claim 2)

[1840] 10. The system of claim 1, further comprising means for identifying and correcting the inaccurate portion of the primary converted text based on appropriate context if the primary converted text is inaccurate.

[1841] (Claim 3)

[1842] The system of claim 1 further comprising means for inputting the primary converted text, surrounding contextual information, and recognized emotion information into a generative AI model means, and providing a method for the generative AI model means to correct inaccuracies based on the context and emotion information. [Explanation of symbols]

[1843] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for obtaining speech input from a user; a speech recognition engine means for converting the acquired speech input into text; A generative AI model means for analyzing the context of the primary converted text and for completing and correcting it; means for transmitting the completed and corrected final text to a user terminal; A system including:

2. 2. The system of claim 1, further comprising means for identifying and correcting the incorrect portion of the primary converted text based on appropriate context if the primary converted text is incorrect.

3. 2. The system of claim 1, further comprising means for inputting the primary converted text and surrounding context information to the generative AI model means, and providing a method for the generative AI model means to correct inaccuracies based on the context.

Images