System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving speech data; means for analyzing the received speech data and detecting a disconnected portion; means for applying generative artificial intelligence to complement the disconnected portion from a context based on the analyzed speech data; means for synthesizing the generated complementary data with the original speech data; and means for transmitting the synthesized speech data to another terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In online meetings and remote communications, there is a problem of audio data being interrupted due to unstable communication environments. This interruption makes it difficult to understand the conversation and hinders smooth communication. Furthermore, it is difficult to determine whether your communication environment or the other party's is poor, making it difficult to take effective measures. It is necessary to solve this audio interruption problem and improve the quality of online meetings. [Means for solving the problem]

[0005] The present invention provides a system for receiving and analyzing voice data in real time. The system includes means for analyzing the received voice data and detecting interruptions. It also includes means for applying generative artificial intelligence to fill in the interruptions from the context of the analyzed voice data. It also includes means for synthesizing the generated completed data with the original voice data and means for transmitting the synthesized voice data to another terminal. It also includes means for converting the voice data to text, completing the text data using generative artificial intelligence, and reconverting the completed text data into voice data. The system also includes means for monitoring the communication quality of the voice data in real time and automatically triggering the completion process if an interruption occurs.

[0006] "Audio data" refers to data obtained by converting audio captured through a device such as a microphone into a digital signal.

[0007] A "receiving means" is a hardware and software system for receiving audio data via a network.

[0008] "Means for analyzing" refers to the algorithms and mechanisms used to process received audio data and understand and classify its content.

[0009] The "means for detecting gaps" is a technology for identifying parts of the audio data that lack consistency or continuity.

[0010] "Context" refers to the context or meaning of a conversation or text, and is the contextual information used to complement the audio data.

[0011] "Generative artificial intelligence" refers to algorithms and systems that use machine learning and natural language processing to generate new text and audio data.

[0012] "Complementary data" refers to new audio data or text generated to fill in missing parts in the interrupted audio data.

[0013] The "means for synthesizing with audio data" is a technology for integrating the generated complementary data with the original audio data and reconstructing it as consistent, continuous audio data.

[0014] The "transmitting means" is a system for distributing the organized voice data to other terminals via a network.

[0015] The "means for converting to text" is a speech recognition technology that converts voice data into text information.

[0016] The "reconversion means" is a speech synthesis technology that converts text data back into speech data.

[0017] The "means for monitoring communication quality" is a system that monitors the quality of voice data during transmission in real time and evaluates its stability.

[0018] The "means for triggering the completion process" is a mechanism for automatically starting the completion process when a gap in the audio data is detected. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention provides a system for solving the problem of audio data interruptions in online conferences and remote communications. This system can provide continuous audio without interruptions even in an unstable communication environment. An embodiment of this system will be described in detail below.

[0041] Overall system overview

[0042] This system receives voice data in real time, analyzes the data to detect interruptions, and uses generative artificial intelligence (AI) to complete the data based on the context. The completed data is then synthesized with the original voice data, and the final completed voice data is sent to another device.

[0043] Server Processing

[0044] Receiving audio data

[0045] The server receives the voice data sent from the device in real time using protocols such as WebSocket, which minimizes conversation delays.

[0046] Analysis of audio data

[0047] Automatic speech recognition (ASR) technology is used to analyze the received voice data and detect interruptions. The analysis results are saved as text data.

[0048] Applying generative AI

[0049] The server applies generative AI (e.g., a GPT model) to fill in gaps from the context of the parsed text data. The generative AI generates natural-sounding text that fits the gaps.

[0050] Complementary data synthesis

[0051] The generated supplemental text data is converted into audio data using text-to-speech (TTS) technology, and then the supplemental audio data is synthesized with the original audio data to create a coherent, continuous audio data.

[0052] Sending complementary data

[0053] The synthesized voice data is sent to other devices in real time, using a real-time communication protocol such as WebSocket.

[0054] Terminal handling

[0055] Capture audio data

[0056] The device captures the user's voice through a microphone and then compresses and encodes the data, allowing for efficient use of network bandwidth.

[0057] Sending audio data

[0058] The captured audio data is sent to the server using an appropriate protocol (e.g., Opus codec) to ensure stable communication.

[0059] Receiving complementary data

[0060] The terminal receives the completed voice data from the server, and the received data is played back to the user without delay.

[0061] Playing audio data

[0062] The device decodes the completed audio data and plays it through the speaker, allowing the user to continue the conversation without interruption.

[0063] User operations

[0064] Users can operate the device to capture their own voice and receive continuous voice data without interruption even if the communication is interrupted. Users can also check the quality of the captured voice and request retransmission if necessary.

[0065] Specific examples

[0066] Situation: When the communication environment is poor and the speaker's voice is interrupted

[0067] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[0068] 2. The server analyzes the received audio data, detects any interruptions, and converts them into text.

[0069] 3. The server uses a generation AI to generate complementary text such as "multiple" and "different."

[0070] 4. The server converts the completed text into audio data and synthesizes it with the original audio data.

[0071] 5. The server sends the completed voice data to User B's device, and User B's device plays the completed voice.

[0072] In this way, the system can provide smooth voice communication even when communication is interrupted.

[0073] The processing flow will be explained below.

[0074] Server Processing

[0075] Step 1: Receiving audio data

[0076] The server receives audio data sent from the device in real time via WebSocket, and the received data is temporarily stored in a buffer.

[0077] Example: let audioBuffer = await receiveAudioData();

[0078] Step 2: Analyzing the audio data

[0079] The server converts the audio data in the buffer into text using ASR technology and detects gaps. This analysis must be done quickly.

[0080] Example: let transcribedText = await speechToText(audioBuffer);

[0081] Detect gaps: let gaps = detectMissingSegments(transcribedText);

[0082] Step 3: Applying generative AI

[0083] The server uses generative AI (e.g., GPT model) to generate text to fill in the gaps from the context. The AI ​​generates the completed text in a way that maintains a natural flow of dialogue.

[0084] Example: let filledText = fillGapsWithAI(transcribedText, gaps);

[0085] Step 4: Synthesis of complementary data

[0086] The server synthesizes the generated supplemental text with the original speech data using text-to-speech (TTS) technology.

[0087] Example: let synthesizedAudio = textToSpeech(filledText);

[0088] Synthesize audio data: let combinedAudio = mergeAudioStreams(audioBuffer, synthesizedAudio);

[0089] Step 5: Sending complementary data

[0090] The server transmits the synthesized complete voice data to the other terminal in real time, providing uninterrupted voice to the receiving terminal.

[0091] Example: sendAudioDataToClients(combinedAudio);

[0092] Terminal handling

[0093] Step 1: Capture audio data

[0094] The terminal captures the user's voice in real time through a microphone and compresses and encodes the data.

[0095] Example: let capturedAudio = captureAudioFromMicrophone();

[0096] Step 2: Sending audio data

[0097] The device transmits the encoded audio data to the server using the WebSocket protocol, where appropriate error checking mechanisms are implemented to ensure reliable transmission.

[0098] Example: sendAudioDataToServer(capturedAudio);

[0099] Step 3: Receiving Complementary Data

[0100] The device receives the synthesized voice data sent from the server in real time, and the received data is stored in a buffer.

[0101] Example: let receivedAudio = await receiveAudioDataFromServer();

[0102] Step 4: Playing back audio data

[0103] The device decodes the received audio data and plays it through the speaker, providing the user with uninterrupted audio.

[0104] Example: playAudioData(receivedAudio);

[0105] User operations

[0106] Step 1: Check the audio quality

[0107] Users can check the quality of the audio played on their device and manually request correction if there are any interruptions.

[0108] Example: if(audioQualityIsPoor()) { requestManualCompletion();}

[0109] Step 2: Working with ongoing conversations

[0110] The user can smoothly continue the conversation based on the completed voice data, and can also request retransmission of the incorrectly completed data if necessary.

[0111] Example: if(incorrectCompletion()) { requestResend();}

[0112] These processing steps compensate for audio interruptions during online meetings, resulting in smoother communication.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] In online meetings and remote communications, an unstable communication environment can cause interruptions in audio data, hindering smooth communication and increasing user stress. Conventional technologies lack a mechanism for automatically and in real time compensating for such interrupted audio, so an effective system that solves this problem is needed.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data to detect interrupted portions, means for applying artificial intelligence to fill in the interrupted portions based on the analyzed voice data from the context before and after, means for converting the generated supplemented data into voice data using text-to-speech technology, means for synthesizing the converted supplemented voice data with the original voice data, and means for transmitting the synthesized voice data to another terminal, thereby making it possible to provide uninterrupted, continuous voice in real time even in an unstable communication environment.

[0118] "Audio data" refers to data obtained by converting an audio signal acquired through an audio input device such as a microphone into a digital format.

[0119] "Means for receiving" refers to the technical means for obtaining audio data over a network and storing it in an appropriate data structure, typically using a communication protocol such as WebSocket.

[0120] The "analyzing means" refers to the technical means for processing the received voice data and detecting the interrupted parts. Automatic Speech Recognition (ASR) technology is typically used.

[0121] "Generative AI" is an abbreviation for generative artificial intelligence, and refers to a machine learning model for generating natural-sounding text and speech based on given input data.

[0122] "Text-to-speech technology" refers to the technical means for converting generated text data into digital audio signals. Text-to-speech (TTS) technology is commonly used.

[0123] "Synthesizing means" means the technical means for combining the converted audio data with the original audio data to create a coherent, continuous audio data.

[0124] "Transmission means" refers to the technical means for transmitting the synthesized voice data to another device in real time. Typically, a communication protocol such as WebSocket is used.

[0125] The "means for monitoring communication quality" refers to a technical means for monitoring the status of transmission and reception of voice data, and automatically triggering a compensation process if a problem such as voice interruption occurs.

[0126] This invention is a system that enables smooth voice communication even in an unstable communication environment. Specifically, it receives and analyzes voice data in real time and uses generative artificial intelligence (generative AI) to fill in any interrupted parts. The filled-in voice data is then synthesized with the original voice data and transmitted to another terminal as continuous voice. The following describes each component of the system and its specific operation.

[0127] Server Processing

[0128] The server first receives the audio data sent from the device in real time using the WebSocket protocol, a communication protocol that enables bidirectional data exchange with low latency.

[0129] The received voice data is converted into text using automatic speech recognition (ASR). This allows the server to identify the interrupted parts of the voice data. As a result of the ASR analysis, the resulting text might be something like, "The new portable charger has multiple ports...each port is different..."

[0130] Next, the server uses a generative AI (for example, a GPT-3 model) to generate text to complete the missing parts. The generated text includes completion parts such as "multiple" and "different." The following prompt sentence is used to apply the generative AI:

[0131] Example prompt sentence:

[0132] "The new portable charger has multiple ports...each port is different..."

[0133] The generated text is converted into audio data using Text-to-Speech (TTS) technology and then combined with the original audio data to create a consistent, continuous audio stream, which is then sent to other devices using the WebSocket protocol.

[0134] Terminal handling

[0135] The device captures the user's voice through a microphone, and this voice data is compressed and encoded using the Opus codec and sent to the server in a manner that efficiently uses network bandwidth.

[0136] The completed voice data sent from the server is received in real time by the device. The received voice data is decoded using the Opus codec and played back from the speaker. This allows users to continue a smooth conversation with continuous audio, regardless of communication interruptions.

[0137] User operations

[0138] Users can operate the device to capture their own voice and receive the completed voice data even if communication is interrupted. Users can also check the quality of the completed voice and request retransmission if necessary.

[0139] Specific examples

[0140] For example, consider a situation where a speaker's voice is interrupted due to a poor communication environment. User A's device sends the interrupted voice, "The new portable charger has multiple... each port is different..." to the server. The server analyzes the received voice data, detects the interrupted part, and converts it into text. Next, the server uses a generative AI (e.g., GPT-3 model) to generate complementary text such as "multiple" and "different." The server converts the complementary text into voice data and synthesizes it with the original voice data. Finally, the server sends the completed voice data to User B's device, which plays back the completed voice.

[0141] In this way, the system can provide smooth voice communication even when communication is interrupted.

[0142] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0143] Step 1: Capture audio data

[0144] Input: User's voice

[0145] Output: Captured audio data

[0146] The device uses a microphone to capture the user's voice, and the captured voice data is compressed and encoded in real time using the Opus codec, reducing the data size and allowing for efficient use of network bandwidth.

[0147] Step 2: Sending audio data

[0148] Input: Captured audio data

[0149] Output: Audio data sent to the server

[0150] The device sends the encoded audio data to the server using the WebSocket protocol, which allows for bidirectional, low-latency communication.

[0151] Step 3: Receiving audio data

[0152] Input: Audio data sent from the device

[0153] Output: Audio data stored on the server

[0154] The server receives the audio data sent from the device in real time. The received audio data is temporarily stored in a buffer. The WebSocket protocol is used to efficiently receive the data.

[0155] Step 4: Analyzing the audio data

[0156] Input: Audio data stored on the server

[0157] Output: Parsed text data

[0158] The server uses ASR (Automatic Speech Recognition) technology to analyze the received voice data and convert it into text data. During this process, the entire voice data is analyzed in detail to detect any interruptions. For example, text data such as "The new portable charger has multiple ports... each port is different..." can be obtained.

[0159] Step 5: Applying generative AI

[0160] Input: Parsed text data

[0161] Output: Generated completed text data

[0162] The server uses generative AI (e.g., the GPT-3 model) based on the analyzed text data to complete the missing parts, generating natural-looking completed text appropriate to the context. The generated text includes appropriate completed text such as "multiple" and "different."

[0163] Step 6: Converting the completed text to speech

[0164] Input: Generated completed text data

[0165] Output: Interpolated audio data

[0166] The server converts the generated supplemental text data into voice data using text-to-speech (TTS) technology. For example, the supplemental text "Multiple ports have different specifications" is converted into natural voice data.

[0167] Step 7: Synthesizing the audio data

[0168] Input: Original audio data and interpolated audio data

[0169] Output: Synthesized continuous audio data

[0170] The server then synthesizes the converted speech data with the original speech data, creating a consistent, continuous speech stream. The speech data is seamlessly integrated into the original speech data, creating a seamless audio stream.

[0171] Step 8: Sending the completed audio data

[0172] Input: Synthesized continuous speech data

[0173] Output: The completed audio data sent to the device.

[0174] The server then transmits the synthesized continuous audio data to other devices using the WebSocket protocol, which allows for efficient transmission of audio data in real time.

[0175] Step 9: Receiving the completed audio data

[0176] Input: Completed audio data sent from the server

[0177] Output: Completed audio data saved on the device

[0178] The terminal receives the completed voice data sent from the server in real time, and the received voice data is temporarily stored in a buffer.

[0179] Step 10: Decode and play the audio data

[0180] Input: Completed voice data saved on the device

[0181] Output: A continuous sound that is played to the user

[0182] The device then decodes the received audio data using the Opus codec and plays it back through the speaker, allowing the user to hear continuous audio without interruptions.

[0183] These steps enable the system to achieve smooth voice communication regardless of the instability of the communication environment.

[0184] (Application example 1)

[0185] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0186] The present invention aims to provide continuous audio without interruptions in audio data even in unstable communication environments during online conferences and remote communications. In the field of security monitoring, it is also an issue to provide a monitor with continuous audio without any sense of interruption even when audio data from a surveillance camera or sensor is interrupted.

[0187] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0188] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data to detect interrupted portions, means for applying artificial intelligence to fill in the interrupted portions based on the analyzed voice data from the context, means for synthesizing the generated completed voice data with the original voice data, means for transmitting the completed voice data to another terminal, and means for providing the completed voice data to the user's playback device in real time. This enables uninterrupted voice communication even in an environment with unstable communication. Furthermore, the monitor can monitor with uninterrupted, continuous voice.

[0189] "Audio data" is a digital representation of a sound signal captured by a microphone or other input device.

[0190] The "receiving means" is a device or module that has the function of acquiring audio data from another device or system.

[0191] "Means for analyzing" means any device or software having the processing capability to decompose received audio data and determine gaps and other characteristics.

[0192] A "gap" is a portion of the audio data where information is missing.

[0193] "Generative AI" is a collection of algorithms, such as machine learning models and neural networks, that are used to complement voice data.

[0194] "Complementary data" refers to a portion of the audio data that has been filled in with information generated by artificial intelligence to fill in the gaps.

[0195] The "synthesizing means" refers to a device or software that has the function of integrating the original audio data and the generated complementary data into one continuous audio data.

[0196] A "playback device" is a device such as a speaker or earphone that allows the user to listen to the synthesized audio data.

[0197] "Communication quality" refers to the state of the network environment when voice data is transmitted continuously without interruption.

[0198] A "triggering means" is a mechanism or function that automatically starts processing when a specific condition is met.

[0199] The present invention provides a system for preventing interruptions in voice data even in unstable communication environments during online conferences and remote communications. This system aims to improve security monitoring and voice call quality. Specific embodiments of this system are described below.

[0200] Overall system overview

[0201] The system receives and analyzes voice data in real time, detects interruptions, applies generative AI based on the context to complete the audio, and finally synthesizes the completed audio data and transmits it to other devices.

[0202] Server Processing

[0203] Receiving audio data

[0204] The server receives audio data sent from the terminal or security monitoring device in real time using the WebSocket protocol, allowing the server to minimize delays in conversation and monitoring.

[0205] Analysis of audio data

[0206] The received voice data is analyzed using automatic speech recognition (ASR). As a result of the analysis, any gaps are detected and the data is converted into text format.

[0207] Applying generative AI

[0208] The server applies generative AI (e.g., the GPT-4 model) based on the context of the analyzed text data to generate text to naturally complete the gaps.

[0209] Complementary data synthesis

[0210] The generated supplemental text data is converted into audio data using text-to-speech (TTS) technology and then combined with the original audio data, creating a continuous, uninterrupted audio stream.

[0211] Sending complementary data

[0212] The synthesized voice data is transmitted in real time to another device or monitoring device using the WebSocket protocol.

[0213] Terminal handling

[0214] Capture audio data

[0215] The device can use a microphone to capture the user's voice and compress and encode the voice data to use network bandwidth efficiently.

[0216] Sending audio data

[0217] The captured audio data is sent to the server, again using the WebSocket protocol.

[0218] Receiving complementary data

[0219] The terminal receives the complemented voice data from the server in real time.

[0220] Playing audio data

[0221] The device decodes the interpolated audio data and plays it back to the user through a speaker or earphones, allowing the user to have a conversation or monitor with uninterrupted audio.

[0222] User operations

[0223] Users can operate the device to capture their own voice and receive continuous voice data without interruption even if the communication is interrupted. Users can also check the quality of the captured voice and request retransmission if necessary.

[0224] Specific examples

[0225] Situation: When the communication environment is poor and the speaker's voice is interrupted

[0226] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[0227] 2. The server analyzes the received audio data, detects any interruptions, and converts them into text.

[0228] 3. The server uses a generation AI to generate complementary text such as "multiple" and "different."

[0229] 4. The server converts the completed text into audio data and synthesizes it with the original audio data.

[0230] 5. The server sends the completed voice data to User B's device, and User B's device plays the completed voice.

[0231] Prompt Sentence Examples

[0232] Audio data from ears: Warning, something on premises. ___Intrusion detected. ___

[0233] Generate context-aligned audio data.

[0234] In this way, the system can provide smooth voice communication and security monitoring even when communication is interrupted.

[0235] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0236] Step 1:

[0237] The server receives audio data from a terminal or security monitoring device in real time using the WebSocket protocol. The input is audio data, which the server stores in a buffer. The output is the stored audio data.

[0238] Step 2:

[0239] The server analyzes the received voice data using ASR technology. The input is the stored voice data, and the analysis converts the voice data into text format. The output is the text data that is the analysis result.

[0240] Step 3:

[0241] The server inspects the parsed text data and detects gaps. The input is the text data, and the detection process identifies the gaps. The output is the text data with the gaps indicated.

[0242] Step 4:

[0243] The server applies generative AI (GPT-4 model) based on the context of the analyzed text data to generate text to complete the gap. The input is the text data showing the gap, and the output is the completed text data. In this process, the following prompt sentence is used as input:

[0244] Audio data from ears: Warning, something on premises. ___Intrusion detected. ___

[0245] Generate context-aligned audio data.

[0246] Step 5:

[0247] The server converts the generated supplemented text data into audio data using text-to-speech (TTS) technology, with the input being the supplemented text data and the output being the supplemented audio data.

[0248] Step 6:

[0249] The server synthesizes the generated complementary speech data with the original speech data to create a single continuous speech data. The input is the complementary speech data and the original speech data, and the output is the synthesized continuous speech data.

[0250] Step 7:

[0251] The server transmits the synthesized voice data to another terminal or monitoring device in real time, where the input is the synthesized continuous voice data and the output is the voice data transmitted to the receiving terminal.

[0252] Step 8:

[0253] The terminal receives the completed voice data from the server in real time. The input is the voice data sent from the server, and the output is the voice data stored in the terminal's buffer.

[0254] Step 9:

[0255] The device decodes the interpolated audio data and plays it back to the user through a speaker or earphone. The input is the audio data stored in the buffer, and the output is the audio played back through the speaker or earphone. This allows the user to have a conversation or monitor with uninterrupted audio.

[0256] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0257] The present invention combines a system for solving the problem of voice data interruptions during online conferences and remote communication with an emotion engine that recognizes the user's emotions. This system provides uninterrupted, continuous voice even in unstable communication environments, and also enables emotion-sensitive communication. An embodiment of this system will be described in detail below.

[0258] Overall system overview

[0259] This system receives voice data in real time, analyzes it to detect interruptions, and uses generative artificial intelligence (AI) to complete the speech based on the context. The generated completed data is then combined with the original voice data, and the completed speech data is finally sent to another device. Furthermore, it uses an emotion engine to recognize the user's emotions and provides appropriate completion based on those emotions.

[0260] Server Processing

[0261] Receiving audio data

[0262] The server receives the voice data sent from the terminal in real time and temporarily stores the received data in a buffer.

[0263] Analysis of audio data

[0264] The server converts the audio data in the buffer into text using ASR technology and detects gaps in the audio data, which are then used in the next processing step.

[0265] Applying generative AI

[0266] The server uses generative AI (e.g., GPT model) to generate text to fill in the gaps based on the context and the emotions recognized by the emotion engine. The AI ​​generates natural-looking text that matches the flow of the conversation and the user's emotions.

[0267] Applying the Emotion Engine

[0268] The server uses an emotion engine to analyze and recognize the user's emotions from the received voice data. The emotion information recognized by the emotion engine is used to complement the generation AI.

[0269] Complementary data synthesis

[0270] The generated text is converted into speech data using text-to-speech (TTS) technology and then synthesized with the original speech data. The synthesized speech data is adjusted to ensure a natural continuation of the context and emotion.

[0271] Sending complementary data

[0272] The synthesized voice data is sent to another device in real time, allowing the user to receive uninterrupted voice.

[0273] Terminal handling

[0274] Capture audio data

[0275] The device captures the speaker's voice through a microphone, then compresses and encodes it, allowing for efficient use of network bandwidth.

[0276] Sending audio data

[0277] The captured audio data is sent to the server using an appropriate protocol (e.g. WebSocket, Opus codec) to ensure reliable transmission.

[0278] Receiving complementary data

[0279] The device receives the synthesized voice data sent from the server in real time, and the received data is temporarily stored in a buffer.

[0280] Playing audio data

[0281] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[0282] User operations

[0283] Checking Voice Quality

[0284] Users can check the quality of the audio played on their device and request corrections if there are any deficiencies or interruptions. Furthermore, if they feel the emotion recognition results are inappropriate, they can request readjustment.

[0285] Emotion-based responses

[0286] The user can continue to communicate more smoothly based on the emotions recognized by the emotion engine. The emotion engine analyzes the user's tone of voice and speaking style and sends the results to the server.

[0287] Specific examples

[0288] Situation: When the communication environment is poor and the speaker's voice is interrupted

[0289] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[0290] 2. The server analyzes the received audio data and detects gaps as text.

[0291] 3. The server analyzes User A's emotions using an emotion engine and recognizes, for example, that he is "excited."

[0292] 4. The server uses generative AI to complete words such as "multiple" and "different," and adds emotional nuance to the completed text based on the results of the emotion engine.

[0293] 5. The server converts the completed text into audio data and synthesizes it with the original audio data.

[0294] 6. The server sends the synthesized voice data to User B's device, which plays it back.

[0295] In this way, smooth and emotionally sensitive voice communication can be provided even when communication is interrupted.

[0296] The processing flow will be explained below.

[0297] Server Processing

[0298] Step 1: Receiving audio data

[0299] The server receives audio data sent from the device in real time using WebSocket. The received audio data is temporarily stored in a buffer.

[0300] Example: let audioBuffer = await receiveAudioData();

[0301] Step 2: Analyzing the audio data

[0302] The server analyzes the received voice data and converts it into text using ASR (Automatic Speech Recognition) technology. It then detects gaps in the converted text.

[0303] Example: let transcribedText = await speechToText(audioBuffer);

[0304] Detect gaps: let gaps = detectMissingSegments(transcribedText);

[0305] Step 3: Applying the Emotion Engine

[0306] The server uses an emotion engine to analyze the user's emotions from the received voice data, and the emotional information is used to complement the generative AI.

[0307] Example: let userEmotion = analyzeEmotion(audioBuffer);

[0308] Step 4: Applying generative AI

[0309] The server uses generative AI (e.g., a GPT model) to generate text to fill in the gaps based on the context and emotions recognized by the emotion engine.

[0310] Example: let filledText = fillGapsWithAI(transcribedText, gaps, userEmotion);

[0311] Step 5: Synthesizing complementary data

[0312] The server converts the generated text into speech data using text-to-speech (TTS) technology and synthesizes it with the original speech data, reconstructing it into a coherent, continuous speech data.

[0313] Example: let synthesizedAudio = textToSpeech(filledText);

[0314] Synthesize audio data: let combinedAudio = mergeAudioStreams(audioBuffer, synthesizedAudio);

[0315] Step 6: Sending complementary data

[0316] The server then transmits the synthesized voice data to other devices in real time, providing uninterrupted voice to the receiving device.

[0317] Example: sendAudioDataToClients(combinedAudio);

[0318] Terminal handling

[0319] Step 1: Capture audio data

[0320] The terminal uses a microphone to capture the user's voice in real time, then compresses and encodes the data.

[0321] Example: let capturedAudio = captureAudioFromMicrophone();

[0322] Step 2: Sending audio data

[0323] The device sends the encoded audio data to the server using the WebSocket protocol or similar, and a mechanism to ensure reliability is implemented.

[0324] Example: sendAudioDataToServer(capturedAudio);

[0325] Step 3: Receiving Complementary Data

[0326] The device receives the synthesized voice data sent from the server in real time, and the received data is stored in a buffer.

[0327] Example: let receivedAudio = await receiveAudioDataFromServer();

[0328] Step 4: Playing back audio data

[0329] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[0330] Example: playAudioData(receivedAudio);

[0331] User operations

[0332] Step 1: Check the audio quality

[0333] Users can check the quality of the audio played on their device and request corrections if there are any deficiencies or interruptions. Furthermore, if they feel the emotion recognition results are inappropriate, they can request readjustment.

[0334] Example: if(audioQualityIsPoor()) { requestManualCompletion();}

[0335] Example: if(emotionRecognitionIsIncorrect()) { requestEmotionReanalysis();}

[0336] Step 2: Respond based on emotions

[0337] Users can continue to communicate more smoothly based on the emotions recognized by the emotion engine. The emotion engine analyzes the user's tone of voice and speaking style and sends the results to the server.

[0338] Example: let userEmotion = captureEmotionDuringConversation();

[0339] Specific examples

[0340] Situation: When the communication environment is poor and the speaker's voice is interrupted

[0341] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[0342] 2. The server analyzes the received audio data, converts the audio into text, and detects gaps.

[0343] 3. The server analyzes User A's emotions using an emotion engine and recognizes, for example, that he is "excited."

[0344] 4. The server uses a generative AI to generate complementary text such as "multiple" and "different," and adds emotional nuance to the complementary text based on the results of the emotion engine.

[0345] 5. The server converts the generated supplemental text into audio data and synthesizes it with the original audio data.

[0346] 6. The server sends the synthesized voice data to User B's device, and User B's device plays the completed voice.

[0347] In this way, smooth and emotionally sensitive voice communication can be provided even when communication is interrupted.

[0348] Example 2

[0349] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0350] In conventional online conference systems and remote communication systems, voice data is often interrupted due to unstable communication environments, hindering smooth communication. Furthermore, because the system does not take into account the user's emotions, the quality of communication can be reduced by mechanical responses. To solve these problems, a system is needed that provides uninterrupted, continuous voice and communication that takes the user's emotions into consideration.

[0351] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0352] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data and detecting interrupted portions, means for applying a generative model to fill in the interrupted portions based on the analyzed voice data from the context, means for synthesizing the completed data with the original voice data, means for transmitting the synthesized voice data to another terminal, and means for analyzing the user's emotions from the received voice data and using the results in the completion process of the generative model, thereby enabling smooth voice communication that takes emotions into consideration even when communication is interrupted.

[0353] 1. "Audio data" means data that represents audio information such as conversation or music in digital form.

[0354] 2. "Means for receiving" means the technology or equipment for receiving audio data through a communications network.

[0355] 3. "Means for analyzing" means technology or devices that process received voice data to decipher and understand its content and characteristics.

[0356] 4. "Interrupted portion" refers to a portion where audio data is missing or interrupted due to an unstable communication environment or other reasons.

[0357] 5. "Means for detection" means the technology or device used to identify gaps in the analyzed audio data.

[0358] 6. A "generative model" is an algorithm or program that uses artificial intelligence techniques to generate text or speech.

[0359] 7. "Context" refers to the content of the audio data or text before and after the interrupted section, and is the information used to complete the sentence.

[0360] 8. "Completion means" refers to technologies or devices that use generative models to fill in gaps in data in a natural way.

[0361] 9. "Synthesizing means" means a technique or device that combines the generated complementary data with the original audio data to form one continuous audio signal.

[0362] 10. "Transmitting means" means the technology or device used to transmit the synthesized voice data to another terminal.

[0363] 11. "Terminal" means an electronic device for capturing, receiving, and playing audio data.

[0364] 12. "Means of analysis" refers to technology or devices that understand the content of audio data and extract specific information.

[0365] 13. "Means for analyzing emotions" refers to technology or devices that identify a user's emotions from voice data and use the results in subsequent processing.

[0366] 14. "Synthesized speech data" refers to the entire speech data to which the parts completed by the generative model have been added.

[0367] 15. "Real-time" means that data processing or communication is immediate and occurs within a time range that does not cause a delay to the user.

[0368] This invention provides a system that eliminates interruptions in voice data during online conferences and remote communications, and complements the data while taking into account the user's emotions. The system receives voice data in real time, analyzes the data to detect interruptions, and complements the data using a generative model based on the context and the user's emotions. The system also has the ability to synthesize the complemented voice data with the original voice data and transmit it to another device in real time.

[0369] Server Processing

[0370] 1. Receiving audio data

[0371] The server receives the voice data sent from the terminal in real time. The received data is temporarily stored in a buffer. The technology used here includes general network communication technology (e.g., TCP / IP).

[0372] 2. Analysis of audio data

[0373] The server uses speech recognition technology, such as the Google Speech-to-Text API, to convert the audio data in the buffer into text and detects gaps in the text data.

[0374] 3. Applying generative AI

[0375] The server uses a generative AI model (such as OpenAI's GPT-3) to generate text to fill in the gaps based on the context and the results of the emotion engine analysis. The generated text reflects natural expressions.

[0376] 4. Applying the Emotion Engine

[0377] The server uses an emotion engine to analyze and recognize the user's emotions from the received voice data. The emotion engine may be, for example, IBM Watson's Tone Analyzer. The emotion information detected by the emotion engine is used to complement the generative AI.

[0378] 5. Synthesis of complementary data

[0379] The generated text is converted into audio data using text-to-speech (TTS) technology, such as the Google Text-to-Speech API. The converted audio data is then synthesized with the original audio data to create a natural continuity.

[0380] 6. Sending supplementary data

[0381] The synthesized voice data is sent to the device in real time using a low-latency, highly reliable protocol (e.g., WebSocket).

[0382] Terminal handling

[0383] 1. Capture audio data

[0384] The device captures the speaker's voice in real time through a microphone, and the captured voice data is compressed and encoded using the Opus codec.

[0385] 2. Sending audio data

[0386] The encoded audio data is sent to the server in real time.

[0387] 3. Receiving complementary data

[0388] The synthesized voice data sent from the server is received in real time and temporarily stored in a buffer.

[0389] 4. Playing audio data

[0390] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[0391] User operations

[0392] 1. Check the audio quality

[0393] The user can check the quality of the audio played on the device and request completion if there are gaps or interruptions.

[0394] 2. Emotional responses

[0395] The user can smoothly communicate based on the emotions recognized by the emotion engine.

[0396] Prompt Sentence Examples

[0397] "Hello, how was your day?"

[0398] These concise prompts form the basis for applying generative AI models.

[0399] Examples:

[0400] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[0401] 2. The server analyzes the received voice data and converts the interrupted part into text: "The new portable charger is complex...each port is different..."

[0402] 3. The server's emotion engine recognizes that User A is "excited."

[0403] 4. The server uses the generative AI model to complement words like "multiple" and "different" and reflect the emotion of "excited," generating "The new portable charger has multiple ports that can be used in different ways."

[0404] 5. The server converts the completed text into audio data and synthesizes it with the original audio data.

[0405] 6. The server sends the synthesized voice data to User B's terminal, which receives it.

[0406] 7. When User B's device plays back the received audio data, he or she can hear the continuous audio saying, "The new portable charger has multiple ports that can be used in different ways."

[0407] In this way, the present invention smoothly compensates for interruptions in communication and realizes emotionally sensitive voice communication.

[0408] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0409] Step 1:

[0410] Capture audio data

[0411] input:

[0412] The voice the user speaks.

[0413] Specific behavior:

[0414] The device uses a microphone to capture the speaker's voice in real time, then compresses and encodes the captured voice data using the Opus codec, and temporarily stores the encoded voice data in a buffer.

[0415] output:

[0416] The encoded audio data.

[0417] Step 2:

[0418] Sending audio data

[0419] input:

[0420] The encoded audio data (output of step 1).

[0421] Specific behavior:

[0422] The device transmits the encoded audio data to the server in real time using the low-latency and highly reliable WebSocket protocol.

[0423] output:

[0424] The audio data sent to the server.

[0425] Step 3:

[0426] Receiving audio data

[0427] input:

[0428] Audio data sent from the device (output from step 2).

[0429] Specific behavior:

[0430] The server receives the voice data sent from the terminal in real time and stores it in a buffer.

[0431] output:

[0432] The audio data stored in the buffer.

[0433] Step 4:

[0434] Analysis of audio data

[0435] input:

[0436] The audio data stored in the buffer (output of step 3).

[0437] Specific behavior:

[0438] The server uses the Google Speech-to-Text API to convert the audio data in the buffer into text.

[0439] Detect gaps in speech data converted into text.

[0440] output:

[0441] Parsed audio data and gap information in text format.

[0442] Step 5:

[0443] Applying the Emotion Engine

[0444] input:

[0445] The analyzed audio data and gap information in text format (output of step 4).

[0446] Specific behavior:

[0447] The server uses an emotion engine such as IBM Watson's Tone Analyzer to analyze the user's emotions from the received text-format voice data.

[0448] The emotion information obtained by the emotion engine is stored for completion processing.

[0449] output:

[0450] Emotion analysis results.

[0451] Step 6:

[0452] Applying generative AI

[0453] input:

[0454] Context and sentiment analysis results (output of steps 4 and 5).

[0455] Specific behavior:

[0456] The server uses a generative AI model, such as OpenAI's GPT-3, to generate text to fill in the gaps based on the context and the results of the sentiment engine. The generated text contains natural, contextually appropriate expressions.

[0457] output:

[0458] The generated completion text.

[0459] Step 7:

[0460] Complementary data synthesis

[0461] input:

[0462] The generated completion text (output of step 6).

[0463] Specific behavior:

[0464] The server converts the generated text into speech using TTS technology such as the Google Text-to-Speech API, and then synthesizes the generated speech with the original speech to create a natural continuity.

[0465] output:

[0466] Synthesized voice data.

[0467] Step 8:

[0468] Sending complementary data

[0469] input:

[0470] The synthesized speech data (output of step 7).

[0471] Specific behavior:

[0472] The server transmits the synthesized voice data to the device in real time, again using the low-latency and highly reliable WebSocket protocol.

[0473] output:

[0474] Synthesized voice data sent to the device.

[0475] Step 9:

[0476] Receiving complementary data

[0477] input:

[0478] Synthesized speech data sent from the server (output of step 8).

[0479] Specific behavior:

[0480] The terminal receives the synthesized voice data sent from the server in real time and temporarily stores it in a buffer.

[0481] output:

[0482] The synthesized speech data stored in the buffer.

[0483] Step 10:

[0484] Playing audio data

[0485] input:

[0486] The synthesized speech data stored in the buffer (output of step 9).

[0487] Specific behavior:

[0488] The device decodes the received synthesized voice data and plays it through the speaker, allowing the user to hear the voice without interruption.

[0489] output:

[0490] The audio played.

[0491] (Application example 2)

[0492] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0493] Conventional voice communication systems suffer from the problem of voice interruptions when the communication environment is unstable. In particular, when interacting with customers in virtual stores, voice interruptions can be a significant factor in impairing the customer experience. Furthermore, conventional systems lack the ability to recognize user emotions, making it difficult to respond naturally and with consideration for the user's emotions. To address these issues, a system is needed that can compensate for communication interruptions and provide appropriate responses based on the user's emotions.

[0494] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0495] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data and detecting interrupted portions, means for applying a generative model to fill in the interrupted portions based on the analyzed voice data from the context and the recognized emotion, means for synthesizing the generated completed data with the original voice data, means for transmitting the synthesized voice data to another terminal, and an emotion engine for recognizing emotions. This enables smooth voice communication that takes emotion into consideration even when communication is interrupted.

[0496] "Voice data" refers to data obtained by converting the user's voice into digital form.

[0497] "Means for analyzing" refers to the processes and techniques for converting received voice data into text data and identifying gaps and voice characteristics.

[0498] A "generative model" is an artificial intelligence model trained on a large dataset and capable of automatically generating text and speech.

[0499] An "emotion engine" is a technology that analyzes emotions from a user's voice and recognizes their emotional state.

[0500] "Means of synthesis" refers to technology that naturally blends supplemented text or audio data with the original audio data.

[0501] "Communication quality" refers to the state of the network environment that allows voice data to be transmitted smoothly without interruption.

[0502] A "virtual store" refers to a virtual sales environment that provides products and services via the Internet.

[0503] "Communication with customers" refers to the exchange of voice and text between customer service staff and customers in a virtual store.

[0504] A system for implementing this invention includes a series of processes for receiving, analyzing, and completing voice data, and providing an appropriate response based on emotion. A specific method for implementing this system is described below.

[0505] System configuration

[0506] 1. Receiving audio data

[0507] The server receives the user's voice data in real time, and stores it in a temporary buffer. The hardware used here is a smartphone, smart glasses, or a head-mounted display, and the software uses WebSocket and an appropriate codec.

[0508] 2. Analysis of audio data

[0509] The server converts the received voice data into text using speech recognition technology (e.g., Google Speech-to-Text API), detects gaps in the converted text, and uses the analysis results in the next processing step. It also recognizes emotions using an emotion engine (e.g., IBM Watson Tone Analyzer).

[0510] 3. Applying generative AI

[0511] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate appropriate completion text based on context and sentiment information. Specifically, it uses the following example prompt:

[0512] Audio data: "The feature of this product is its long... different... battery life."

[0513] Emotion Engine: "Passionate"

[0514] Prompt: "Complete the following broken text based on context and sentiment: 'This product's feature is its long... different... battery life.'"

[0515] 4. Synthesis of complementary data

[0516] The completed text is converted to audio data (e.g., Google Text-to-Speech API) and then combined with the original audio data, resulting in a seamless, natural-sounding speech.

[0517] 5. Sending complementary data

[0518] The synthesized voice data is sent in real time to the customer's device, where it is decoded and played through a speaker.

[0519] Specific examples

[0520] Situation: The communication environment becomes poor while serving a customer in a virtual store, causing the customer service representative's voice to be interrupted.

[0521] 1. Customer Service Representative: "The special feature of this product is its long... different... battery life."

[0522] 2. Server: Analyzes the audio data and detects interruptions. At the same time, the emotion engine recognizes passionate emotions.

[0523] 3. Server: The generation AI generates complementary text such as "long duration" and "difference," adding emotional nuance.

[0524] 4. Server: The completed text is converted into audio data and combined with the original audio data.

[0525] 5. Server: Sends the synthesized voice data to the customer's device and plays the voice without interruption.

[0526] This system allows smooth communication with customers in virtual stores even in the event of a communication interruption, improving the customer experience.

[0527] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0528] Step 1:

[0529] Receiving audio data

[0530] The server receives the audio data sent from the device in real time. This audio data, which is input from the device, is temporarily stored in a buffer. Specifically, the server receives the audio data using WebSocket and buffers it to prevent data loss.

[0531] Step 2:

[0532] Analysis of audio data

[0533] The server converts the buffered audio data into text using speech recognition technology (e.g., Google Speech-to-Text API). The audio data is used as input and text data is generated as output. The specific operation here is to send the audio file to the speech recognition API and analyze the returned text.

[0534] Step 3:

[0535] Gap detection and emotion recognition

[0536] Detect discontinuities in the analyzed text data and simultaneously recognize emotions using an emotion engine (e.g., IBM Watson Tone Analyzer). The input data is the converted text, and the output is information about the discontinuities and emotion information. Specifically, discontinuities in the text are detected, and the text is input to the emotion analysis engine to obtain emotion labels.

[0537] Step 4:

[0538] Complemented by generative AI

[0539] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate text that completes the missing part based on context and emotional information. The input is the surrounding text and emotional information, and the output is the completed text. A specific example includes providing the following prompt to the generative AI:

[0540] Audio data: "The feature of this product is its long... different... battery life."

[0541] Emotion Engine: "Passionate"

[0542] Prompt: "Complete the following broken text based on context and sentiment: 'This product's feature is its long... different... battery life.'"

[0543] Step 5:

[0544] Complementary data synthesis

[0545] The completed text is converted back into audio data (e.g., Google Text-to-Speech API) and then synthesized with the original audio data. The input is the completed text and the original audio data, and the output is the synthesized audio data. Specifically, the completed text is sent to the text-to-speech API, which generates audio and then combines it with the original audio data.

[0546] Step 6:

[0547] Sending complementary data

[0548] The server transmits the synthesized voice data to other devices in real time. The input is the synthesized voice data, and the output is the retransmitted voice data. Specifically, the device receives the synthesized voice using WebSocket and plays it back in real time.

[0549] Step 7:

[0550] Playing audio

[0551] The device decodes the received audio data and plays it back to the user through the speaker. The input is the received audio data, and the output is the audio from the speaker. Specifically, the device sends the decoded audio data to the speaker and plays it back in a format that reaches the human ear.

[0552] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0553] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0554] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0555] [Second embodiment]

[0556] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0557] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0558] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0559] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0560] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0561] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0562] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0563] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0564] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0565] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0566] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0567] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0568] The present invention provides a system for solving the problem of audio data interruptions in online conferences and remote communications. This system can provide continuous audio without interruptions even in an unstable communication environment. An embodiment of this system will be described in detail below.

[0569] Overall system overview

[0570] This system receives voice data in real time, analyzes the data to detect interruptions, and uses generative artificial intelligence (AI) to complete the data based on the context. The completed data is then synthesized with the original voice data, and the final completed voice data is sent to another device.

[0571] Server Processing

[0572] Receiving audio data

[0573] The server receives the voice data sent from the device in real time using protocols such as WebSocket, which minimizes conversation delays.

[0574] Analysis of audio data

[0575] Automatic speech recognition (ASR) technology is used to analyze the received voice data and detect interruptions. The analysis results are saved as text data.

[0576] Applying generative AI

[0577] The server applies generative AI (e.g., a GPT model) to fill in gaps from the context of the parsed text data. The generative AI generates natural-sounding text that fits the gaps.

[0578] Complementary data synthesis

[0579] The generated supplemental text data is converted into audio data using text-to-speech (TTS) technology, and then the supplemental audio data is synthesized with the original audio data to create a coherent, continuous audio data.

[0580] Sending complementary data

[0581] The synthesized voice data is sent to other devices in real time, using a real-time communication protocol such as WebSocket.

[0582] Terminal handling

[0583] Capture audio data

[0584] The device captures the user's voice through a microphone and then compresses and encodes the data, allowing for efficient use of network bandwidth.

[0585] Sending audio data

[0586] The captured audio data is sent to the server using an appropriate protocol (e.g., Opus codec) to ensure stable communication.

[0587] Receiving complementary data

[0588] The terminal receives the completed voice data from the server, and the received data is played back to the user without delay.

[0589] Playing audio data

[0590] The device decodes the completed audio data and plays it through the speaker, allowing the user to continue the conversation without interruption.

[0591] User operations

[0592] Users can operate the device to capture their own voice and receive continuous voice data without interruption even if the communication is interrupted. Users can also check the quality of the captured voice and request retransmission if necessary.

[0593] Specific examples

[0594] Situation: When the communication environment is poor and the speaker's voice is interrupted

[0595] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[0596] 2. The server analyzes the received audio data, detects any interruptions, and converts them into text.

[0597] 3. The server uses a generation AI to generate complementary text such as "multiple" and "different."

[0598] 4. The server converts the completed text into audio data and synthesizes it with the original audio data.

[0599] 5. The server sends the completed voice data to User B's device, and User B's device plays the completed voice.

[0600] In this way, the system can provide smooth voice communication even when communication is interrupted.

[0601] The processing flow will be explained below.

[0602] Server Processing

[0603] Step 1: Receiving audio data

[0604] The server receives audio data sent from the device in real time via WebSocket, and the received data is temporarily stored in a buffer.

[0605] Example: let audioBuffer = await receiveAudioData();

[0606] Step 2: Analyzing the audio data

[0607] The server converts the audio data in the buffer into text using ASR technology and detects gaps. This analysis must be done quickly.

[0608] Example: let transcribedText = await speechToText(audioBuffer);

[0609] Detect gaps: let gaps = detectMissingSegments(transcribedText);

[0610] Step 3: Applying generative AI

[0611] The server uses generative AI (e.g., GPT model) to generate text to fill in the gaps from the context. The AI ​​generates the completed text in a way that maintains a natural flow of dialogue.

[0612] Example: let filledText = fillGapsWithAI(transcribedText, gaps);

[0613] Step 4: Synthesis of complementary data

[0614] The server synthesizes the generated supplemental text with the original speech data using text-to-speech (TTS) technology.

[0615] Example: let synthesizedAudio = textToSpeech(filledText);

[0616] Synthesize audio data: let combinedAudio = mergeAudioStreams(audioBuffer, synthesizedAudio);

[0617] Step 5: Sending complementary data

[0618] The server transmits the synthesized complete voice data to the other terminal in real time, providing uninterrupted voice to the receiving terminal.

[0619] Example: sendAudioDataToClients(combinedAudio);

[0620] Terminal handling

[0621] Step 1: Capture audio data

[0622] The terminal captures the user's voice in real time through a microphone and compresses and encodes the data.

[0623] Example: let capturedAudio = captureAudioFromMicrophone();

[0624] Step 2: Sending audio data

[0625] The device transmits the encoded audio data to the server using the WebSocket protocol, where appropriate error checking mechanisms are implemented to ensure reliable transmission.

[0626] Example: sendAudioDataToServer(capturedAudio);

[0627] Step 3: Receiving Complementary Data

[0628] The device receives the synthesized voice data sent from the server in real time, and the received data is stored in a buffer.

[0629] Example: let receivedAudio = await receiveAudioDataFromServer();

[0630] Step 4: Playing back audio data

[0631] The device decodes the received audio data and plays it through the speaker, providing the user with uninterrupted audio.

[0632] Example: playAudioData(receivedAudio);

[0633] User operations

[0634] Step 1: Check the audio quality

[0635] Users can check the quality of the audio played on their device and manually request correction if there are any interruptions.

[0636] Example: if(audioQualityIsPoor()) { requestManualCompletion();}

[0637] Step 2: Working with ongoing conversations

[0638] The user can smoothly continue the conversation based on the completed voice data, and can also request retransmission of the incorrectly completed data if necessary.

[0639] Example: if(incorrectCompletion()) { requestResend();}

[0640] These processing steps compensate for audio interruptions during online meetings, resulting in smoother communication.

[0641] Example 1

[0642] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0643] In online meetings and remote communications, an unstable communication environment can cause interruptions in audio data, hindering smooth communication and increasing user stress. Conventional technologies lack a mechanism for automatically and in real time compensating for such interrupted audio, so an effective system that solves this problem is needed.

[0644] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0645] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data to detect interrupted portions, means for applying artificial intelligence to fill in the interrupted portions based on the analyzed voice data from the context before and after, means for converting the generated supplemented data into voice data using text-to-speech technology, means for synthesizing the converted supplemented voice data with the original voice data, and means for transmitting the synthesized voice data to another terminal, thereby making it possible to provide uninterrupted, continuous voice in real time even in an unstable communication environment.

[0646] "Audio data" refers to data obtained by converting an audio signal acquired through an audio input device such as a microphone into a digital format.

[0647] "Means for receiving" refers to the technical means for obtaining audio data over a network and storing it in an appropriate data structure, typically using a communication protocol such as WebSocket.

[0648] The "analyzing means" refers to the technical means for processing the received voice data and detecting the interrupted parts. Automatic Speech Recognition (ASR) technology is typically used.

[0649] "Generative AI" is an abbreviation for generative artificial intelligence, and refers to a machine learning model for generating natural-sounding text and speech based on given input data.

[0650] "Text-to-speech technology" refers to the technical means for converting generated text data into digital audio signals. Text-to-speech (TTS) technology is commonly used.

[0651] "Synthesizing means" means the technical means for combining the converted audio data with the original audio data to create a coherent, continuous audio data.

[0652] "Transmission means" refers to the technical means for transmitting the synthesized voice data to another device in real time. Typically, a communication protocol such as WebSocket is used.

[0653] The "means for monitoring communication quality" refers to a technical means for monitoring the status of transmission and reception of voice data, and automatically triggering a compensation process if a problem such as voice interruption occurs.

[0654] This invention is a system that enables smooth voice communication even in an unstable communication environment. Specifically, it receives and analyzes voice data in real time and uses generative artificial intelligence (generative AI) to fill in any interrupted parts. The filled-in voice data is then synthesized with the original voice data and transmitted to another terminal as continuous voice. The following describes each component of the system and its specific operation.

[0655] Server Processing

[0656] The server first receives the audio data sent from the device in real time using the WebSocket protocol, a communication protocol that enables bidirectional data exchange with low latency.

[0657] The received voice data is converted into text using automatic speech recognition (ASR). This allows the server to identify the interrupted parts of the voice data. As a result of the ASR analysis, the resulting text might be something like, "The new portable charger has multiple ports...each port is different..."

[0658] Next, the server uses a generative AI (for example, a GPT-3 model) to generate text to complete the missing parts. The generated text includes completion parts such as "multiple" and "different." The following prompt sentence is used to apply the generative AI:

[0659] Example prompt sentence:

[0660] "The new portable charger has multiple ports...each port is different..."

[0661] The generated text is converted into audio data using Text-to-Speech (TTS) technology and then combined with the original audio data to create a consistent, continuous audio stream, which is then sent to other devices using the WebSocket protocol.

[0662] Terminal handling

[0663] The device captures the user's voice through a microphone, and this voice data is compressed and encoded using the Opus codec and sent to the server in a manner that efficiently uses network bandwidth.

[0664] The completed voice data sent from the server is received in real time by the device. The received voice data is decoded using the Opus codec and played back from the speaker. This allows users to continue a smooth conversation with continuous audio, regardless of communication interruptions.

[0665] User operations

[0666] Users can operate the device to capture their own voice and receive the completed voice data even if communication is interrupted. Users can also check the quality of the completed voice and request retransmission if necessary.

[0667] Specific examples

[0668] For example, consider a situation where a speaker's voice is interrupted due to a poor communication environment. User A's device sends the interrupted voice, "The new portable charger has multiple... each port is different..." to the server. The server analyzes the received voice data, detects the interrupted part, and converts it into text. Next, the server uses a generative AI (e.g., GPT-3 model) to generate complementary text such as "multiple" and "different." The server converts the complementary text into voice data and synthesizes it with the original voice data. Finally, the server sends the completed voice data to User B's device, which plays back the completed voice.

[0669] In this way, the system can provide smooth voice communication even when communication is interrupted.

[0670] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0671] Step 1: Capture audio data

[0672] Input: User's voice

[0673] Output: Captured audio data

[0674] The device uses a microphone to capture the user's voice, and the captured voice data is compressed and encoded in real time using the Opus codec, reducing the data size and allowing for efficient use of network bandwidth.

[0675] Step 2: Sending audio data

[0676] Input: Captured audio data

[0677] Output: Audio data sent to the server

[0678] The device sends the encoded audio data to the server using the WebSocket protocol, which allows for bidirectional, low-latency communication.

[0679] Step 3: Receiving audio data

[0680] Input: Audio data sent from the device

[0681] Output: Audio data stored on the server

[0682] The server receives the audio data sent from the device in real time. The received audio data is temporarily stored in a buffer. The WebSocket protocol is used to efficiently receive the data.

[0683] Step 4: Analyzing the audio data

[0684] Input: Audio data stored on the server

[0685] Output: Parsed text data

[0686] The server uses ASR (Automatic Speech Recognition) technology to analyze the received voice data and convert it into text data. During this process, the entire voice data is analyzed in detail to detect any interruptions. For example, text data such as "The new portable charger has multiple ports... each port is different..." can be obtained.

[0687] Step 5: Applying generative AI

[0688] Input: Parsed text data

[0689] Output: Generated completed text data

[0690] The server uses generative AI (e.g., the GPT-3 model) based on the analyzed text data to complete the missing parts, generating natural-looking completed text appropriate to the context. The generated text includes appropriate completed text such as "multiple" and "different."

[0691] Step 6: Converting the completed text to speech

[0692] Input: Generated completed text data

[0693] Output: Interpolated audio data

[0694] The server converts the generated supplemental text data into voice data using text-to-speech (TTS) technology. For example, the supplemental text "Multiple ports have different specifications" is converted into natural voice data.

[0695] Step 7: Synthesizing the audio data

[0696] Input: Original audio data and interpolated audio data

[0697] Output: Synthesized continuous audio data

[0698] The server then synthesizes the converted speech data with the original speech data, creating a consistent, continuous speech stream. The speech data is seamlessly integrated into the original speech data, creating a seamless audio stream.

[0699] Step 8: Sending the completed audio data

[0700] Input: Synthesized continuous speech data

[0701] Output: The completed audio data sent to the device.

[0702] The server then transmits the synthesized continuous audio data to other devices using the WebSocket protocol, which allows for efficient transmission of audio data in real time.

[0703] Step 9: Receiving the completed audio data

[0704] Input: Completed audio data sent from the server

[0705] Output: Completed audio data saved on the device

[0706] The terminal receives the completed voice data sent from the server in real time, and the received voice data is temporarily stored in a buffer.

[0707] Step 10: Decode and play the audio data

[0708] Input: Completed voice data saved on the device

[0709] Output: A continuous sound that is played to the user

[0710] The device then decodes the received audio data using the Opus codec and plays it back through the speaker, allowing the user to hear continuous audio without interruptions.

[0711] These steps enable the system to achieve smooth voice communication regardless of the instability of the communication environment.

[0712] (Application example 1)

[0713] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0714] The present invention aims to provide continuous audio without interruptions in audio data even in unstable communication environments during online conferences and remote communications. In the field of security monitoring, it is also an issue to provide a monitor with continuous audio without any sense of interruption even when audio data from a surveillance camera or sensor is interrupted.

[0715] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0716] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data to detect interrupted portions, means for applying artificial intelligence to fill in the interrupted portions based on the analyzed voice data from the context, means for synthesizing the generated completed voice data with the original voice data, means for transmitting the completed voice data to another terminal, and means for providing the completed voice data to the user's playback device in real time. This enables uninterrupted voice communication even in an environment with unstable communication. Furthermore, the monitor can monitor with uninterrupted, continuous voice.

[0717] "Audio data" is a digital representation of a sound signal captured by a microphone or other input device.

[0718] The "receiving means" is a device or module that has the function of acquiring audio data from another device or system.

[0719] "Means for analyzing" means any device or software having the processing capability to decompose received audio data and determine gaps and other characteristics.

[0720] A "gap" is a portion of the audio data where information is missing.

[0721] "Generative AI" is a collection of algorithms, such as machine learning models and neural networks, that are used to complement voice data.

[0722] "Complementary data" refers to a portion of the audio data that has been filled in with information generated by artificial intelligence to fill in the gaps.

[0723] The "synthesizing means" refers to a device or software that has the function of integrating the original audio data and the generated complementary data into one continuous audio data.

[0724] A "playback device" is a device such as a speaker or earphone that allows the user to listen to the synthesized audio data.

[0725] "Communication quality" refers to the state of the network environment when voice data is transmitted continuously without interruption.

[0726] A "triggering means" is a mechanism or function that automatically starts processing when a specific condition is met.

[0727] The present invention provides a system for preventing interruptions in voice data even in unstable communication environments during online conferences and remote communications. This system aims to improve security monitoring and voice call quality. Specific embodiments of this system are described below.

[0728] Overall system overview

[0729] The system receives and analyzes voice data in real time, detects interruptions, applies generative AI based on the context to complete the audio, and finally synthesizes the completed audio data and transmits it to other devices.

[0730] Server Processing

[0731] Receiving audio data

[0732] The server receives audio data sent from the terminal or security monitoring device in real time using the WebSocket protocol, allowing the server to minimize delays in conversation and monitoring.

[0733] Analysis of audio data

[0734] The received voice data is analyzed using automatic speech recognition (ASR). As a result of the analysis, any gaps are detected and the data is converted into text format.

[0735] Applying generative AI

[0736] The server applies generative AI (e.g., the GPT-4 model) based on the context of the analyzed text data to generate text to naturally complete the gaps.

[0737] Complementary data synthesis

[0738] The generated supplemental text data is converted into audio data using text-to-speech (TTS) technology and then combined with the original audio data, creating a continuous, uninterrupted audio stream.

[0739] Sending complementary data

[0740] The synthesized voice data is transmitted in real time to another device or monitoring device using the WebSocket protocol.

[0741] Terminal handling

[0742] Capture audio data

[0743] The device can use a microphone to capture the user's voice and compress and encode the voice data to use network bandwidth efficiently.

[0744] Sending audio data

[0745] The captured audio data is sent to the server, again using the WebSocket protocol.

[0746] Receiving complementary data

[0747] The terminal receives the complemented voice data from the server in real time.

[0748] Playing audio data

[0749] The device decodes the interpolated audio data and plays it back to the user through a speaker or earphones, allowing the user to have a conversation or monitor with uninterrupted audio.

[0750] User operations

[0751] Users can operate the device to capture their own voice and receive continuous voice data without interruption even if the communication is interrupted. Users can also check the quality of the captured voice and request retransmission if necessary.

[0752] Specific examples

[0753] Situation: When the communication environment is poor and the speaker's voice is interrupted

[0754] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[0755] 2. The server analyzes the received audio data, detects any interruptions, and converts them into text.

[0756] 3. The server uses a generation AI to generate complementary text such as "multiple" and "different."

[0757] 4. The server converts the completed text into audio data and synthesizes it with the original audio data.

[0758] 5. The server sends the completed voice data to User B's device, and User B's device plays the completed voice.

[0759] Prompt Sentence Examples

[0760] Audio data from ears: Warning, something on premises. ___Intrusion detected. ___

[0761] Generate context-aligned audio data.

[0762] In this way, the system can provide smooth voice communication and security monitoring even when communication is interrupted.

[0763] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0764] Step 1:

[0765] The server receives audio data from a terminal or security monitoring device in real time using the WebSocket protocol. The input is audio data, which the server stores in a buffer. The output is the stored audio data.

[0766] Step 2:

[0767] The server analyzes the received voice data using ASR technology. The input is the stored voice data, and the analysis converts the voice data into text format. The output is the text data that is the analysis result.

[0768] Step 3:

[0769] The server inspects the parsed text data and detects gaps. The input is the text data, and the detection process identifies the gaps. The output is the text data with the gaps indicated.

[0770] Step 4:

[0771] The server applies generative AI (GPT-4 model) based on the context of the analyzed text data to generate text to complete the gap. The input is the text data showing the gap, and the output is the completed text data. In this process, the following prompt sentence is used as input:

[0772] Audio data from ears: Warning, something on premises. ___Intrusion detected. ___

[0773] Generate context-aligned audio data.

[0774] Step 5:

[0775] The server converts the generated supplemented text data into audio data using text-to-speech (TTS) technology, with the input being the supplemented text data and the output being the supplemented audio data.

[0776] Step 6:

[0777] The server synthesizes the generated complementary speech data with the original speech data to create a single continuous speech data. The input is the complementary speech data and the original speech data, and the output is the synthesized continuous speech data.

[0778] Step 7:

[0779] The server transmits the synthesized voice data to another terminal or monitoring device in real time, where the input is the synthesized continuous voice data and the output is the voice data transmitted to the receiving terminal.

[0780] Step 8:

[0781] The terminal receives the completed voice data from the server in real time. The input is the voice data sent from the server, and the output is the voice data stored in the terminal's buffer.

[0782] Step 9:

[0783] The device decodes the interpolated audio data and plays it back to the user through a speaker or earphone. The input is the audio data stored in the buffer, and the output is the audio played back through the speaker or earphone. This allows the user to have a conversation or monitor with uninterrupted audio.

[0784] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0785] The present invention combines a system for solving the problem of voice data interruptions during online conferences and remote communication with an emotion engine that recognizes the user's emotions. This system provides uninterrupted, continuous voice even in unstable communication environments, and also enables emotion-sensitive communication. An embodiment of this system will be described in detail below.

[0786] Overall system overview

[0787] This system receives voice data in real time, analyzes it to detect interruptions, and uses generative artificial intelligence (AI) to complete the speech based on the context. The generated completed data is then combined with the original voice data, and the completed speech data is finally sent to another device. Furthermore, it uses an emotion engine to recognize the user's emotions and provides appropriate completion based on those emotions.

[0788] Server Processing

[0789] Receiving audio data

[0790] The server receives the voice data sent from the terminal in real time and temporarily stores the received data in a buffer.

[0791] Analysis of audio data

[0792] The server converts the audio data in the buffer into text using ASR technology and detects gaps in the audio data, which are then used in the next processing step.

[0793] Applying generative AI

[0794] The server uses generative AI (e.g., GPT model) to generate text to fill in the gaps based on the context and the emotions recognized by the emotion engine. The AI ​​generates natural-looking text that matches the flow of the conversation and the user's emotions.

[0795] Applying the Emotion Engine

[0796] The server uses an emotion engine to analyze and recognize the user's emotions from the received voice data. The emotion information recognized by the emotion engine is used to complement the generation AI.

[0797] Complementary data synthesis

[0798] The generated text is converted into speech data using text-to-speech (TTS) technology and then synthesized with the original speech data. The synthesized speech data is adjusted to ensure a natural continuation of the context and emotion.

[0799] Sending complementary data

[0800] The synthesized voice data is sent to another device in real time, allowing the user to receive uninterrupted voice.

[0801] Terminal handling

[0802] Capture audio data

[0803] The device captures the speaker's voice through a microphone, then compresses and encodes it, allowing for efficient use of network bandwidth.

[0804] Sending audio data

[0805] The captured audio data is sent to the server using an appropriate protocol (e.g. WebSocket, Opus codec) to ensure reliable transmission.

[0806] Receiving complementary data

[0807] The device receives the synthesized voice data sent from the server in real time, and the received data is temporarily stored in a buffer.

[0808] Playing audio data

[0809] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[0810] User operations

[0811] Checking Voice Quality

[0812] Users can check the quality of the audio played on their device and request corrections if there are any deficiencies or interruptions. Furthermore, if they feel the emotion recognition results are inappropriate, they can request readjustment.

[0813] Emotion-based responses

[0814] The user can continue to communicate more smoothly based on the emotions recognized by the emotion engine. The emotion engine analyzes the user's tone of voice and speaking style and sends the results to the server.

[0815] Specific examples

[0816] Situation: When the communication environment is poor and the speaker's voice is interrupted

[0817] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[0818] 2. The server analyzes the received audio data and detects gaps as text.

[0819] 3. The server analyzes User A's emotions using an emotion engine and recognizes, for example, that he is "excited."

[0820] 4. The server uses generative AI to complete words such as "multiple" and "different," and adds emotional nuance to the completed text based on the results of the emotion engine.

[0821] 5. The server converts the completed text into audio data and synthesizes it with the original audio data.

[0822] 6. The server sends the synthesized voice data to User B's device, which plays it back.

[0823] In this way, smooth and emotionally sensitive voice communication can be provided even when communication is interrupted.

[0824] The processing flow will be explained below.

[0825] Server Processing

[0826] Step 1: Receiving audio data

[0827] The server receives audio data sent from the device in real time using WebSocket. The received audio data is temporarily stored in a buffer.

[0828] Example: let audioBuffer = await receiveAudioData();

[0829] Step 2: Analyzing the audio data

[0830] The server analyzes the received voice data and converts it into text using ASR (Automatic Speech Recognition) technology. It then detects gaps in the converted text.

[0831] Example: let transcribedText = await speechToText(audioBuffer);

[0832] Detect gaps: let gaps = detectMissingSegments(transcribedText);

[0833] Step 3: Applying the Emotion Engine

[0834] The server uses an emotion engine to analyze the user's emotions from the received voice data, and the emotional information is used to complement the generative AI.

[0835] Example: let userEmotion = analyzeEmotion(audioBuffer);

[0836] Step 4: Applying generative AI

[0837] The server uses generative AI (e.g., a GPT model) to generate text to fill in the gaps based on the context and emotions recognized by the emotion engine.

[0838] Example: let filledText = fillGapsWithAI(transcribedText, gaps, userEmotion);

[0839] Step 5: Synthesizing complementary data

[0840] The server converts the generated text into speech data using text-to-speech (TTS) technology and synthesizes it with the original speech data, reconstructing it into a coherent, continuous speech data.

[0841] Example: let synthesizedAudio = textToSpeech(filledText);

[0842] Synthesize audio data: let combinedAudio = mergeAudioStreams(audioBuffer, synthesizedAudio);

[0843] Step 6: Sending complementary data

[0844] The server then transmits the synthesized voice data to other devices in real time, providing uninterrupted voice to the receiving device.

[0845] Example: sendAudioDataToClients(combinedAudio);

[0846] Terminal handling

[0847] Step 1: Capture audio data

[0848] The terminal uses a microphone to capture the user's voice in real time, then compresses and encodes the data.

[0849] Example: let capturedAudio = captureAudioFromMicrophone();

[0850] Step 2: Sending audio data

[0851] The device sends the encoded audio data to the server using the WebSocket protocol or similar, and a mechanism to ensure reliability is implemented.

[0852] Example: sendAudioDataToServer(capturedAudio);

[0853] Step 3: Receiving Complementary Data

[0854] The device receives the synthesized voice data sent from the server in real time, and the received data is stored in a buffer.

[0855] Example: let receivedAudio = await receiveAudioDataFromServer();

[0856] Step 4: Playing back audio data

[0857] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[0858] Example: playAudioData(receivedAudio);

[0859] User operations

[0860] Step 1: Check the audio quality

[0861] Users can check the quality of the audio played on their device and request corrections if there are any deficiencies or interruptions. Furthermore, if they feel the emotion recognition results are inappropriate, they can request readjustment.

[0862] Example: if(audioQualityIsPoor()) { requestManualCompletion();}

[0863] Example: if(emotionRecognitionIsIncorrect()) { requestEmotionReanalysis();}

[0864] Step 2: Respond based on emotions

[0865] Users can continue to communicate more smoothly based on the emotions recognized by the emotion engine. The emotion engine analyzes the user's tone of voice and speaking style and sends the results to the server.

[0866] Example: let userEmotion = captureEmotionDuringConversation();

[0867] Specific examples

[0868] Situation: When the communication environment is poor and the speaker's voice is interrupted

[0869] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[0870] 2. The server analyzes the received audio data, converts the audio into text, and detects gaps.

[0871] 3. The server analyzes User A's emotions using an emotion engine and recognizes, for example, that he is "excited."

[0872] 4. The server uses a generative AI to generate complementary text such as "multiple" and "different," and adds emotional nuance to the complementary text based on the results of the emotion engine.

[0873] 5. The server converts the generated supplemental text into audio data and synthesizes it with the original audio data.

[0874] 6. The server sends the synthesized voice data to User B's device, and User B's device plays the completed voice.

[0875] In this way, smooth and emotionally sensitive voice communication can be provided even when communication is interrupted.

[0876] Example 2

[0877] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0878] In conventional online conference systems and remote communication systems, voice data is often interrupted due to unstable communication environments, hindering smooth communication. Furthermore, because the system does not take into account the user's emotions, the quality of communication can be reduced by mechanical responses. To solve these problems, a system is needed that provides uninterrupted, continuous voice and communication that takes the user's emotions into consideration.

[0879] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0880] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data and detecting interrupted portions, means for applying a generative model to fill in the interrupted portions based on the analyzed voice data from the context, means for synthesizing the completed data with the original voice data, means for transmitting the synthesized voice data to another terminal, and means for analyzing the user's emotions from the received voice data and using the results in the completion process of the generative model, thereby enabling smooth voice communication that takes emotions into consideration even when communication is interrupted.

[0881] 1. "Audio data" means data that represents audio information such as conversation or music in digital form.

[0882] 2. "Means for receiving" means the technology or equipment for receiving audio data through a communications network.

[0883] 3. "Means for analyzing" means technology or devices that process received voice data to decipher and understand its content and characteristics.

[0884] 4. "Interrupted portion" refers to a portion where audio data is missing or interrupted due to an unstable communication environment or other reasons.

[0885] 5. "Means for detection" means the technology or device used to identify gaps in the analyzed audio data.

[0886] 6. A "generative model" is an algorithm or program that uses artificial intelligence techniques to generate text or speech.

[0887] 7. "Context" refers to the content of the audio data or text before and after the interrupted section, and is the information used to complete the sentence.

[0888] 8. "Completion means" refers to technologies or devices that use generative models to fill in gaps in data in a natural way.

[0889] 9. "Synthesizing means" means a technique or device that combines the generated complementary data with the original audio data to form one continuous audio signal.

[0890] 10. "Transmitting means" means the technology or device used to transmit the synthesized voice data to another terminal.

[0891] 11. "Terminal" means an electronic device for capturing, receiving, and playing audio data.

[0892] 12. "Means of analysis" refers to technology or devices that understand the content of audio data and extract specific information.

[0893] 13. "Means for analyzing emotions" refers to technology or devices that identify a user's emotions from voice data and use the results in subsequent processing.

[0894] 14. "Synthesized speech data" refers to the entire speech data to which the parts completed by the generative model have been added.

[0895] 15. "Real-time" means that data processing or communication is immediate and occurs within a time range that does not cause a delay to the user.

[0896] This invention provides a system that eliminates interruptions in voice data during online conferences and remote communications, and complements the data while taking into account the user's emotions. The system receives voice data in real time, analyzes the data to detect interruptions, and complements the data using a generative model based on the context and the user's emotions. The system also has the ability to synthesize the complemented voice data with the original voice data and transmit it to another device in real time.

[0897] Server Processing

[0898] 1. Receiving audio data

[0899] The server receives the voice data sent from the terminal in real time. The received data is temporarily stored in a buffer. The technology used here includes general network communication technology (e.g., TCP / IP).

[0900] 2. Analysis of audio data

[0901] The server uses speech recognition technology, such as the Google Speech-to-Text API, to convert the audio data in the buffer into text and detects gaps in the text data.

[0902] 3. Applying generative AI

[0903] The server uses a generative AI model (such as OpenAI's GPT-3) to generate text to fill in the gaps based on the context and the results of the emotion engine analysis. The generated text reflects natural expressions.

[0904] 4. Applying the Emotion Engine

[0905] The server uses an emotion engine to analyze and recognize the user's emotions from the received voice data. The emotion engine may be, for example, IBM Watson's Tone Analyzer. The emotion information detected by the emotion engine is used to complement the generative AI.

[0906] 5. Synthesis of complementary data

[0907] The generated text is converted into audio data using text-to-speech (TTS) technology, such as the Google Text-to-Speech API. The converted audio data is then synthesized with the original audio data to create a natural continuity.

[0908] 6. Sending supplementary data

[0909] The synthesized voice data is sent to the device in real time using a low-latency, highly reliable protocol (e.g., WebSocket).

[0910] Terminal handling

[0911] 1. Capture audio data

[0912] The device captures the speaker's voice in real time through a microphone, and the captured voice data is compressed and encoded using the Opus codec.

[0913] 2. Sending audio data

[0914] The encoded audio data is sent to the server in real time.

[0915] 3. Receiving complementary data

[0916] The synthesized voice data sent from the server is received in real time and temporarily stored in a buffer.

[0917] 4. Playing audio data

[0918] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[0919] User operations

[0920] 1. Check the audio quality

[0921] The user can check the quality of the audio played on the device and request completion if there are gaps or interruptions.

[0922] 2. Emotional responses

[0923] The user can smoothly communicate based on the emotions recognized by the emotion engine.

[0924] Prompt Sentence Examples

[0925] "Hello, how was your day?"

[0926] These concise prompts form the basis for applying generative AI models.

[0927] Examples:

[0928] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[0929] 2. The server analyzes the received voice data and converts the interrupted part into text: "The new portable charger is complex...each port is different..."

[0930] 3. The server's emotion engine recognizes that User A is "excited."

[0931] 4. The server uses the generative AI model to complement words like "multiple" and "different" and reflect the emotion of "excited," generating "The new portable charger has multiple ports that can be used in different ways."

[0932] 5. The server converts the completed text into audio data and synthesizes it with the original audio data.

[0933] 6. The server sends the synthesized voice data to User B's terminal, which receives it.

[0934] 7. When User B's device plays back the received audio data, he or she can hear the continuous audio saying, "The new portable charger has multiple ports that can be used in different ways."

[0935] In this way, the present invention smoothly compensates for interruptions in communication and realizes emotionally sensitive voice communication.

[0936] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0937] Step 1:

[0938] Capture audio data

[0939] input:

[0940] The voice the user speaks.

[0941] Specific behavior:

[0942] The device uses a microphone to capture the speaker's voice in real time, then compresses and encodes the captured voice data using the Opus codec, and temporarily stores the encoded voice data in a buffer.

[0943] output:

[0944] The encoded audio data.

[0945] Step 2:

[0946] Sending audio data

[0947] input:

[0948] The encoded audio data (output of step 1).

[0949] Specific behavior:

[0950] The device transmits the encoded audio data to the server in real time using the low-latency and highly reliable WebSocket protocol.

[0951] output:

[0952] The audio data sent to the server.

[0953] Step 3:

[0954] Receiving audio data

[0955] input:

[0956] Audio data sent from the device (output from step 2).

[0957] Specific behavior:

[0958] The server receives the voice data sent from the terminal in real time and stores it in a buffer.

[0959] output:

[0960] The audio data stored in the buffer.

[0961] Step 4:

[0962] Analysis of audio data

[0963] input:

[0964] The audio data stored in the buffer (output of step 3).

[0965] Specific behavior:

[0966] The server uses the Google Speech-to-Text API to convert the audio data in the buffer into text.

[0967] Detect gaps in speech data converted into text.

[0968] output:

[0969] Parsed audio data and gap information in text format.

[0970] Step 5:

[0971] Applying the Emotion Engine

[0972] input:

[0973] The analyzed audio data and gap information in text format (output of step 4).

[0974] Specific behavior:

[0975] The server uses an emotion engine such as IBM Watson's Tone Analyzer to analyze the user's emotions from the received text-format voice data.

[0976] The emotion information obtained by the emotion engine is stored for completion processing.

[0977] output:

[0978] Emotion analysis results.

[0979] Step 6:

[0980] Applying generative AI

[0981] input:

[0982] Context and sentiment analysis results (output of steps 4 and 5).

[0983] Specific behavior:

[0984] The server uses a generative AI model, such as OpenAI's GPT-3, to generate text to fill in the gaps based on the context and the results of the sentiment engine. The generated text contains natural, contextually appropriate expressions.

[0985] output:

[0986] The generated completion text.

[0987] Step 7:

[0988] Complementary data synthesis

[0989] input:

[0990] The generated completion text (output of step 6).

[0991] Specific behavior:

[0992] The server converts the generated text into speech using TTS technology such as the Google Text-to-Speech API, and then synthesizes the generated speech with the original speech to create a natural continuity.

[0993] output:

[0994] Synthesized voice data.

[0995] Step 8:

[0996] Sending complementary data

[0997] input:

[0998] The synthesized speech data (output of step 7).

[0999] Specific behavior:

[1000] The server transmits the synthesized voice data to the device in real time, again using the low-latency and highly reliable WebSocket protocol.

[1001] output:

[1002] Synthesized voice data sent to the device.

[1003] Step 9:

[1004] Receiving complementary data

[1005] input:

[1006] Synthesized speech data sent from the server (output of step 8).

[1007] Specific behavior:

[1008] The terminal receives the synthesized voice data sent from the server in real time and temporarily stores it in a buffer.

[1009] output:

[1010] The synthesized speech data stored in the buffer.

[1011] Step 10:

[1012] Playing audio data

[1013] input:

[1014] The synthesized speech data stored in the buffer (output of step 9).

[1015] Specific behavior:

[1016] The device decodes the received synthesized voice data and plays it through the speaker, allowing the user to hear the voice without interruption.

[1017] output:

[1018] The audio played.

[1019] (Application example 2)

[1020] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1021] Conventional voice communication systems suffer from the problem of voice interruptions when the communication environment is unstable. In particular, when interacting with customers in virtual stores, voice interruptions can be a significant factor in impairing the customer experience. Furthermore, conventional systems lack the ability to recognize user emotions, making it difficult to respond naturally and with consideration for the user's emotions. To address these issues, a system is needed that can compensate for communication interruptions and provide appropriate responses based on the user's emotions.

[1022] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1023] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data and detecting interrupted portions, means for applying a generative model to fill in the interrupted portions based on the analyzed voice data from the context and the recognized emotion, means for synthesizing the generated completed data with the original voice data, means for transmitting the synthesized voice data to another terminal, and an emotion engine for recognizing emotions. This enables smooth voice communication that takes emotion into consideration even when communication is interrupted.

[1024] "Voice data" refers to data obtained by converting the user's voice into digital form.

[1025] "Means for analyzing" refers to the processes and techniques for converting received voice data into text data and identifying gaps and voice characteristics.

[1026] A "generative model" is an artificial intelligence model trained on a large dataset and capable of automatically generating text and speech.

[1027] An "emotion engine" is a technology that analyzes emotions from a user's voice and recognizes their emotional state.

[1028] "Means of synthesis" refers to technology that naturally blends supplemented text or audio data with the original audio data.

[1029] "Communication quality" refers to the state of the network environment that allows voice data to be transmitted smoothly without interruption.

[1030] A "virtual store" refers to a virtual sales environment that provides products and services via the Internet.

[1031] "Communication with customers" refers to the exchange of voice and text between customer service staff and customers in a virtual store.

[1032] A system for implementing this invention includes a series of processes for receiving, analyzing, and completing voice data, and providing an appropriate response based on emotion. A specific method for implementing this system is described below.

[1033] System configuration

[1034] 1. Receiving audio data

[1035] The server receives the user's voice data in real time, and stores it in a temporary buffer. The hardware used here is a smartphone, smart glasses, or a head-mounted display, and the software uses WebSocket and an appropriate codec.

[1036] 2. Analysis of audio data

[1037] The server converts the received voice data into text using speech recognition technology (e.g., Google Speech-to-Text API), detects gaps in the converted text, and uses the analysis results in the next processing step. It also recognizes emotions using an emotion engine (e.g., IBM Watson Tone Analyzer).

[1038] 3. Applying generative AI

[1039] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate appropriate completion text based on context and sentiment information. Specifically, it uses the following example prompt:

[1040] Audio data: "The feature of this product is its long... different... battery life."

[1041] Emotion Engine: "Passionate"

[1042] Prompt: "Complete the following broken text based on context and sentiment: 'This product's feature is its long... different... battery life.'"

[1043] 4. Synthesis of complementary data

[1044] The completed text is converted to audio data (e.g., Google Text-to-Speech API) and then combined with the original audio data, resulting in a seamless, natural-sounding speech.

[1045] 5. Sending complementary data

[1046] The synthesized voice data is sent in real time to the customer's device, where it is decoded and played through a speaker.

[1047] Specific examples

[1048] Situation: The communication environment becomes poor while serving a customer in a virtual store, causing the customer service representative's voice to be interrupted.

[1049] 1. Customer Service Representative: "The special feature of this product is its long... different... battery life."

[1050] 2. Server: Analyzes the audio data and detects interruptions. At the same time, the emotion engine recognizes passionate emotions.

[1051] 3. Server: The generation AI generates complementary text such as "long duration" and "difference," adding emotional nuance.

[1052] 4. Server: The completed text is converted into audio data and combined with the original audio data.

[1053] 5. Server: Sends the synthesized voice data to the customer's device and plays the voice without interruption.

[1054] This system allows smooth communication with customers in virtual stores even in the event of a communication interruption, improving the customer experience.

[1055] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1056] Step 1:

[1057] Receiving audio data

[1058] The server receives the audio data sent from the device in real time. This audio data, which is input from the device, is temporarily stored in a buffer. Specifically, the server receives the audio data using WebSocket and buffers it to prevent data loss.

[1059] Step 2:

[1060] Analysis of audio data

[1061] The server converts the buffered audio data into text using speech recognition technology (e.g., Google Speech-to-Text API). The audio data is used as input and text data is generated as output. The specific operation here is to send the audio file to the speech recognition API and analyze the returned text.

[1062] Step 3:

[1063] Gap detection and emotion recognition

[1064] Detect discontinuities in the analyzed text data and simultaneously recognize emotions using an emotion engine (e.g., IBM Watson Tone Analyzer). The input data is the converted text, and the output is information about the discontinuities and emotion information. Specifically, discontinuities in the text are detected, and the text is input to the emotion analysis engine to obtain emotion labels.

[1065] Step 4:

[1066] Complemented by generative AI

[1067] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate text that completes the missing part based on context and emotional information. The input is the surrounding text and emotional information, and the output is the completed text. A specific example includes providing the following prompt to the generative AI:

[1068] Audio data: "The feature of this product is its long... different... battery life."

[1069] Emotion Engine: "Passionate"

[1070] Prompt: "Complete the following broken text based on context and sentiment: 'This product's feature is its long... different... battery life.'"

[1071] Step 5:

[1072] Complementary data synthesis

[1073] The completed text is converted back into audio data (e.g., Google Text-to-Speech API) and then synthesized with the original audio data. The input is the completed text and the original audio data, and the output is the synthesized audio data. Specifically, the completed text is sent to the text-to-speech API, which generates audio and then combines it with the original audio data.

[1074] Step 6:

[1075] Sending complementary data

[1076] The server transmits the synthesized voice data to other devices in real time. The input is the synthesized voice data, and the output is the retransmitted voice data. Specifically, the device receives the synthesized voice using WebSocket and plays it back in real time.

[1077] Step 7:

[1078] Playing audio

[1079] The device decodes the received audio data and plays it back to the user through the speaker. The input is the received audio data, and the output is the audio from the speaker. Specifically, the device sends the decoded audio data to the speaker and plays it back in a format that reaches the human ear.

[1080] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1081] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1082] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1083] [Third embodiment]

[1084] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1085] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1086] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1087] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1088] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1089] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1090] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1091] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1092] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1093] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1094] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1095] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1096] The present invention provides a system for solving the problem of audio data interruptions in online conferences and remote communications. This system can provide continuous audio without interruptions even in an unstable communication environment. An embodiment of this system will be described in detail below.

[1097] Overall system overview

[1098] This system receives voice data in real time, analyzes the data to detect interruptions, and uses generative artificial intelligence (AI) to complete the data based on the context. The completed data is then synthesized with the original voice data, and the final completed voice data is sent to another device.

[1099] Server Processing

[1100] Receiving audio data

[1101] The server receives the voice data sent from the device in real time using protocols such as WebSocket, which minimizes conversation delays.

[1102] Analysis of audio data

[1103] Automatic speech recognition (ASR) technology is used to analyze the received voice data and detect interruptions. The analysis results are saved as text data.

[1104] Applying generative AI

[1105] The server applies generative AI (e.g., a GPT model) to fill in gaps from the context of the parsed text data. The generative AI generates natural-sounding text that fits the gaps.

[1106] Complementary data synthesis

[1107] The generated supplemental text data is converted into audio data using text-to-speech (TTS) technology, and then the supplemental audio data is synthesized with the original audio data to create a coherent, continuous audio data.

[1108] Sending complementary data

[1109] The synthesized voice data is sent to other devices in real time, using a real-time communication protocol such as WebSocket.

[1110] Terminal handling

[1111] Capture audio data

[1112] The device captures the user's voice through a microphone and then compresses and encodes the data, allowing for efficient use of network bandwidth.

[1113] Sending audio data

[1114] The captured audio data is sent to the server using an appropriate protocol (e.g., Opus codec) to ensure stable communication.

[1115] Receiving complementary data

[1116] The terminal receives the completed voice data from the server, and the received data is played back to the user without delay.

[1117] Playing audio data

[1118] The device decodes the completed audio data and plays it through the speaker, allowing the user to continue the conversation without interruption.

[1119] User operations

[1120] Users can operate the device to capture their own voice and receive continuous voice data without interruption even if the communication is interrupted. Users can also check the quality of the captured voice and request retransmission if necessary.

[1121] Specific examples

[1122] Situation: When the communication environment is poor and the speaker's voice is interrupted

[1123] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[1124] 2. The server analyzes the received audio data, detects any interruptions, and converts them into text.

[1125] 3. The server uses a generation AI to generate complementary text such as "multiple" and "different."

[1126] 4. The server converts the completed text into audio data and synthesizes it with the original audio data.

[1127] 5. The server sends the completed voice data to User B's device, and User B's device plays the completed voice.

[1128] In this way, the system can provide smooth voice communication even when communication is interrupted.

[1129] The processing flow will be explained below.

[1130] Server Processing

[1131] Step 1: Receiving audio data

[1132] The server receives audio data sent from the device in real time via WebSocket, and the received data is temporarily stored in a buffer.

[1133] Example: let audioBuffer = await receiveAudioData();

[1134] Step 2: Analyzing the audio data

[1135] The server converts the audio data in the buffer into text using ASR technology and detects gaps. This analysis must be done quickly.

[1136] Example: let transcribedText = await speechToText(audioBuffer);

[1137] Detect gaps: let gaps = detectMissingSegments(transcribedText);

[1138] Step 3: Applying generative AI

[1139] The server uses generative AI (e.g., GPT model) to generate text to fill in the gaps from the context. The AI ​​generates the completed text in a way that maintains a natural flow of dialogue.

[1140] Example: let filledText = fillGapsWithAI(transcribedText, gaps);

[1141] Step 4: Synthesis of complementary data

[1142] The server synthesizes the generated supplemental text with the original speech data using text-to-speech (TTS) technology.

[1143] Example: let synthesizedAudio = textToSpeech(filledText);

[1144] Synthesize audio data: let combinedAudio = mergeAudioStreams(audioBuffer, synthesizedAudio);

[1145] Step 5: Sending complementary data

[1146] The server transmits the synthesized complete voice data to the other terminal in real time, providing uninterrupted voice to the receiving terminal.

[1147] Example: sendAudioDataToClients(combinedAudio);

[1148] Terminal handling

[1149] Step 1: Capture audio data

[1150] The terminal captures the user's voice in real time through a microphone and compresses and encodes the data.

[1151] Example: let capturedAudio = captureAudioFromMicrophone();

[1152] Step 2: Sending audio data

[1153] The device transmits the encoded audio data to the server using the WebSocket protocol, where appropriate error checking mechanisms are implemented to ensure reliable transmission.

[1154] Example: sendAudioDataToServer(capturedAudio);

[1155] Step 3: Receiving Complementary Data

[1156] The device receives the synthesized voice data sent from the server in real time, and the received data is stored in a buffer.

[1157] Example: let receivedAudio = await receiveAudioDataFromServer();

[1158] Step 4: Playing back audio data

[1159] The device decodes the received audio data and plays it through the speaker, providing the user with uninterrupted audio.

[1160] Example: playAudioData(receivedAudio);

[1161] User operations

[1162] Step 1: Check the audio quality

[1163] Users can check the quality of the audio played on their device and manually request correction if there are any interruptions.

[1164] Example: if(audioQualityIsPoor()) { requestManualCompletion();}

[1165] Step 2: Working with ongoing conversations

[1166] The user can smoothly continue the conversation based on the completed voice data, and can also request retransmission of the incorrectly completed data if necessary.

[1167] Example: if(incorrectCompletion()) { requestResend();}

[1168] These processing steps compensate for audio interruptions during online meetings, resulting in smoother communication.

[1169] Example 1

[1170] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1171] In online meetings and remote communications, an unstable communication environment can cause interruptions in audio data, hindering smooth communication and increasing user stress. Conventional technologies lack a mechanism for automatically and in real time compensating for such interrupted audio, so an effective system that solves this problem is needed.

[1172] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1173] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data to detect interrupted portions, means for applying artificial intelligence to fill in the interrupted portions based on the analyzed voice data from the context before and after, means for converting the generated supplemented data into voice data using text-to-speech technology, means for synthesizing the converted supplemented voice data with the original voice data, and means for transmitting the synthesized voice data to another terminal, thereby making it possible to provide uninterrupted, continuous voice in real time even in an unstable communication environment.

[1174] "Audio data" refers to data obtained by converting an audio signal acquired through an audio input device such as a microphone into a digital format.

[1175] "Means for receiving" refers to the technical means for obtaining audio data over a network and storing it in an appropriate data structure, typically using a communication protocol such as WebSocket.

[1176] The "analyzing means" refers to the technical means for processing the received voice data and detecting the interrupted parts. Automatic Speech Recognition (ASR) technology is typically used.

[1177] "Generative AI" is an abbreviation for generative artificial intelligence, and refers to a machine learning model for generating natural-sounding text and speech based on given input data.

[1178] "Text-to-speech technology" refers to the technical means for converting generated text data into digital audio signals. Text-to-speech (TTS) technology is commonly used.

[1179] "Synthesizing means" means the technical means for combining the converted audio data with the original audio data to create a coherent, continuous audio data.

[1180] "Transmission means" refers to the technical means for transmitting the synthesized voice data to another device in real time. Typically, a communication protocol such as WebSocket is used.

[1181] The "means for monitoring communication quality" refers to a technical means for monitoring the status of transmission and reception of voice data, and automatically triggering a compensation process if a problem such as voice interruption occurs.

[1182] This invention is a system that enables smooth voice communication even in an unstable communication environment. Specifically, it receives and analyzes voice data in real time and uses generative artificial intelligence (generative AI) to fill in any interrupted parts. The filled-in voice data is then synthesized with the original voice data and transmitted to another terminal as continuous voice. The following describes each component of the system and its specific operation.

[1183] Server Processing

[1184] The server first receives the audio data sent from the device in real time using the WebSocket protocol, a communication protocol that enables bidirectional data exchange with low latency.

[1185] The received voice data is converted into text using automatic speech recognition (ASR). This allows the server to identify the interrupted parts of the voice data. As a result of the ASR analysis, the resulting text might be something like, "The new portable charger has multiple ports...each port is different..."

[1186] Next, the server uses a generative AI (for example, a GPT-3 model) to generate text to complete the missing parts. The generated text includes completion parts such as "multiple" and "different." The following prompt sentence is used to apply the generative AI:

[1187] Example prompt sentence:

[1188] "The new portable charger has multiple ports...each port is different..."

[1189] The generated text is converted into audio data using Text-to-Speech (TTS) technology and then combined with the original audio data to create a consistent, continuous audio stream, which is then sent to other devices using the WebSocket protocol.

[1190] Terminal handling

[1191] The device captures the user's voice through a microphone, and this voice data is compressed and encoded using the Opus codec and sent to the server in a manner that efficiently uses network bandwidth.

[1192] The completed voice data sent from the server is received in real time by the device. The received voice data is decoded using the Opus codec and played back from the speaker. This allows users to continue a smooth conversation with continuous audio, regardless of communication interruptions.

[1193] User operations

[1194] Users can operate the device to capture their own voice and receive the completed voice data even if communication is interrupted. Users can also check the quality of the completed voice and request retransmission if necessary.

[1195] Specific examples

[1196] For example, consider a situation where a speaker's voice is interrupted due to a poor communication environment. User A's device sends the interrupted voice, "The new portable charger has multiple... each port is different..." to the server. The server analyzes the received voice data, detects the interrupted part, and converts it into text. Next, the server uses a generative AI (e.g., GPT-3 model) to generate complementary text such as "multiple" and "different." The server converts the complementary text into voice data and synthesizes it with the original voice data. Finally, the server sends the completed voice data to User B's device, which plays back the completed voice.

[1197] In this way, the system can provide smooth voice communication even when communication is interrupted.

[1198] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1199] Step 1: Capture audio data

[1200] Input: User's voice

[1201] Output: Captured audio data

[1202] The device uses a microphone to capture the user's voice, and the captured voice data is compressed and encoded in real time using the Opus codec, reducing the data size and allowing for efficient use of network bandwidth.

[1203] Step 2: Sending audio data

[1204] Input: Captured audio data

[1205] Output: Audio data sent to the server

[1206] The device sends the encoded audio data to the server using the WebSocket protocol, which allows for bidirectional, low-latency communication.

[1207] Step 3: Receiving audio data

[1208] Input: Audio data sent from the device

[1209] Output: Audio data stored on the server

[1210] The server receives the audio data sent from the device in real time. The received audio data is temporarily stored in a buffer. The WebSocket protocol is used to efficiently receive the data.

[1211] Step 4: Analyzing the audio data

[1212] Input: Audio data stored on the server

[1213] Output: Parsed text data

[1214] The server uses ASR (Automatic Speech Recognition) technology to analyze the received voice data and convert it into text data. During this process, the entire voice data is analyzed in detail to detect any interruptions. For example, text data such as "The new portable charger has multiple ports... each port is different..." can be obtained.

[1215] Step 5: Applying generative AI

[1216] Input: Parsed text data

[1217] Output: Generated completed text data

[1218] The server uses generative AI (e.g., the GPT-3 model) based on the analyzed text data to complete the missing parts, generating natural-looking completed text appropriate to the context. The generated text includes appropriate completed text such as "multiple" and "different."

[1219] Step 6: Converting the completed text to speech

[1220] Input: Generated completed text data

[1221] Output: Interpolated audio data

[1222] The server converts the generated supplemental text data into voice data using text-to-speech (TTS) technology. For example, the supplemental text "Multiple ports have different specifications" is converted into natural voice data.

[1223] Step 7: Synthesizing the audio data

[1224] Input: Original audio data and interpolated audio data

[1225] Output: Synthesized continuous audio data

[1226] The server then synthesizes the converted speech data with the original speech data, creating a consistent, continuous speech stream. The speech data is seamlessly integrated into the original speech data, creating a seamless audio stream.

[1227] Step 8: Sending the completed audio data

[1228] Input: Synthesized continuous speech data

[1229] Output: The completed audio data sent to the device.

[1230] The server then transmits the synthesized continuous audio data to other devices using the WebSocket protocol, which allows for efficient transmission of audio data in real time.

[1231] Step 9: Receiving the completed audio data

[1232] Input: Completed audio data sent from the server

[1233] Output: Completed audio data saved on the device

[1234] The terminal receives the completed voice data sent from the server in real time, and the received voice data is temporarily stored in a buffer.

[1235] Step 10: Decode and play the audio data

[1236] Input: Completed voice data saved on the device

[1237] Output: A continuous sound that is played to the user

[1238] The device then decodes the received audio data using the Opus codec and plays it back through the speaker, allowing the user to hear continuous audio without interruptions.

[1239] These steps enable the system to achieve smooth voice communication regardless of the instability of the communication environment.

[1240] (Application example 1)

[1241] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1242] The present invention aims to provide continuous audio without interruptions in audio data even in unstable communication environments during online conferences and remote communications. In the field of security monitoring, it is also an issue to provide a monitor with continuous audio without any sense of interruption even when audio data from a surveillance camera or sensor is interrupted.

[1243] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1244] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data to detect interrupted portions, means for applying artificial intelligence to fill in the interrupted portions based on the analyzed voice data from the context, means for synthesizing the generated completed voice data with the original voice data, means for transmitting the completed voice data to another terminal, and means for providing the completed voice data to the user's playback device in real time. This enables uninterrupted voice communication even in an environment with unstable communication. Furthermore, the monitor can monitor with uninterrupted, continuous voice.

[1245] "Audio data" is a digital representation of a sound signal captured by a microphone or other input device.

[1246] The "receiving means" is a device or module that has the function of acquiring audio data from another device or system.

[1247] "Means for analyzing" means any device or software having the processing capability to decompose received audio data and determine gaps and other characteristics.

[1248] A "gap" is a portion of the audio data where information is missing.

[1249] "Generative AI" is a collection of algorithms, such as machine learning models and neural networks, that are used to complement voice data.

[1250] "Complementary data" refers to a portion of the audio data that has been filled in with information generated by artificial intelligence to fill in the gaps.

[1251] The "synthesizing means" refers to a device or software that has the function of integrating the original audio data and the generated complementary data into one continuous audio data.

[1252] A "playback device" is a device such as a speaker or earphone that allows the user to listen to the synthesized audio data.

[1253] "Communication quality" refers to the state of the network environment when voice data is transmitted continuously without interruption.

[1254] A "triggering means" is a mechanism or function that automatically starts processing when a specific condition is met.

[1255] The present invention provides a system for preventing interruptions in voice data even in unstable communication environments during online conferences and remote communications. This system aims to improve security monitoring and voice call quality. Specific embodiments of this system are described below.

[1256] Overall system overview

[1257] The system receives and analyzes voice data in real time, detects interruptions, applies generative AI based on the context to complete the audio, and finally synthesizes the completed audio data and transmits it to other devices.

[1258] Server Processing

[1259] Receiving audio data

[1260] The server receives audio data sent from the terminal or security monitoring device in real time using the WebSocket protocol, allowing the server to minimize delays in conversation and monitoring.

[1261] Analysis of audio data

[1262] The received voice data is analyzed using automatic speech recognition (ASR). As a result of the analysis, any gaps are detected and the data is converted into text format.

[1263] Applying generative AI

[1264] The server applies generative AI (e.g., the GPT-4 model) based on the context of the analyzed text data to generate text to naturally complete the gaps.

[1265] Complementary data synthesis

[1266] The generated supplemental text data is converted into audio data using text-to-speech (TTS) technology and then combined with the original audio data, creating a continuous, uninterrupted audio stream.

[1267] Sending complementary data

[1268] The synthesized voice data is transmitted in real time to another device or monitoring device using the WebSocket protocol.

[1269] Terminal handling

[1270] Capture audio data

[1271] The device can use a microphone to capture the user's voice and compress and encode the voice data to use network bandwidth efficiently.

[1272] Sending audio data

[1273] The captured audio data is sent to the server, again using the WebSocket protocol.

[1274] Receiving complementary data

[1275] The terminal receives the complemented voice data from the server in real time.

[1276] Playing audio data

[1277] The device decodes the interpolated audio data and plays it back to the user through a speaker or earphones, allowing the user to have a conversation or monitor with uninterrupted audio.

[1278] User operations

[1279] Users can operate the device to capture their own voice and receive continuous voice data without interruption even if the communication is interrupted. Users can also check the quality of the captured voice and request retransmission if necessary.

[1280] Specific examples

[1281] Situation: When the communication environment is poor and the speaker's voice is interrupted

[1282] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[1283] 2. The server analyzes the received audio data, detects any interruptions, and converts them into text.

[1284] 3. The server uses a generation AI to generate complementary text such as "multiple" and "different."

[1285] 4. The server converts the completed text into audio data and synthesizes it with the original audio data.

[1286] 5. The server sends the completed voice data to User B's device, and User B's device plays the completed voice.

[1287] Prompt Sentence Examples

[1288] Audio data from ears: Warning, something on premises. ___Intrusion detected. ___

[1289] Generate context-aligned audio data.

[1290] In this way, the system can provide smooth voice communication and security monitoring even when communication is interrupted.

[1291] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1292] Step 1:

[1293] The server receives audio data from a terminal or security monitoring device in real time using the WebSocket protocol. The input is audio data, which the server stores in a buffer. The output is the stored audio data.

[1294] Step 2:

[1295] The server analyzes the received voice data using ASR technology. The input is the stored voice data, and the analysis converts the voice data into text format. The output is the text data that is the analysis result.

[1296] Step 3:

[1297] The server inspects the parsed text data and detects gaps. The input is the text data, and the detection process identifies the gaps. The output is the text data with the gaps indicated.

[1298] Step 4:

[1299] The server applies generative AI (GPT-4 model) based on the context of the analyzed text data to generate text to complete the gap. The input is the text data showing the gap, and the output is the completed text data. In this process, the following prompt sentence is used as input:

[1300] Audio data from ears: Warning, something on premises. ___Intrusion detected. ___

[1301] Generate context-aligned audio data.

[1302] Step 5:

[1303] The server converts the generated supplemented text data into audio data using text-to-speech (TTS) technology, with the input being the supplemented text data and the output being the supplemented audio data.

[1304] Step 6:

[1305] The server synthesizes the generated complementary speech data with the original speech data to create a single continuous speech data. The input is the complementary speech data and the original speech data, and the output is the synthesized continuous speech data.

[1306] Step 7:

[1307] The server transmits the synthesized voice data to another terminal or monitoring device in real time, where the input is the synthesized continuous voice data and the output is the voice data transmitted to the receiving terminal.

[1308] Step 8:

[1309] The terminal receives the completed voice data from the server in real time. The input is the voice data sent from the server, and the output is the voice data stored in the terminal's buffer.

[1310] Step 9:

[1311] The device decodes the interpolated audio data and plays it back to the user through a speaker or earphone. The input is the audio data stored in the buffer, and the output is the audio played back through the speaker or earphone. This allows the user to have a conversation or monitor with uninterrupted audio.

[1312] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1313] The present invention combines a system for solving the problem of voice data interruptions during online conferences and remote communication with an emotion engine that recognizes the user's emotions. This system provides uninterrupted, continuous voice even in unstable communication environments, and also enables emotion-sensitive communication. An embodiment of this system will be described in detail below.

[1314] Overall system overview

[1315] This system receives voice data in real time, analyzes it to detect interruptions, and uses generative artificial intelligence (AI) to complete the speech based on the context. The generated completed data is then combined with the original voice data, and the completed speech data is finally sent to another device. Furthermore, it uses an emotion engine to recognize the user's emotions and provides appropriate completion based on those emotions.

[1316] Server Processing

[1317] Receiving audio data

[1318] The server receives the voice data sent from the terminal in real time and temporarily stores the received data in a buffer.

[1319] Analysis of audio data

[1320] The server converts the audio data in the buffer into text using ASR technology and detects gaps in the audio data, which are then used in the next processing step.

[1321] Applying generative AI

[1322] The server uses generative AI (e.g., GPT model) to generate text to fill in the gaps based on the context and the emotions recognized by the emotion engine. The AI ​​generates natural-looking text that matches the flow of the conversation and the user's emotions.

[1323] Applying the Emotion Engine

[1324] The server uses an emotion engine to analyze and recognize the user's emotions from the received voice data. The emotion information recognized by the emotion engine is used to complement the generation AI.

[1325] Complementary data synthesis

[1326] The generated text is converted into speech data using text-to-speech (TTS) technology and then synthesized with the original speech data. The synthesized speech data is adjusted to ensure a natural continuation of the context and emotion.

[1327] Sending complementary data

[1328] The synthesized voice data is sent to another device in real time, allowing the user to receive uninterrupted voice.

[1329] Terminal handling

[1330] Capture audio data

[1331] The device captures the speaker's voice through a microphone, then compresses and encodes it, allowing for efficient use of network bandwidth.

[1332] Sending audio data

[1333] The captured audio data is sent to the server using an appropriate protocol (e.g. WebSocket, Opus codec) to ensure reliable transmission.

[1334] Receiving complementary data

[1335] The device receives the synthesized voice data sent from the server in real time, and the received data is temporarily stored in a buffer.

[1336] Playing audio data

[1337] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[1338] User operations

[1339] Checking Voice Quality

[1340] Users can check the quality of the audio played on their device and request corrections if there are any deficiencies or interruptions. Furthermore, if they feel the emotion recognition results are inappropriate, they can request readjustment.

[1341] Emotion-based responses

[1342] The user can continue to communicate more smoothly based on the emotions recognized by the emotion engine. The emotion engine analyzes the user's tone of voice and speaking style and sends the results to the server.

[1343] Specific examples

[1344] Situation: When the communication environment is poor and the speaker's voice is interrupted

[1345] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[1346] 2. The server analyzes the received audio data and detects gaps as text.

[1347] 3. The server analyzes User A's emotions using an emotion engine and recognizes, for example, that he is "excited."

[1348] 4. The server uses generative AI to complete words such as "multiple" and "different," and adds emotional nuance to the completed text based on the results of the emotion engine.

[1349] 5. The server converts the completed text into audio data and synthesizes it with the original audio data.

[1350] 6. The server sends the synthesized voice data to User B's device, which plays it back.

[1351] In this way, smooth and emotionally sensitive voice communication can be provided even when communication is interrupted.

[1352] The processing flow will be explained below.

[1353] Server Processing

[1354] Step 1: Receiving audio data

[1355] The server receives audio data sent from the device in real time using WebSocket. The received audio data is temporarily stored in a buffer.

[1356] Example: let audioBuffer = await receiveAudioData();

[1357] Step 2: Analyzing the audio data

[1358] The server analyzes the received voice data and converts it into text using ASR (Automatic Speech Recognition) technology. It then detects gaps in the converted text.

[1359] Example: let transcribedText = await speechToText(audioBuffer);

[1360] Detect gaps: let gaps = detectMissingSegments(transcribedText);

[1361] Step 3: Applying the Emotion Engine

[1362] The server uses an emotion engine to analyze the user's emotions from the received voice data, and the emotional information is used to complement the generative AI.

[1363] Example: let userEmotion = analyzeEmotion(audioBuffer);

[1364] Step 4: Applying generative AI

[1365] The server uses generative AI (e.g., a GPT model) to generate text to fill in the gaps based on the context and emotions recognized by the emotion engine.

[1366] Example: let filledText = fillGapsWithAI(transcribedText, gaps, userEmotion);

[1367] Step 5: Synthesizing complementary data

[1368] The server converts the generated text into speech data using text-to-speech (TTS) technology and synthesizes it with the original speech data, reconstructing it into a coherent, continuous speech data.

[1369] Example: let synthesizedAudio = textToSpeech(filledText);

[1370] Synthesize audio data: let combinedAudio = mergeAudioStreams(audioBuffer, synthesizedAudio);

[1371] Step 6: Sending complementary data

[1372] The server then transmits the synthesized voice data to other devices in real time, providing uninterrupted voice to the receiving device.

[1373] Example: sendAudioDataToClients(combinedAudio);

[1374] Terminal handling

[1375] Step 1: Capture audio data

[1376] The terminal uses a microphone to capture the user's voice in real time, then compresses and encodes the data.

[1377] Example: let capturedAudio = captureAudioFromMicrophone();

[1378] Step 2: Sending audio data

[1379] The device sends the encoded audio data to the server using the WebSocket protocol or similar, and a mechanism to ensure reliability is implemented.

[1380] Example: sendAudioDataToServer(capturedAudio);

[1381] Step 3: Receiving Complementary Data

[1382] The device receives the synthesized voice data sent from the server in real time, and the received data is stored in a buffer.

[1383] Example: let receivedAudio = await receiveAudioDataFromServer();

[1384] Step 4: Playing back audio data

[1385] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[1386] Example: playAudioData(receivedAudio);

[1387] User operations

[1388] Step 1: Check the audio quality

[1389] Users can check the quality of the audio played on their device and request corrections if there are any deficiencies or interruptions. Furthermore, if they feel the emotion recognition results are inappropriate, they can request readjustment.

[1390] Example: if(audioQualityIsPoor()) { requestManualCompletion();}

[1391] Example: if(emotionRecognitionIsIncorrect()) { requestEmotionReanalysis();}

[1392] Step 2: Respond based on emotions

[1393] Users can continue to communicate more smoothly based on the emotions recognized by the emotion engine. The emotion engine analyzes the user's tone of voice and speaking style and sends the results to the server.

[1394] Example: let userEmotion = captureEmotionDuringConversation();

[1395] Specific examples

[1396] Situation: When the communication environment is poor and the speaker's voice is interrupted

[1397] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[1398] 2. The server analyzes the received audio data, converts the audio into text, and detects gaps.

[1399] 3. The server analyzes User A's emotions using an emotion engine and recognizes, for example, that he is "excited."

[1400] 4. The server uses a generative AI to generate complementary text such as "multiple" and "different," and adds emotional nuance to the complementary text based on the results of the emotion engine.

[1401] 5. The server converts the generated supplemental text into audio data and synthesizes it with the original audio data.

[1402] 6. The server sends the synthesized voice data to User B's device, and User B's device plays the completed voice.

[1403] In this way, smooth and emotionally sensitive voice communication can be provided even when communication is interrupted.

[1404] Example 2

[1405] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1406] In conventional online conference systems and remote communication systems, voice data is often interrupted due to unstable communication environments, hindering smooth communication. Furthermore, because the system does not take into account the user's emotions, the quality of communication can be reduced by mechanical responses. To solve these problems, a system is needed that provides uninterrupted, continuous voice and communication that takes the user's emotions into consideration.

[1407] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1408] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data and detecting interrupted portions, means for applying a generative model to fill in the interrupted portions based on the analyzed voice data from the context, means for synthesizing the completed data with the original voice data, means for transmitting the synthesized voice data to another terminal, and means for analyzing the user's emotions from the received voice data and using the results in the completion process of the generative model, thereby enabling smooth voice communication that takes emotions into consideration even when communication is interrupted.

[1409] 1. "Audio data" means data that represents audio information such as conversation or music in digital form.

[1410] 2. "Means for receiving" means the technology or equipment for receiving audio data through a communications network.

[1411] 3. "Means for analyzing" means technology or devices that process received voice data to decipher and understand its content and characteristics.

[1412] 4. "Interrupted portion" refers to a portion where audio data is missing or interrupted due to an unstable communication environment or other reasons.

[1413] 5. "Means for detection" means the technology or device used to identify gaps in the analyzed audio data.

[1414] 6. A "generative model" is an algorithm or program that uses artificial intelligence techniques to generate text or speech.

[1415] 7. "Context" refers to the content of the audio data or text before and after the interrupted section, and is the information used to complete the sentence.

[1416] 8. "Completion means" refers to technologies or devices that use generative models to fill in gaps in data in a natural way.

[1417] 9. "Synthesizing means" means a technique or device that combines the generated complementary data with the original audio data to form one continuous audio signal.

[1418] 10. "Transmitting means" means the technology or device used to transmit the synthesized voice data to another terminal.

[1419] 11. "Terminal" means an electronic device for capturing, receiving, and playing audio data.

[1420] 12. "Means of analysis" refers to technology or devices that understand the content of audio data and extract specific information.

[1421] 13. "Means for analyzing emotions" refers to technology or devices that identify a user's emotions from voice data and use the results in subsequent processing.

[1422] 14. "Synthesized speech data" refers to the entire speech data to which the parts completed by the generative model have been added.

[1423] 15. "Real-time" means that data processing or communication is immediate and occurs within a time range that does not cause a delay to the user.

[1424] This invention provides a system that eliminates interruptions in voice data during online conferences and remote communications, and complements the data while taking into account the user's emotions. The system receives voice data in real time, analyzes the data to detect interruptions, and complements the data using a generative model based on the context and the user's emotions. The system also has the ability to synthesize the complemented voice data with the original voice data and transmit it to another device in real time.

[1425] Server Processing

[1426] 1. Receiving audio data

[1427] The server receives the voice data sent from the terminal in real time. The received data is temporarily stored in a buffer. The technology used here includes general network communication technology (e.g., TCP / IP).

[1428] 2. Analysis of audio data

[1429] The server uses speech recognition technology, such as the Google Speech-to-Text API, to convert the audio data in the buffer into text and detects gaps in the text data.

[1430] 3. Applying generative AI

[1431] The server uses a generative AI model (such as OpenAI's GPT-3) to generate text to fill in the gaps based on the context and the results of the emotion engine analysis. The generated text reflects natural expressions.

[1432] 4. Applying the Emotion Engine

[1433] The server uses an emotion engine to analyze and recognize the user's emotions from the received voice data. The emotion engine may be, for example, IBM Watson's Tone Analyzer. The emotion information detected by the emotion engine is used to complement the generative AI.

[1434] 5. Synthesis of complementary data

[1435] The generated text is converted into audio data using text-to-speech (TTS) technology, such as the Google Text-to-Speech API. The converted audio data is then synthesized with the original audio data to create a natural continuity.

[1436] 6. Sending supplementary data

[1437] The synthesized voice data is sent to the device in real time using a low-latency, highly reliable protocol (e.g., WebSocket).

[1438] Terminal handling

[1439] 1. Capture audio data

[1440] The device captures the speaker's voice in real time through a microphone, and the captured voice data is compressed and encoded using the Opus codec.

[1441] 2. Sending audio data

[1442] The encoded audio data is sent to the server in real time.

[1443] 3. Receiving complementary data

[1444] The synthesized voice data sent from the server is received in real time and temporarily stored in a buffer.

[1445] 4. Playing audio data

[1446] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[1447] User operations

[1448] 1. Check the audio quality

[1449] The user can check the quality of the audio played on the device and request completion if there are gaps or interruptions.

[1450] 2. Emotional responses

[1451] The user can smoothly communicate based on the emotions recognized by the emotion engine.

[1452] Prompt Sentence Examples

[1453] "Hello, how was your day?"

[1454] These concise prompts form the basis for applying generative AI models.

[1455] Examples:

[1456] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[1457] 2. The server analyzes the received voice data and converts the interrupted part into text: "The new portable charger is complex...each port is different..."

[1458] 3. The server's emotion engine recognizes that User A is "excited."

[1459] 4. The server uses the generative AI model to complement words like "multiple" and "different" and reflect the emotion of "excited," generating "The new portable charger has multiple ports that can be used in different ways."

[1460] 5. The server converts the completed text into audio data and synthesizes it with the original audio data.

[1461] 6. The server sends the synthesized voice data to User B's terminal, which receives it.

[1462] 7. When User B's device plays back the received audio data, he or she can hear the continuous audio saying, "The new portable charger has multiple ports that can be used in different ways."

[1463] In this way, the present invention smoothly compensates for interruptions in communication and realizes emotionally sensitive voice communication.

[1464] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1465] Step 1:

[1466] Capture audio data

[1467] input:

[1468] The voice the user speaks.

[1469] Specific behavior:

[1470] The device uses a microphone to capture the speaker's voice in real time, then compresses and encodes the captured voice data using the Opus codec, and temporarily stores the encoded voice data in a buffer.

[1471] output:

[1472] The encoded audio data.

[1473] Step 2:

[1474] Sending audio data

[1475] input:

[1476] The encoded audio data (output of step 1).

[1477] Specific behavior:

[1478] The device transmits the encoded audio data to the server in real time using the low-latency and highly reliable WebSocket protocol.

[1479] output:

[1480] The audio data sent to the server.

[1481] Step 3:

[1482] Receiving audio data

[1483] input:

[1484] Audio data sent from the device (output from step 2).

[1485] Specific behavior:

[1486] The server receives the voice data sent from the terminal in real time and stores it in a buffer.

[1487] output:

[1488] The audio data stored in the buffer.

[1489] Step 4:

[1490] Analysis of audio data

[1491] input:

[1492] The audio data stored in the buffer (output of step 3).

[1493] Specific behavior:

[1494] The server uses the Google Speech-to-Text API to convert the audio data in the buffer into text.

[1495] Detect gaps in speech data converted into text.

[1496] output:

[1497] Parsed audio data and gap information in text format.

[1498] Step 5:

[1499] Applying the Emotion Engine

[1500] input:

[1501] The analyzed audio data and gap information in text format (output of step 4).

[1502] Specific behavior:

[1503] The server uses an emotion engine such as IBM Watson's Tone Analyzer to analyze the user's emotions from the received text-format voice data.

[1504] The emotion information obtained by the emotion engine is stored for completion processing.

[1505] output:

[1506] Emotion analysis results.

[1507] Step 6:

[1508] Applying generative AI

[1509] input:

[1510] Context and sentiment analysis results (output of steps 4 and 5).

[1511] Specific behavior:

[1512] The server uses a generative AI model, such as OpenAI's GPT-3, to generate text to fill in the gaps based on the context and the results of the sentiment engine. The generated text contains natural, contextually appropriate expressions.

[1513] output:

[1514] The generated completion text.

[1515] Step 7:

[1516] Complementary data synthesis

[1517] input:

[1518] The generated completion text (output of step 6).

[1519] Specific behavior:

[1520] The server converts the generated text into speech using TTS technology such as the Google Text-to-Speech API, and then synthesizes the generated speech with the original speech to create a natural continuity.

[1521] output:

[1522] Synthesized voice data.

[1523] Step 8:

[1524] Sending complementary data

[1525] input:

[1526] The synthesized speech data (output of step 7).

[1527] Specific behavior:

[1528] The server transmits the synthesized voice data to the device in real time, again using the low-latency and highly reliable WebSocket protocol.

[1529] output:

[1530] Synthesized voice data sent to the device.

[1531] Step 9:

[1532] Receiving complementary data

[1533] input:

[1534] Synthesized speech data sent from the server (output of step 8).

[1535] Specific behavior:

[1536] The terminal receives the synthesized voice data sent from the server in real time and temporarily stores it in a buffer.

[1537] output:

[1538] The synthesized speech data stored in the buffer.

[1539] Step 10:

[1540] Playing audio data

[1541] input:

[1542] The synthesized speech data stored in the buffer (output of step 9).

[1543] Specific behavior:

[1544] The device decodes the received synthesized voice data and plays it through the speaker, allowing the user to hear the voice without interruption.

[1545] output:

[1546] The audio played.

[1547] (Application example 2)

[1548] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1549] Conventional voice communication systems suffer from the problem of voice interruptions when the communication environment is unstable. In particular, when interacting with customers in virtual stores, voice interruptions can be a significant factor in impairing the customer experience. Furthermore, conventional systems lack the ability to recognize user emotions, making it difficult to respond naturally and with consideration for the user's emotions. To address these issues, a system is needed that can compensate for communication interruptions and provide appropriate responses based on the user's emotions.

[1550] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1551] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data and detecting interrupted portions, means for applying a generative model to fill in the interrupted portions based on the analyzed voice data from the context and the recognized emotion, means for synthesizing the generated completed data with the original voice data, means for transmitting the synthesized voice data to another terminal, and an emotion engine for recognizing emotions. This enables smooth voice communication that takes emotion into consideration even when communication is interrupted.

[1552] "Voice data" refers to data obtained by converting the user's voice into digital form.

[1553] "Means for analyzing" refers to the processes and techniques for converting received voice data into text data and identifying gaps and voice characteristics.

[1554] A "generative model" is an artificial intelligence model trained on a large dataset and capable of automatically generating text and speech.

[1555] An "emotion engine" is a technology that analyzes emotions from a user's voice and recognizes their emotional state.

[1556] "Means of synthesis" refers to technology that naturally blends supplemented text or audio data with the original audio data.

[1557] "Communication quality" refers to the state of the network environment that allows voice data to be transmitted smoothly without interruption.

[1558] A "virtual store" refers to a virtual sales environment that provides products and services via the Internet.

[1559] "Communication with customers" refers to the exchange of voice and text between customer service staff and customers in a virtual store.

[1560] A system for implementing this invention includes a series of processes for receiving, analyzing, and completing voice data, and providing an appropriate response based on emotion. A specific method for implementing this system is described below.

[1561] System configuration

[1562] 1. Receiving audio data

[1563] The server receives the user's voice data in real time, and stores it in a temporary buffer. The hardware used here is a smartphone, smart glasses, or a head-mounted display, and the software uses WebSocket and an appropriate codec.

[1564] 2. Analysis of audio data

[1565] The server converts the received voice data into text using speech recognition technology (e.g., Google Speech-to-Text API), detects gaps in the converted text, and uses the analysis results in the next processing step. It also recognizes emotions using an emotion engine (e.g., IBM Watson Tone Analyzer).

[1566] 3. Applying generative AI

[1567] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate appropriate completion text based on context and sentiment information. Specifically, it uses the following example prompt:

[1568] Audio data: "The feature of this product is its long... different... battery life."

[1569] Emotion Engine: "Passionate"

[1570] Prompt: "Complete the following broken text based on context and sentiment: 'This product's feature is its long... different... battery life.'"

[1571] 4. Synthesis of complementary data

[1572] The completed text is converted to audio data (e.g., Google Text-to-Speech API) and then combined with the original audio data, resulting in a seamless, natural-sounding speech.

[1573] 5. Sending complementary data

[1574] The synthesized voice data is sent in real time to the customer's device, where it is decoded and played through a speaker.

[1575] Specific examples

[1576] Situation: The communication environment becomes poor while serving a customer in a virtual store, causing the customer service representative's voice to be interrupted.

[1577] 1. Customer Service Representative: "The special feature of this product is its long... different... battery life."

[1578] 2. Server: Analyzes the audio data and detects interruptions. At the same time, the emotion engine recognizes passionate emotions.

[1579] 3. Server: The generation AI generates complementary text such as "long duration" and "difference," adding emotional nuance.

[1580] 4. Server: The completed text is converted into audio data and combined with the original audio data.

[1581] 5. Server: Sends the synthesized voice data to the customer's device and plays the voice without interruption.

[1582] This system allows smooth communication with customers in virtual stores even in the event of a communication interruption, improving the customer experience.

[1583] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1584] Step 1:

[1585] Receiving audio data

[1586] The server receives the audio data sent from the device in real time. This audio data, which is input from the device, is temporarily stored in a buffer. Specifically, the server receives the audio data using WebSocket and buffers it to prevent data loss.

[1587] Step 2:

[1588] Analysis of audio data

[1589] The server converts the buffered audio data into text using speech recognition technology (e.g., Google Speech-to-Text API). The audio data is used as input and text data is generated as output. The specific operation here is to send the audio file to the speech recognition API and analyze the returned text.

[1590] Step 3:

[1591] Gap detection and emotion recognition

[1592] Detect discontinuities in the analyzed text data and simultaneously recognize emotions using an emotion engine (e.g., IBM Watson Tone Analyzer). The input data is the converted text, and the output is information about the discontinuities and emotion information. Specifically, discontinuities in the text are detected, and the text is input to the emotion analysis engine to obtain emotion labels.

[1593] Step 4:

[1594] Complemented by generative AI

[1595] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate text that completes the missing part based on context and emotional information. The input is the surrounding text and emotional information, and the output is the completed text. A specific example includes providing the following prompt to the generative AI:

[1596] Audio data: "The feature of this product is its long... different... battery life."

[1597] Emotion Engine: "Passionate"

[1598] Prompt: "Complete the following broken text based on context and sentiment: 'This product's feature is its long... different... battery life.'"

[1599] Step 5:

[1600] Complementary data synthesis

[1601] The completed text is converted back into audio data (e.g., Google Text-to-Speech API) and then synthesized with the original audio data. The input is the completed text and the original audio data, and the output is the synthesized audio data. Specifically, the completed text is sent to the text-to-speech API, which generates audio and then combines it with the original audio data.

[1602] Step 6:

[1603] Sending complementary data

[1604] The server transmits the synthesized voice data to other devices in real time. The input is the synthesized voice data, and the output is the retransmitted voice data. Specifically, the device receives the synthesized voice using WebSocket and plays it back in real time.

[1605] Step 7:

[1606] Playing audio

[1607] The device decodes the received audio data and plays it back to the user through the speaker. The input is the received audio data, and the output is the audio from the speaker. Specifically, the device sends the decoded audio data to the speaker and plays it back in a format that reaches the human ear.

[1608] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1609] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1610] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1611] [Fourth embodiment]

[1612] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1613] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1614] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1615] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1616] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1617] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1618] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1619] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1620] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1621] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1622] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1623] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1624] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1625] The present invention provides a system for solving the problem of audio data interruptions in online conferences and remote communications. This system can provide continuous audio without interruptions even in an unstable communication environment. An embodiment of this system will be described in detail below.

[1626] Overall system overview

[1627] This system receives voice data in real time, analyzes the data to detect interruptions, and uses generative artificial intelligence (AI) to complete the data based on the context. The completed data is then synthesized with the original voice data, and the final completed voice data is sent to another device.

[1628] Server Processing

[1629] Receiving audio data

[1630] The server receives the voice data sent from the device in real time using protocols such as WebSocket, which minimizes conversation delays.

[1631] Analysis of audio data

[1632] Automatic speech recognition (ASR) technology is used to analyze the received voice data and detect interruptions. The analysis results are saved as text data.

[1633] Applying generative AI

[1634] The server applies generative AI (e.g., a GPT model) to fill in gaps from the context of the parsed text data. The generative AI generates natural-sounding text that fits the gaps.

[1635] Complementary data synthesis

[1636] The generated supplemental text data is converted into audio data using text-to-speech (TTS) technology, and then the supplemental audio data is synthesized with the original audio data to create a coherent, continuous audio data.

[1637] Sending complementary data

[1638] The synthesized voice data is sent to other devices in real time, using a real-time communication protocol such as WebSocket.

[1639] Terminal handling

[1640] Capture audio data

[1641] The device captures the user's voice through a microphone and then compresses and encodes the data, allowing for efficient use of network bandwidth.

[1642] Sending audio data

[1643] The captured audio data is sent to the server using an appropriate protocol (e.g., Opus codec) to ensure stable communication.

[1644] Receiving complementary data

[1645] The terminal receives the completed voice data from the server, and the received data is played back to the user without delay.

[1646] Playing audio data

[1647] The device decodes the completed audio data and plays it through the speaker, allowing the user to continue the conversation without interruption.

[1648] User operations

[1649] Users can operate the device to capture their own voice and receive continuous voice data without interruption even if the communication is interrupted. Users can also check the quality of the captured voice and request retransmission if necessary.

[1650] Specific examples

[1651] Situation: When the communication environment is poor and the speaker's voice is interrupted

[1652] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[1653] 2. The server analyzes the received audio data, detects any interruptions, and converts them into text.

[1654] 3. The server uses a generation AI to generate complementary text such as "multiple" and "different."

[1655] 4. The server converts the completed text into audio data and synthesizes it with the original audio data.

[1656] 5. The server sends the completed voice data to User B's device, and User B's device plays the completed voice.

[1657] In this way, the system can provide smooth voice communication even when communication is interrupted.

[1658] The processing flow will be explained below.

[1659] Server Processing

[1660] Step 1: Receiving audio data

[1661] The server receives audio data sent from the device in real time via WebSocket, and the received data is temporarily stored in a buffer.

[1662] Example: let audioBuffer = await receiveAudioData();

[1663] Step 2: Analyzing the audio data

[1664] The server converts the audio data in the buffer into text using ASR technology and detects gaps. This analysis must be done quickly.

[1665] Example: let transcribedText = await speechToText(audioBuffer);

[1666] Detect gaps: let gaps = detectMissingSegments(transcribedText);

[1667] Step 3: Applying generative AI

[1668] The server uses generative AI (e.g., GPT model) to generate text to fill in the gaps from the context. The AI ​​generates the completed text in a way that maintains a natural flow of dialogue.

[1669] Example: let filledText = fillGapsWithAI(transcribedText, gaps);

[1670] Step 4: Synthesis of complementary data

[1671] The server synthesizes the generated supplemental text with the original speech data using text-to-speech (TTS) technology.

[1672] Example: let synthesizedAudio = textToSpeech(filledText);

[1673] Synthesize audio data: let combinedAudio = mergeAudioStreams(audioBuffer, synthesizedAudio);

[1674] Step 5: Sending complementary data

[1675] The server transmits the synthesized complete voice data to the other terminal in real time, providing uninterrupted voice to the receiving terminal.

[1676] Example: sendAudioDataToClients(combinedAudio);

[1677] Terminal handling

[1678] Step 1: Capture audio data

[1679] The terminal captures the user's voice in real time through a microphone and compresses and encodes the data.

[1680] Example: let capturedAudio = captureAudioFromMicrophone();

[1681] Step 2: Sending audio data

[1682] The device transmits the encoded audio data to the server using the WebSocket protocol, where appropriate error checking mechanisms are implemented to ensure reliable transmission.

[1683] Example: sendAudioDataToServer(capturedAudio);

[1684] Step 3: Receiving Complementary Data

[1685] The device receives the synthesized voice data sent from the server in real time, and the received data is stored in a buffer.

[1686] Example: let receivedAudio = await receiveAudioDataFromServer();

[1687] Step 4: Playing back audio data

[1688] The device decodes the received audio data and plays it through the speaker, providing the user with uninterrupted audio.

[1689] Example: playAudioData(receivedAudio);

[1690] User operations

[1691] Step 1: Check the audio quality

[1692] Users can check the quality of the audio played on their device and manually request correction if there are any interruptions.

[1693] Example: if(audioQualityIsPoor()) { requestManualCompletion();}

[1694] Step 2: Working with ongoing conversations

[1695] The user can smoothly continue the conversation based on the completed voice data, and can also request retransmission of the incorrectly completed data if necessary.

[1696] Example: if(incorrectCompletion()) { requestResend();}

[1697] These processing steps compensate for audio interruptions during online meetings, resulting in smoother communication.

[1698] Example 1

[1699] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1700] In online meetings and remote communications, an unstable communication environment can cause interruptions in audio data, hindering smooth communication and increasing user stress. Conventional technologies lack a mechanism for automatically and in real time compensating for such interrupted audio, so an effective system that solves this problem is needed.

[1701] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1702] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data to detect interrupted portions, means for applying artificial intelligence to fill in the interrupted portions based on the analyzed voice data from the context before and after, means for converting the generated supplemented data into voice data using text-to-speech technology, means for synthesizing the converted supplemented voice data with the original voice data, and means for transmitting the synthesized voice data to another terminal, thereby making it possible to provide uninterrupted, continuous voice in real time even in an unstable communication environment.

[1703] "Audio data" refers to data obtained by converting an audio signal acquired through an audio input device such as a microphone into a digital format.

[1704] "Means for receiving" refers to the technical means for obtaining audio data over a network and storing it in an appropriate data structure, typically using a communication protocol such as WebSocket.

[1705] The "analyzing means" refers to the technical means for processing the received voice data and detecting the interrupted parts. Automatic Speech Recognition (ASR) technology is typically used.

[1706] "Generative AI" is an abbreviation for generative artificial intelligence, and refers to a machine learning model for generating natural-sounding text and speech based on given input data.

[1707] "Text-to-speech technology" refers to the technical means for converting generated text data into digital audio signals. Text-to-speech (TTS) technology is commonly used.

[1708] "Synthesizing means" means the technical means for combining the converted audio data with the original audio data to create a coherent, continuous audio data.

[1709] "Transmission means" refers to the technical means for transmitting the synthesized voice data to another device in real time. Typically, a communication protocol such as WebSocket is used.

[1710] The "means for monitoring communication quality" refers to a technical means for monitoring the status of transmission and reception of voice data, and automatically triggering a compensation process if a problem such as voice interruption occurs.

[1711] This invention is a system that enables smooth voice communication even in an unstable communication environment. Specifically, it receives and analyzes voice data in real time and uses generative artificial intelligence (generative AI) to fill in any interrupted parts. The filled-in voice data is then synthesized with the original voice data and transmitted to another terminal as continuous voice. The following describes each component of the system and its specific operation.

[1712] Server Processing

[1713] The server first receives the audio data sent from the device in real time using the WebSocket protocol, a communication protocol that enables bidirectional data exchange with low latency.

[1714] The received voice data is converted into text using automatic speech recognition (ASR). This allows the server to identify the interrupted parts of the voice data. As a result of the ASR analysis, the resulting text might be something like, "The new portable charger has multiple ports...each port is different..."

[1715] Next, the server uses a generative AI (for example, a GPT-3 model) to generate text to complete the missing parts. The generated text includes completion parts such as "multiple" and "different." The following prompt sentence is used to apply the generative AI:

[1716] Example prompt sentence:

[1717] "The new portable charger has multiple ports...each port is different..."

[1718] The generated text is converted into audio data using Text-to-Speech (TTS) technology and then combined with the original audio data to create a consistent, continuous audio stream, which is then sent to other devices using the WebSocket protocol.

[1719] Terminal handling

[1720] The device captures the user's voice through a microphone, and this voice data is compressed and encoded using the Opus codec and sent to the server in a manner that efficiently uses network bandwidth.

[1721] The completed voice data sent from the server is received in real time by the device. The received voice data is decoded using the Opus codec and played back from the speaker. This allows users to continue a smooth conversation with continuous audio, regardless of communication interruptions.

[1722] User operations

[1723] Users can operate the device to capture their own voice and receive the completed voice data even if communication is interrupted. Users can also check the quality of the completed voice and request retransmission if necessary.

[1724] Specific examples

[1725] For example, consider a situation where a speaker's voice is interrupted due to a poor communication environment. User A's device sends the interrupted voice, "The new portable charger has multiple... each port is different..." to the server. The server analyzes the received voice data, detects the interrupted part, and converts it into text. Next, the server uses a generative AI (e.g., GPT-3 model) to generate complementary text such as "multiple" and "different." The server converts the complementary text into voice data and synthesizes it with the original voice data. Finally, the server sends the completed voice data to User B's device, which plays back the completed voice.

[1726] In this way, the system can provide smooth voice communication even when communication is interrupted.

[1727] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1728] Step 1: Capture audio data

[1729] Input: User's voice

[1730] Output: Captured audio data

[1731] The device uses a microphone to capture the user's voice, and the captured voice data is compressed and encoded in real time using the Opus codec, reducing the data size and allowing for efficient use of network bandwidth.

[1732] Step 2: Sending audio data

[1733] Input: Captured audio data

[1734] Output: Audio data sent to the server

[1735] The device sends the encoded audio data to the server using the WebSocket protocol, which allows for bidirectional, low-latency communication.

[1736] Step 3: Receiving audio data

[1737] Input: Audio data sent from the device

[1738] Output: Audio data stored on the server

[1739] The server receives the audio data sent from the device in real time. The received audio data is temporarily stored in a buffer. The WebSocket protocol is used to efficiently receive the data.

[1740] Step 4: Analyzing the audio data

[1741] Input: Audio data stored on the server

[1742] Output: Parsed text data

[1743] The server uses ASR (Automatic Speech Recognition) technology to analyze the received voice data and convert it into text data. During this process, the entire voice data is analyzed in detail to detect any interruptions. For example, text data such as "The new portable charger has multiple ports... each port is different..." can be obtained.

[1744] Step 5: Applying generative AI

[1745] Input: Parsed text data

[1746] Output: Generated completed text data

[1747] The server uses generative AI (e.g., the GPT-3 model) based on the analyzed text data to complete the missing parts, generating natural-looking completed text appropriate to the context. The generated text includes appropriate completed text such as "multiple" and "different."

[1748] Step 6: Converting the completed text to speech

[1749] Input: Generated completed text data

[1750] Output: Interpolated audio data

[1751] The server converts the generated supplemental text data into voice data using text-to-speech (TTS) technology. For example, the supplemental text "Multiple ports have different specifications" is converted into natural voice data.

[1752] Step 7: Synthesizing the audio data

[1753] Input: Original audio data and interpolated audio data

[1754] Output: Synthesized continuous audio data

[1755] The server then synthesizes the converted speech data with the original speech data, creating a consistent, continuous speech stream. The speech data is seamlessly integrated into the original speech data, creating a seamless audio stream.

[1756] Step 8: Sending the completed audio data

[1757] Input: Synthesized continuous speech data

[1758] Output: The completed audio data sent to the device.

[1759] The server then transmits the synthesized continuous audio data to other devices using the WebSocket protocol, which allows for efficient transmission of audio data in real time.

[1760] Step 9: Receiving the completed audio data

[1761] Input: Completed audio data sent from the server

[1762] Output: Completed audio data saved on the device

[1763] The terminal receives the completed voice data sent from the server in real time, and the received voice data is temporarily stored in a buffer.

[1764] Step 10: Decode and play the audio data

[1765] Input: Completed voice data saved on the device

[1766] Output: A continuous sound that is played to the user

[1767] The device then decodes the received audio data using the Opus codec and plays it back through the speaker, allowing the user to hear continuous audio without interruptions.

[1768] These steps enable the system to achieve smooth voice communication regardless of the instability of the communication environment.

[1769] (Application example 1)

[1770] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1771] The present invention aims to provide continuous audio without interruptions in audio data even in unstable communication environments during online conferences and remote communications. In the field of security monitoring, it is also an issue to provide a monitor with continuous audio without any sense of interruption even when audio data from a surveillance camera or sensor is interrupted.

[1772] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1773] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data to detect interrupted portions, means for applying artificial intelligence to fill in the interrupted portions based on the analyzed voice data from the context, means for synthesizing the generated completed voice data with the original voice data, means for transmitting the completed voice data to another terminal, and means for providing the completed voice data to the user's playback device in real time. This enables uninterrupted voice communication even in an environment with unstable communication. Furthermore, the monitor can monitor with uninterrupted, continuous voice.

[1774] "Audio data" is a digital representation of a sound signal captured by a microphone or other input device.

[1775] The "receiving means" is a device or module that has the function of acquiring audio data from another device or system.

[1776] "Means for analyzing" means any device or software having the processing capability to decompose received audio data and determine gaps and other characteristics.

[1777] A "gap" is a portion of the audio data where information is missing.

[1778] "Generative AI" is a collection of algorithms, such as machine learning models and neural networks, that are used to complement voice data.

[1779] "Complementary data" refers to a portion of the audio data that has been filled in with information generated by artificial intelligence to fill in the gaps.

[1780] The "synthesizing means" refers to a device or software that has the function of integrating the original audio data and the generated complementary data into one continuous audio data.

[1781] A "playback device" is a device such as a speaker or earphone that allows the user to listen to the synthesized audio data.

[1782] "Communication quality" refers to the state of the network environment when voice data is transmitted continuously without interruption.

[1783] A "triggering means" is a mechanism or function that automatically starts processing when a specific condition is met.

[1784] The present invention provides a system for preventing interruptions in voice data even in unstable communication environments during online conferences and remote communications. This system aims to improve security monitoring and voice call quality. Specific embodiments of this system are described below.

[1785] Overall system overview

[1786] The system receives and analyzes voice data in real time, detects interruptions, applies generative AI based on the context to complete the audio, and finally synthesizes the completed audio data and transmits it to other devices.

[1787] Server Processing

[1788] Receiving audio data

[1789] The server receives audio data sent from the terminal or security monitoring device in real time using the WebSocket protocol, allowing the server to minimize delays in conversation and monitoring.

[1790] Analysis of audio data

[1791] The received voice data is analyzed using automatic speech recognition (ASR). As a result of the analysis, any gaps are detected and the data is converted into text format.

[1792] Applying generative AI

[1793] The server applies generative AI (e.g., the GPT-4 model) based on the context of the analyzed text data to generate text to naturally complete the gaps.

[1794] Complementary data synthesis

[1795] The generated supplemental text data is converted into audio data using text-to-speech (TTS) technology and then combined with the original audio data, creating a continuous, uninterrupted audio stream.

[1796] Sending complementary data

[1797] The synthesized voice data is transmitted in real time to another device or monitoring device using the WebSocket protocol.

[1798] Terminal handling

[1799] Capture audio data

[1800] The device can use a microphone to capture the user's voice and compress and encode the voice data to use network bandwidth efficiently.

[1801] Sending audio data

[1802] The captured audio data is sent to the server, again using the WebSocket protocol.

[1803] Receiving complementary data

[1804] The terminal receives the complemented voice data from the server in real time.

[1805] Playing audio data

[1806] The device decodes the interpolated audio data and plays it back to the user through a speaker or earphones, allowing the user to have a conversation or monitor with uninterrupted audio.

[1807] User operations

[1808] Users can operate the device to capture their own voice and receive continuous voice data without interruption even if the communication is interrupted. Users can also check the quality of the captured voice and request retransmission if necessary.

[1809] Specific examples

[1810] Situation: When the communication environment is poor and the speaker's voice is interrupted

[1811] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[1812] 2. The server analyzes the received audio data, detects any interruptions, and converts them into text.

[1813] 3. The server uses a generation AI to generate complementary text such as "multiple" and "different."

[1814] 4. The server converts the completed text into audio data and synthesizes it with the original audio data.

[1815] 5. The server sends the completed voice data to User B's device, and User B's device plays the completed voice.

[1816] Prompt Sentence Examples

[1817] Audio data from ears: Warning, something on premises. ___Intrusion detected. ___

[1818] Generate context-aligned audio data.

[1819] In this way, the system can provide smooth voice communication and security monitoring even when communication is interrupted.

[1820] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1821] Step 1:

[1822] The server receives audio data from a terminal or security monitoring device in real time using the WebSocket protocol. The input is audio data, which the server stores in a buffer. The output is the stored audio data.

[1823] Step 2:

[1824] The server analyzes the received voice data using ASR technology. The input is the stored voice data, and the analysis converts the voice data into text format. The output is the text data that is the analysis result.

[1825] Step 3:

[1826] The server inspects the parsed text data and detects gaps. The input is the text data, and the detection process identifies the gaps. The output is the text data with the gaps indicated.

[1827] Step 4:

[1828] The server applies generative AI (GPT-4 model) based on the context of the analyzed text data to generate text to complete the gap. The input is the text data showing the gap, and the output is the completed text data. In this process, the following prompt sentence is used as input:

[1829] Audio data from ears: Warning, something on premises. ___Intrusion detected. ___

[1830] Generate context-aligned audio data.

[1831] Step 5:

[1832] The server converts the generated supplemented text data into audio data using text-to-speech (TTS) technology, with the input being the supplemented text data and the output being the supplemented audio data.

[1833] Step 6:

[1834] The server synthesizes the generated complementary speech data with the original speech data to create a single continuous speech data. The input is the complementary speech data and the original speech data, and the output is the synthesized continuous speech data.

[1835] Step 7:

[1836] The server transmits the synthesized voice data to another terminal or monitoring device in real time, where the input is the synthesized continuous voice data and the output is the voice data transmitted to the receiving terminal.

[1837] Step 8:

[1838] The terminal receives the completed voice data from the server in real time. The input is the voice data sent from the server, and the output is the voice data stored in the terminal's buffer.

[1839] Step 9:

[1840] The device decodes the interpolated audio data and plays it back to the user through a speaker or earphone. The input is the audio data stored in the buffer, and the output is the audio played back through the speaker or earphone. This allows the user to have a conversation or monitor with uninterrupted audio.

[1841] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1842] The present invention combines a system for solving the problem of voice data interruptions during online conferences and remote communication with an emotion engine that recognizes the user's emotions. This system provides uninterrupted, continuous voice even in unstable communication environments, and also enables emotion-sensitive communication. An embodiment of this system will be described in detail below.

[1843] Overall system overview

[1844] This system receives voice data in real time, analyzes it to detect interruptions, and uses generative artificial intelligence (AI) to complete the speech based on the context. The generated completed data is then combined with the original voice data, and the completed speech data is finally sent to another device. Furthermore, it uses an emotion engine to recognize the user's emotions and provides appropriate completion based on those emotions.

[1845] Server Processing

[1846] Receiving audio data

[1847] The server receives the voice data sent from the terminal in real time and temporarily stores the received data in a buffer.

[1848] Analysis of audio data

[1849] The server converts the audio data in the buffer into text using ASR technology and detects gaps in the audio data, which are then used in the next processing step.

[1850] Applying generative AI

[1851] The server uses generative AI (e.g., GPT model) to generate text to fill in the gaps based on the context and the emotions recognized by the emotion engine. The AI ​​generates natural-looking text that matches the flow of the conversation and the user's emotions.

[1852] Applying the Emotion Engine

[1853] The server uses an emotion engine to analyze and recognize the user's emotions from the received voice data. The emotion information recognized by the emotion engine is used to complement the generation AI.

[1854] Complementary data synthesis

[1855] The generated text is converted into speech data using text-to-speech (TTS) technology and then synthesized with the original speech data. The synthesized speech data is adjusted to ensure a natural continuation of the context and emotion.

[1856] Sending complementary data

[1857] The synthesized voice data is sent to another device in real time, allowing the user to receive uninterrupted voice.

[1858] Terminal handling

[1859] Capture audio data

[1860] The device captures the speaker's voice through a microphone, then compresses and encodes it, allowing for efficient use of network bandwidth.

[1861] Sending audio data

[1862] The captured audio data is sent to the server using an appropriate protocol (e.g. WebSocket, Opus codec) to ensure reliable transmission.

[1863] Receiving complementary data

[1864] The device receives the synthesized voice data sent from the server in real time, and the received data is temporarily stored in a buffer.

[1865] Playing audio data

[1866] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[1867] User operations

[1868] Checking Voice Quality

[1869] Users can check the quality of the audio played on their device and request corrections if there are any deficiencies or interruptions. Furthermore, if they feel the emotion recognition results are inappropriate, they can request readjustment.

[1870] Emotion-based responses

[1871] The user can continue to communicate more smoothly based on the emotions recognized by the emotion engine. The emotion engine analyzes the user's tone of voice and speaking style and sends the results to the server.

[1872] Specific examples

[1873] Situation: When the communication environment is poor and the speaker's voice is interrupted

[1874] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[1875] 2. The server analyzes the received audio data and detects gaps as text.

[1876] 3. The server analyzes User A's emotions using an emotion engine and recognizes, for example, that he is "excited."

[1877] 4. The server uses generative AI to complete words such as "multiple" and "different," and adds emotional nuance to the completed text based on the results of the emotion engine.

[1878] 5. The server converts the completed text into audio data and synthesizes it with the original audio data.

[1879] 6. The server sends the synthesized voice data to User B's device, which plays it back.

[1880] In this way, smooth and emotionally sensitive voice communication can be provided even when communication is interrupted.

[1881] The processing flow will be explained below.

[1882] Server Processing

[1883] Step 1: Receiving audio data

[1884] The server receives audio data sent from the device in real time using WebSocket. The received audio data is temporarily stored in a buffer.

[1885] Example: let audioBuffer = await receiveAudioData();

[1886] Step 2: Analyzing the audio data

[1887] The server analyzes the received voice data and converts it into text using ASR (Automatic Speech Recognition) technology. It then detects gaps in the converted text.

[1888] Example: let transcribedText = await speechToText(audioBuffer);

[1889] Detect gaps: let gaps = detectMissingSegments(transcribedText);

[1890] Step 3: Applying the Emotion Engine

[1891] The server uses an emotion engine to analyze the user's emotions from the received voice data, and the emotional information is used to complement the generative AI.

[1892] Example: let userEmotion = analyzeEmotion(audioBuffer);

[1893] Step 4: Applying generative AI

[1894] The server uses generative AI (e.g., a GPT model) to generate text to fill in the gaps based on the context and emotions recognized by the emotion engine.

[1895] Example: let filledText = fillGapsWithAI(transcribedText, gaps, userEmotion);

[1896] Step 5: Synthesizing complementary data

[1897] The server converts the generated text into speech data using text-to-speech (TTS) technology and synthesizes it with the original speech data, reconstructing it into a coherent, continuous speech data.

[1898] Example: let synthesizedAudio = textToSpeech(filledText);

[1899] Synthesize audio data: let combinedAudio = mergeAudioStreams(audioBuffer, synthesizedAudio);

[1900] Step 6: Sending complementary data

[1901] The server then transmits the synthesized voice data to other devices in real time, providing uninterrupted voice to the receiving device.

[1902] Example: sendAudioDataToClients(combinedAudio);

[1903] Terminal handling

[1904] Step 1: Capture audio data

[1905] The terminal uses a microphone to capture the user's voice in real time, then compresses and encodes the data.

[1906] Example: let capturedAudio = captureAudioFromMicrophone();

[1907] Step 2: Sending audio data

[1908] The device sends the encoded audio data to the server using the WebSocket protocol or similar, and a mechanism to ensure reliability is implemented.

[1909] Example: sendAudioDataToServer(capturedAudio);

[1910] Step 3: Receiving Complementary Data

[1911] The device receives the synthesized voice data sent from the server in real time, and the received data is stored in a buffer.

[1912] Example: let receivedAudio = await receiveAudioDataFromServer();

[1913] Step 4: Playing back audio data

[1914] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[1915] Example: playAudioData(receivedAudio);

[1916] User operations

[1917] Step 1: Check the audio quality

[1918] Users can check the quality of the audio played on their device and request corrections if there are any deficiencies or interruptions. Furthermore, if they feel the emotion recognition results are inappropriate, they can request readjustment.

[1919] Example: if(audioQualityIsPoor()) { requestManualCompletion();}

[1920] Example: if(emotionRecognitionIsIncorrect()) { requestEmotionReanalysis();}

[1921] Step 2: Respond based on emotions

[1922] Users can continue to communicate more smoothly based on the emotions recognized by the emotion engine. The emotion engine analyzes the user's tone of voice and speaking style and sends the results to the server.

[1923] Example: let userEmotion = captureEmotionDuringConversation();

[1924] Specific examples

[1925] Situation: When the communication environment is poor and the speaker's voice is interrupted

[1926] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[1927] 2. The server analyzes the received audio data, converts the audio into text, and detects gaps.

[1928] 3. The server analyzes User A's emotions using an emotion engine and recognizes, for example, that he is "excited."

[1929] 4. The server uses a generative AI to generate complementary text such as "multiple" and "different," and adds emotional nuance to the complementary text based on the results of the emotion engine.

[1930] 5. The server converts the generated supplemental text into audio data and synthesizes it with the original audio data.

[1931] 6. The server sends the synthesized voice data to User B's device, and User B's device plays the completed voice.

[1932] In this way, smooth and emotionally sensitive voice communication can be provided even when communication is interrupted.

[1933] Example 2

[1934] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1935] In conventional online conference systems and remote communication systems, voice data is often interrupted due to unstable communication environments, hindering smooth communication. Furthermore, because the system does not take into account the user's emotions, the quality of communication can be reduced by mechanical responses. To solve these problems, a system is needed that provides uninterrupted, continuous voice and communication that takes the user's emotions into consideration.

[1936] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1937] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data and detecting interrupted portions, means for applying a generative model to fill in the interrupted portions based on the analyzed voice data from the context, means for synthesizing the completed data with the original voice data, means for transmitting the synthesized voice data to another terminal, and means for analyzing the user's emotions from the received voice data and using the results in the completion process of the generative model, thereby enabling smooth voice communication that takes emotions into consideration even when communication is interrupted.

[1938] 1. "Audio data" means data that represents audio information such as conversation or music in digital form.

[1939] 2. "Means for receiving" means the technology or equipment for receiving audio data through a communications network.

[1940] 3. "Means for analyzing" means technology or devices that process received voice data to decipher and understand its content and characteristics.

[1941] 4. "Interrupted portion" refers to a portion where audio data is missing or interrupted due to an unstable communication environment or other reasons.

[1942] 5. "Means for detection" means the technology or device used to identify gaps in the analyzed audio data.

[1943] 6. A "generative model" is an algorithm or program that uses artificial intelligence techniques to generate text or speech.

[1944] 7. "Context" refers to the content of the audio data or text before and after the interrupted section, and is the information used to complete the sentence.

[1945] 8. "Completion means" refers to technologies or devices that use generative models to fill in gaps in data in a natural way.

[1946] 9. "Synthesizing means" means a technique or device that combines the generated complementary data with the original audio data to form one continuous audio signal.

[1947] 10. "Transmitting means" means the technology or device used to transmit the synthesized voice data to another terminal.

[1948] 11. "Terminal" means an electronic device for capturing, receiving, and playing audio data.

[1949] 12. "Means of analysis" refers to technology or devices that understand the content of audio data and extract specific information.

[1950] 13. "Means for analyzing emotions" refers to technology or devices that identify a user's emotions from voice data and use the results in subsequent processing.

[1951] 14. "Synthesized speech data" refers to the entire speech data to which the parts completed by the generative model have been added.

[1952] 15. "Real-time" means that data processing or communication is immediate and occurs within a time range that does not cause a delay to the user.

[1953] This invention provides a system that eliminates interruptions in voice data during online conferences and remote communications, and complements the data while taking into account the user's emotions. The system receives voice data in real time, analyzes the data to detect interruptions, and complements the data using a generative model based on the context and the user's emotions. The system also has the ability to synthesize the complemented voice data with the original voice data and transmit it to another device in real time.

[1954] Server Processing

[1955] 1. Receiving audio data

[1956] The server receives the voice data sent from the terminal in real time. The received data is temporarily stored in a buffer. The technology used here includes general network communication technology (e.g., TCP / IP).

[1957] 2. Analysis of audio data

[1958] The server uses speech recognition technology, such as the Google Speech-to-Text API, to convert the audio data in the buffer into text and detects gaps in the text data.

[1959] 3. Applying generative AI

[1960] The server uses a generative AI model (such as OpenAI's GPT-3) to generate text to fill in the gaps based on the context and the results of the emotion engine analysis. The generated text reflects natural expressions.

[1961] 4. Applying the Emotion Engine

[1962] The server uses an emotion engine to analyze and recognize the user's emotions from the received voice data. The emotion engine may be, for example, IBM Watson's Tone Analyzer. The emotion information detected by the emotion engine is used to complement the generative AI.

[1963] 5. Synthesis of complementary data

[1964] The generated text is converted into audio data using text-to-speech (TTS) technology, such as the Google Text-to-Speech API. The converted audio data is then synthesized with the original audio data to create a natural continuity.

[1965] 6. Sending supplementary data

[1966] The synthesized voice data is sent to the device in real time using a low-latency, highly reliable protocol (e.g., WebSocket).

[1967] Terminal handling

[1968] 1. Capture audio data

[1969] The device captures the speaker's voice in real time through a microphone, and the captured voice data is compressed and encoded using the Opus codec.

[1970] 2. Sending audio data

[1971] The encoded audio data is sent to the server in real time.

[1972] 3. Receiving complementary data

[1973] The synthesized voice data sent from the server is received in real time and temporarily stored in a buffer.

[1974] 4. Playing audio data

[1975] The device decodes the received audio data and plays it through the speaker, allowing the user to hear uninterrupted audio.

[1976] User operations

[1977] 1. Check the audio quality

[1978] The user can check the quality of the audio played on the device and request completion if there are gaps or interruptions.

[1979] 2. Emotional responses

[1980] The user can smoothly communicate based on the emotions recognized by the emotion engine.

[1981] Prompt Sentence Examples

[1982] "Hello, how was your day?"

[1983] These concise prompts form the basis for applying generative AI models.

[1984] Examples:

[1985] 1. User A's device sends a broken voice message to the server saying, "The new portable charger has multiple ports...each port is different..."

[1986] 2. The server analyzes the received voice data and converts the interrupted part into text: "The new portable charger is complex...each port is different..."

[1987] 3. The server's emotion engine recognizes that User A is "excited."

[1988] 4. The server uses the generative AI model to complement words like "multiple" and "different" and reflect the emotion of "excited," generating "The new portable charger has multiple ports that can be used in different ways."

[1989] 5. The server converts the completed text into audio data and synthesizes it with the original audio data.

[1990] 6. The server sends the synthesized voice data to User B's terminal, which receives it.

[1991] 7. When User B's device plays back the received audio data, he or she can hear the continuous audio saying, "The new portable charger has multiple ports that can be used in different ways."

[1992] In this way, the present invention smoothly compensates for interruptions in communication and realizes emotionally sensitive voice communication.

[1993] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1994] Step 1:

[1995] Capture audio data

[1996] input:

[1997] The voice the user speaks.

[1998] Specific behavior:

[1999] The device uses a microphone to capture the speaker's voice in real time, then compresses and encodes the captured voice data using the Opus codec, and temporarily stores the encoded voice data in a buffer.

[2000] output:

[2001] The encoded audio data.

[2002] Step 2:

[2003] Sending audio data

[2004] input:

[2005] The encoded audio data (output of step 1).

[2006] Specific behavior:

[2007] The device transmits the encoded audio data to the server in real time using the low-latency and highly reliable WebSocket protocol.

[2008] output:

[2009] The audio data sent to the server.

[2010] Step 3:

[2011] Receiving audio data

[2012] input:

[2013] Audio data sent from the device (output from step 2).

[2014] Specific behavior:

[2015] The server receives the voice data sent from the terminal in real time and stores it in a buffer.

[2016] output:

[2017] The audio data stored in the buffer.

[2018] Step 4:

[2019] Analysis of audio data

[2020] input:

[2021] The audio data stored in the buffer (output of step 3).

[2022] Specific behavior:

[2023] The server uses the Google Speech-to-Text API to convert the audio data in the buffer into text.

[2024] Detect gaps in speech data converted into text.

[2025] output:

[2026] Parsed audio data and gap information in text format.

[2027] Step 5:

[2028] Applying the Emotion Engine

[2029] input:

[2030] The analyzed audio data and gap information in text format (output of step 4).

[2031] Specific behavior:

[2032] The server uses an emotion engine such as IBM Watson's Tone Analyzer to analyze the user's emotions from the received text-format voice data.

[2033] The emotion information obtained by the emotion engine is stored for completion processing.

[2034] output:

[2035] Emotion analysis results.

[2036] Step 6:

[2037] Applying generative AI

[2038] input:

[2039] Context and sentiment analysis results (output of steps 4 and 5).

[2040] Specific behavior:

[2041] The server uses a generative AI model, such as OpenAI's GPT-3, to generate text to fill in the gaps based on the context and the results of the sentiment engine. The generated text contains natural, contextually appropriate expressions.

[2042] output:

[2043] The generated completion text.

[2044] Step 7:

[2045] Complementary data synthesis

[2046] input:

[2047] The generated completion text (output of step 6).

[2048] Specific behavior:

[2049] The server converts the generated text into speech using TTS technology such as the Google Text-to-Speech API, and then synthesizes the generated speech with the original speech to create a natural continuity.

[2050] output:

[2051] Synthesized voice data.

[2052] Step 8:

[2053] Sending complementary data

[2054] input:

[2055] The synthesized speech data (output of step 7).

[2056] Specific behavior:

[2057] The server transmits the synthesized voice data to the device in real time, again using the low-latency and highly reliable WebSocket protocol.

[2058] output:

[2059] Synthesized voice data sent to the device.

[2060] Step 9:

[2061] Receiving complementary data

[2062] input:

[2063] Synthesized speech data sent from the server (output of step 8).

[2064] Specific behavior:

[2065] The terminal receives the synthesized voice data sent from the server in real time and temporarily stores it in a buffer.

[2066] output:

[2067] The synthesized speech data stored in the buffer.

[2068] Step 10:

[2069] Playing audio data

[2070] input:

[2071] The synthesized speech data stored in the buffer (output of step 9).

[2072] Specific behavior:

[2073] The device decodes the received synthesized voice data and plays it through the speaker, allowing the user to hear the voice without interruption.

[2074] output:

[2075] The audio played.

[2076] (Application example 2)

[2077] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2078] Conventional voice communication systems suffer from the problem of voice interruptions when the communication environment is unstable. In particular, when interacting with customers in virtual stores, voice interruptions can be a significant factor in impairing the customer experience. Furthermore, conventional systems lack the ability to recognize user emotions, making it difficult to respond naturally and with consideration for the user's emotions. To address these issues, a system is needed that can compensate for communication interruptions and provide appropriate responses based on the user's emotions.

[2079] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2080] In this invention, the server includes means for receiving voice data, means for analyzing the received voice data and detecting interrupted portions, means for applying a generative model to fill in the interrupted portions based on the analyzed voice data from the context and the recognized emotion, means for synthesizing the generated completed data with the original voice data, means for transmitting the synthesized voice data to another terminal, and an emotion engine for recognizing emotions. This enables smooth voice communication that takes emotion into consideration even when communication is interrupted.

[2081] "Voice data" refers to data obtained by converting the user's voice into digital form.

[2082] "Means for analyzing" refers to the processes and techniques for converting received voice data into text data and identifying gaps and voice characteristics.

[2083] A "generative model" is an artificial intelligence model trained on a large dataset and capable of automatically generating text and speech.

[2084] An "emotion engine" is a technology that analyzes emotions from a user's voice and recognizes their emotional state.

[2085] "Means of synthesis" refers to technology that naturally blends supplemented text or audio data with the original audio data.

[2086] "Communication quality" refers to the state of the network environment that allows voice data to be transmitted smoothly without interruption.

[2087] A "virtual store" refers to a virtual sales environment that provides products and services via the Internet.

[2088] "Communication with customers" refers to the exchange of voice and text between customer service staff and customers in a virtual store.

[2089] A system for implementing this invention includes a series of processes for receiving, analyzing, and completing voice data, and providing an appropriate response based on emotion. A specific method for implementing this system is described below.

[2090] System configuration

[2091] 1. Receiving audio data

[2092] The server receives the user's voice data in real time, and stores it in a temporary buffer. The hardware used here is a smartphone, smart glasses, or a head-mounted display, and the software uses WebSocket and an appropriate codec.

[2093] 2. Analysis of audio data

[2094] The server converts the received voice data into text using speech recognition technology (e.g., Google Speech-to-Text API), detects gaps in the converted text, and uses the analysis results in the next processing step. It also recognizes emotions using an emotion engine (e.g., IBM Watson Tone Analyzer).

[2095] 3. Applying generative AI

[2096] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate appropriate completion text based on context and sentiment information. Specifically, it uses the following example prompt:

[2097] Audio data: "The feature of this product is its long... different... battery life."

[2098] Emotion Engine: "Passionate"

[2099] Prompt: "Complete the following broken text based on context and sentiment: 'This product's feature is its long... different... battery life.'"

[2100] 4. Synthesis of complementary data

[2101] The completed text is converted to audio data (e.g., Google Text-to-Speech API) and then combined with the original audio data, resulting in a seamless, natural-sounding speech.

[2102] 5. Sending complementary data

[2103] The synthesized voice data is sent in real time to the customer's device, where it is decoded and played through a speaker.

[2104] Specific examples

[2105] Situation: The communication environment becomes poor while serving a customer in a virtual store, causing the customer service representative's voice to be interrupted.

[2106] 1. Customer Service Representative: "The special feature of this product is its long... different... battery life."

[2107] 2. Server: Analyzes the audio data and detects interruptions. At the same time, the emotion engine recognizes passionate emotions.

[2108] 3. Server: The generation AI generates complementary text such as "long duration" and "difference," adding emotional nuance.

[2109] 4. Server: The completed text is converted into audio data and combined with the original audio data.

[2110] 5. Server: Sends the synthesized voice data to the customer's device and plays the voice without interruption.

[2111] This system allows smooth communication with customers in virtual stores even in the event of a communication interruption, improving the customer experience.

[2112] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2113] Step 1:

[2114] Receiving audio data

[2115] The server receives the audio data sent from the device in real time. This audio data, which is input from the device, is temporarily stored in a buffer. Specifically, the server receives the audio data using WebSocket and buffers it to prevent data loss.

[2116] Step 2:

[2117] Analysis of audio data

[2118] The server converts the buffered audio data into text using speech recognition technology (e.g., Google Speech-to-Text API). The audio data is used as input and text data is generated as output. The specific operation here is to send the audio file to the speech recognition API and analyze the returned text.

[2119] Step 3:

[2120] Gap detection and emotion recognition

[2121] Detect discontinuities in the analyzed text data and simultaneously recognize emotions using an emotion engine (e.g., IBM Watson Tone Analyzer). The input data is the converted text, and the output is information about the discontinuities and emotion information. Specifically, discontinuities in the text are detected, and the text is input to the emotion analysis engine to obtain emotion labels.

[2122] Step 4:

[2123] Complemented by generative AI

[2124] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate text that completes the missing part based on context and emotional information. The input is the surrounding text and emotional information, and the output is the completed text. A specific example includes providing the following prompt to the generative AI:

[2125] Audio data: "The feature of this product is its long... different... battery life."

[2126] Emotion Engine: "Passionate"

[2127] Prompt: "Complete the following broken text based on context and sentiment: 'This product's feature is its long... different... battery life.'"

[2128] Step 5:

[2129] Complementary data synthesis

[2130] The completed text is converted back into audio data (e.g., Google Text-to-Speech API) and then synthesized with the original audio data. The input is the completed text and the original audio data, and the output is the synthesized audio data. Specifically, the completed text is sent to the text-to-speech API, which generates audio and then combines it with the original audio data.

[2131] Step 6:

[2132] Sending complementary data

[2133] The server transmits the synthesized voice data to other devices in real time. The input is the synthesized voice data, and the output is the retransmitted voice data. Specifically, the device receives the synthesized voice using WebSocket and plays it back in real time.

[2134] Step 7:

[2135] Playing audio

[2136] The device decodes the received audio data and plays it back to the user through the speaker. The input is the received audio data, and the output is the audio from the speaker. Specifically, the device sends the decoded audio data to the speaker and plays it back in a format that reaches the human ear.

[2137] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2138] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2139] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2140] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2141] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2142] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2143] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2144] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2145] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2146] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2147] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2148] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2149] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2150] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2151] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2152] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2153] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2154] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2155] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2156] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2157] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2158] The following is further disclosed regarding the above embodiment.

[2159] (Claim 1)

[2160] means for receiving audio data;

[2161] means for analyzing received audio data and detecting interrupted portions;

[2162] and applying generative artificial intelligence to fill in gaps in the speech from the context based on the analyzed speech data.

[2163] means for synthesizing the generated complementary data with the original voice data;

[2164] The system includes means for transmitting the synthesized voice data to another terminal.

[2165] (Claim 2)

[2166] a means for converting the audio data into text;

[2167] A means for complementing text data using generative artificial intelligence;

[2168] 10. The system of claim 1, further comprising means for converting the completed text data back into audio data.

[2169] (Claim 3)

[2170] Real-time monitoring of voice data communication quality

[2171] 10. The system of claim 1, further comprising means for automatically triggering a completion process when a gap occurs.

[2172] "Example 1"

[2173] (Claim 1)

[2174] means for receiving audio data;

[2175] means for analyzing received audio data and detecting interrupted portions;

[2176] and applying generative artificial intelligence to fill in gaps in the speech from the context based on the analyzed speech data.

[2177] means for converting the generated complementary data into speech data using text-to-speech technology;

[2178] means for synthesizing the converted complementary voice data with the original voice data;

[2179] The system includes means for transmitting the synthesized voice data to another terminal.

[2180] (Claim 2)

[2181] a means for converting the audio data into text;

[2182] A means for complementing text data using generative artificial intelligence;

[2183] 10. The system of claim 1, further comprising means for converting the imputed text data back into audio data using text-to-speech technology.

[2184] (Claim 3)

[2185] 2. The system according to claim 1, further comprising means for using the WebSocket protocol to ensure communication stability and minimize delays in each of the processing steps of receiving, analyzing, completing, converting, synthesizing, and transmitting.

[2186] (Claim 4)

[2187] 2. The system according to claim 1, further comprising means for monitoring communication quality of voice data in real time and automatically triggering a compensation process when an interruption occurs.

[2188] "Application Example 1"

[2189] (Claim 1)

[2190] means for receiving audio data;

[2191] means for analyzing received audio data and detecting interrupted portions;

[2192] and applying generative artificial intelligence to fill in gaps in the speech from the context based on the analyzed speech data.

[2193] means for synthesizing the generated complementary data with the original voice data;

[2194] means for transmitting the synthesized voice data to another terminal;

[2195] A system including means for providing the interpolated audio data to a user's playback device in real time.

[2196] (Claim 2)

[2197] a means for converting the audio data into text;

[2198] A means for complementing text data using generative artificial intelligence;

[2199] means for reconverting the complemented text data into audio data;

[2200] 10. The system of claim 1, further comprising means for playing the completed audio data to a user's visual and / or auditory devices.

[2201] (Claim 3)

[2202] Real-time monitoring of voice data communication quality

[2203] means for automatically triggering a completion process when a discontinuity occurs;

[2204] 10. The system of claim 1, further comprising means for playing the interpolated audio data to the user without delay.

[2205] "Example 2: Combining Emotion Engines"

[2206] (Claim 1)

[2207] means for receiving audio data;

[2208] means for analyzing received audio data and detecting interrupted portions;

[2209] a means for applying a generative model to fill in gaps from the context based on the analyzed speech data;

[2210] means for combining the complemented data with the original audio data;

[2211] means for transmitting the synthesized voice data to another terminal;

[2212] A system that includes a means for analyzing a user's emotions from received voice data and utilizing the results to complement the generative model.

[2213] (Claim 2)

[2214] a means for converting the audio data into text;

[2215] a means for complementing text data with a generative model;

[2216] 10. The system of claim 1, further comprising means for converting the completed text data back into audio data.

[2217] (Claim 3)

[2218] Real-time monitoring of voice data communication quality

[2219] means for automatically triggering a completion process when a discontinuity occurs;

[2220] 10. The system of claim 1, further comprising means for playing the interpolated audio data to a receiving terminal in real time.

[2221] "Application example 2 when combining emotion engines"

[2222] (Claim 1)

[2223] means for receiving audio data;

[2224] means for analyzing received audio data and detecting interrupted portions;

[2225] means for applying a generative model to fill in gaps from the context and the recognized emotion based on the analyzed speech data;

[2226] means for synthesizing the generated complementary data with the original voice data;

[2227] means for transmitting the synthesized voice data to another terminal;

[2228] A system including an emotion engine for recognizing emotions.

[2229] (Claim 2)

[2230] a means for converting the audio data into text;

[2231] a means for complementing text data with a generative model;

[2232] 10. The system of claim 1, further comprising means for converting the completed text data back into audio data.

[2233] (Claim 3)

[2234] Real-time monitoring of voice data communication quality

[2235] means for automatically triggering a completion process when a discontinuity occurs;

[2236] The system of claim 1, used in communicating with customers in a virtual store. [Explanation of symbols]

[2237] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving audio data; means for analyzing received audio data and detecting interrupted portions; and applying generative artificial intelligence to fill in gaps in the speech from the context based on the analyzed speech data. means for synthesizing the generated complementary data with the original voice data; The system includes means for transmitting the synthesized voice data to another terminal.

2. a means for converting the audio data into text; A means for complementing text data using generative artificial intelligence; 2. The system of claim 1, further comprising means for converting the completed text data back into audio data.

3. Real-time monitoring of voice data communication quality 2. The system of claim 1, further comprising means for automatically triggering a compensation process when a discontinuity occurs.

Images