Enhanced speech reproduction on computing systems

By identifying and enhancing text segments of audio signals in a computing system, and generating and synthesizing enhanced speech segments, the problems of background noise and network performance affecting speech quality are solved, achieving clearer speech reproduction and more efficient utilization of network resources.

CN116783648BActive Publication Date: 2026-11-17INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202280009931.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-01-20
Filing Date
2022-01-05
Publication Date
2026-11-17
Estimated Expiration
2042-01-05

AI Technical Summary

Technical Problem

In computing systems, the quality of speech reproduction is affected by background noise and poor telecommunications network performance, leading to a decline in speech quality and impacting communication effectiveness.

Method used

By recognizing text segments in audio signals, enhanced text segments are generated and enhanced speech segments are synthesized. Speech quality is improved by using speech recognition and synthesis technologies, especially by enhancing text segments whose quality indicators have not reached the threshold.

Benefits of technology

It improves the clarity of voice reproduction, reduces the impact of noise and distortion, enhances communication effectiveness, and reduces the consumption of telecommunications network resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116783648B_ABST
    Figure CN116783648B_ABST
Patent Text Reader

Abstract

This disclosure relates to a method for enhancing speech reproduction based on identifying text segments from sound segments of an audio signal. The enhanced text segments are generated from any text segments whose quality indicators do not reach a threshold level, and enhanced speech segments are synthesized from the enhanced text segments. Reproduction based on the enhanced speech segments includes the enhanced sound of the enhanced sound segments.
Need to check novelty before this filing date? Find Prior Art

Description

Background Technology

[0001] This disclosure relates to the field of information technology. More specifically, the present invention relates to the reproduction of speech.

[0002] Computing systems are typically used to output (digital) content in various formats. Specifically, one of the most effective formats for conveying information to users of a computing system is speech; for this purpose, most computing systems are equipped with devices, such as speakers, for reproducing sound, including speech from corresponding audio signals.

[0003] A typical example is a voice or conference call, used to provide a live exchange of sound between two or more users on corresponding computing systems (referred to as participants) over a telecommunications network, potentially with the additional sharing of multimedia content (e.g., video, images, data, documents, etc.). Typically, each voice / conference call is executed over the internet (e.g., by utilizing VoIP technology) to allow participants to communicate from geographically dispersed remote locations. These calls enable discussions of topics of common interest in a time- and cost-effective manner (eliminating the need for face-to-face meetings, which would typically require participants to travel long distances to reach a shared physical location); this becomes even more important in the final stretches of the day due to the widespread adoption of smart work in many offices.

[0004] The effectiveness of each call depends significantly on the quality of the reproduction of the speech of each participant in the call (included in the corresponding sounds received by the other participants in the call); the true integrity and fidelity of the speech reproduction are important to ensure that each participant speaking in the call is understood by all the other participants listening to him / her.

[0005] However, several factors can adversely affect the reproduction of speech during a call. For example, background noise present in the context of each participant's speech may combine with their speech to form the corresponding sound transmitted to other participants; this noise may hinder the understanding of the content of the speech. Noise reduction techniques can then be used to remove (at least partially) the noise from the sound. Specifically, passive techniques (such as soundproof rooms) can be used to reduce the generation of noise at the participants' locations; however, these techniques are very expensive and difficult (if not impossible) to implement in many situations (such as in private settings). Active techniques can be used to reduce noise in the transmitted or received sound. For example, cancellation interference techniques can be applied to reduce noise in the sound at the location of the speaking participant, and / or filtering techniques can be applied to reduce noise in the sound at the location of the listening participant.

[0006] In any case, the quality of received voice can be adversely affected by the performance of the telecommunications network used to transmit the corresponding audio signal. In fact, poor performance of the telecommunications network (e.g., low speed, high bandwidth, etc.) creates the illusion of degraded voice quality; for example, the voice may become noisy, distorted, intermittent, etc. Summary of the Invention

[0007] A simplified overview of this disclosure is presented herein to provide a basic understanding of it; however, the sole purpose of this overview is to introduce some concepts of this disclosure in a simplified form as a prelude to its more detailed description thereafter, and it is not to be construed as an identification of its key elements or a definition of its scope.

[0008] Generally speaking, this disclosure is based on the idea of ​​synthesizing speech from enhanced text recognized from speech.

[0009] Specifically, an embodiment provides a method for enhancing speech reproduction. The method includes identifying text segments from sound segments of an audio signal. Enhanced text segments are generated from any text segments whose quality indicators do not reach a threshold level, and enhanced speech segments are synthesized from the enhanced text segments. Reproduction based on the enhanced speech segments includes enhanced sound from the enhanced sound segments.

[0010] On the other hand, a computer program for implementing the method is provided.

[0011] On the other hand, a corresponding computer program product is provided.

[0012] On the other hand, a corresponding system is provided.

[0013] More specifically, one or more aspects of this disclosure are set forth in the independent claims and their advantageous features are set forth in the dependent claims, wherein the wording of all claims is incorporated herein by reference verbatim (any advantageous feature is provided with reference to any particular aspect which, with necessary modifications, applies to each of the other aspects). Attached Figure Description

[0014] The present disclosure and its further features and advantages will be better understood by referring to the following detailed description of the present disclosure, which is given only in a non-limiting manner and read in conjunction with the accompanying drawings (wherein, for simplicity, corresponding elements are indicated by the same or similar reference numerals and their explanations are not repeated, and the name of each entity is generally used to indicate its type and its attributes, such as value, content and representation).

[0015] Figures 1A-1D The general principles of the solution according to embodiments of this disclosure are illustrated;

[0016] Figure 2A schematic block diagram is shown of a computing infrastructure in which solutions according to embodiments of the present disclosure can be implemented;

[0017] Figure 3 The main software components that can be used to implement solutions according to embodiments of this disclosure are shown; and

[0018] Figures 4A to 4B A flowchart describing the activities related to the implementation of a solution according to an embodiment of this disclosure is shown. Detailed Implementation

[0019] For details, please refer to the following: Figures 1A to 1D This illustrates the general principles of embodiments of the present disclosure.

[0020] from Figure 1A Initially, users (participants) of the computing system can participate in a call via a telecommunications network (e.g., a conference call between multiple participants who are typically geographically dispersed and far apart from each other). During the call, participants can speak or listen; in the latter case, the computing system receives an audio signal (in a digital form readable by the computing system) representing the sounds of other participants speaking (i.e., pressure waves generated by vibrations) (i.e., speech communication of human language defined by a socially understood structured system, such as English). Text (represented as speech in a human-readable form) can be identified from the audio signal; this text can include a sequence of text fragments Ti, where i = 1…N (e.g., words and pauses), which can be identified from the corresponding sequence of speech fragments Si. Simultaneously, a corresponding quality indicator Qi can be determined for each text fragment Ti; the quality indicator Qi provides an indication of the correctness of the corresponding text fragment Ti, indicating that the speech fragment Si from which the text fragment has been identified.

[0021] Move to Figure 1B According to embodiments of this disclosure, a corresponding enhanced text segment ETi can be generated from any text segment Ti (underlined in the figure) whose quality indicator Qi does not reach a threshold level (meaning that its accuracy in representing the corresponding text segment Ti is too low); the enhanced text segment ETi can be generated based on additional information about the corresponding speech segment Si (e.g., the context of the speech inferred from the text).

[0022] Move to Figure 1C It is possible to synthesize corresponding enhanced speech segments ESi from enhanced text segments ETi.

[0023] Move to Figure 1DThe enhanced sound can be reproduced. The enhanced sound may include a sound segment Si and an enhanced sound segment based on the enhanced speech segment ESi (e.g., obtained by mixing its speech with the background of the corresponding segment Si); specifically, the sound segment Si can be reproduced for a text segment Ti whose quality indicator Qi reaches a threshold level, and the enhanced sound segment (based on the enhanced speech segment ESi) can be reproduced for a text segment Ti whose quality indicator Qi does not reach a threshold level. Thus, the enhanced sound can provide the participant's (original) speech (when intelligible); however, whenever the participant's speech becomes incomprehensible, it can be replaced with corresponding synthesized speech that reproduces its inferred content.

[0024] The above embodiments can significantly improve call quality. In fact, in this way, the effects of any artifacts in the sound can be removed (or at least substantially attenuated), which contribute to noise, distortion, intermittency, etc. (due to added background noise or poor telecommunications network performance). Therefore, participants can more easily understand the speech; thus, participants can more easily listen to the speech and then focus on its content, with a reduced risk of missing key points.

[0025] Improved call quality can also avoid (or at least greatly reduce) the need for additional information exchange over the telecommunications network (e.g., retransmission of poor quality or completely lost parts of voice); this limits the consumption of corresponding network resources and the potential degradation of telecommunications network performance (which has a beneficial effect on all network users).

[0026] Now for reference Figure 2 A schematic block diagram of a computing infrastructure 200 is shown, in which embodiments of the present disclosure may be implemented.

[0027] The computing infrastructure 200 includes multiple client computing systems, or simply clients 205 used by participants for calls, and server computing systems, or simply servers 210, for managing calls; for example, client 205 is a participant's personal computer (PC), and server 210 is implemented in the data center of a service provider offering the corresponding call service. Client 105 and server 110 are connected to telecommunications network 215 (e.g., Internet-based) to communicate between them.

[0028] Each of the aforementioned computing systems (i.e., client 205 and server 210) includes several units connected among them via a bus structure 220 having one or more levels (with an architecture appropriately scaled according to the type of computing systems 205, 210). Specifically, a microprocessor (μP) 225 or more provides the logical capabilities of computing systems 205, 210; non-volatile memory (ROM) 230 stores the basic code for booting computing systems 205, 210, and volatile memory (RAM) 235 is used as working memory by the microprocessor 225. Computing systems 205, 210 have a large-capacity memory 240 for storing programs and data (e.g., an SSD for each client 205 and a corresponding data center storage device for the server 210). Furthermore, computing systems 205 and 210 include multiple controllers for peripheral devices or input / output (I / O) units 245; for example, as long as relevant to this disclosure, the peripheral device 245 of each client 205 includes a network adapter (NIC) for connecting to the telecommunications network 215, a microphone for recording (input) sound, the sound including the (input) voice of the corresponding participant in the call; a speaker for reproducing (output) sound, the sound including the (output) voice of other participants in the call, and a drive for reading / writing removable storage units (such as USB type), while the peripheral device 245 of server 210 includes a network card for inserting server 210 into a data center and then connecting it to a console in the data center for its control (e.g., a personal computer, which also has a drive for reading / writing removable storage units such as USB type) and a switch / router subsystem for accessing the data center of the telecommunications network 215.

[0029] Now for reference Figure 3 The main software components that can be used to implement embodiments of this disclosure are shown.

[0030] Specifically, all software components (programs and data) are collectively indicated by reference numeral 300. Software component 300 is typically stored in mass storage and is loaded (at least partially) into the working memory of each client 205 (only one shown in the figure) when the program is run along with the operating system and other applications (not shown). The program is initially installed into mass storage, for example, from a removable storage unit or from a telecommunications network. In this respect, each program can be a module, fragment, or section of code, comprising one or more executable instructions for implementing a specified logical function.

[0031] Network driver 305 drives the network adapter, microphone driver 310 drives the microphone, and speaker driver 315 drives the speaker. Call agent 320 controls each ongoing call on client 205 for its user as a participant. Call agent 320 utilizes network driver 305 to communicate with a call manager running on a server to manage calls (not shown). Call agent 320 utilizes microphone driver 310 to register input sounds (including any input voice of its participants) at client 205 as corresponding (input) audio signals transmitted via call manager to other participants' clients. Call agent 320 writes to playback buffer 325, which is read by speaker driver 315. Playback buffer 325 is used to store (output) audio signals representing output sounds received via call manager from other participants' clients, including any output voice; playback buffer 325 has a FIFO structure for temporarily storing output audio signals (e.g., having a capacity corresponding to 5-10 minutes of playback of output sounds) so that playback of output sounds is allowed without interruption even in the event of brief transmission problems of the output audio signals.

[0032] In embodiments of this disclosure, call agent 320 utilizes quality enhancer 360; as described above, quality enhancer 360 is used to enhance the quality of the reproduction of the output speech (of other participants in the call). For this purpose, quality enhancer 360 utilizes speech recognizer 330 (e.g., IBM's Watson speech-to-text); speech recognizer 330 is used to recognize text from the output speech (contained in the output sound represented in the output audio signal). For this purpose, speech recognizer 330 reads a general vocabulary 335g and multiple contextual vocabulary 335c. The general vocabulary 335g is generic; the contextual vocabulary 335c, on the other hand, is specific to different contexts, such as information technology, electronics, marketing, finance, sports, etc. (where each context has one or more corresponding contextual vocabulary 335c).

[0033] Each (general / contextual) vocabulary 335g, 335c defines a set of corresponding words that can be recognized in speech (each word is defined by a minimal sequence of phonemes that can be independently expressed in the meaning of the corresponding language); for each word, vocabulary 335g, 335c indicates its written representation (defined by one or more morphemes, such as letters in the alphabet) and one or more of its pronunciations (defined by its phonemes, such as in IPA representation). Speech recognizer 330 writes to text file 340; text file 340 stores a representation (in digital form) of the text that has been recognized (or at least its most recent portion, such as the last 30-60 minutes of the output sound). Cognitive engine 345 (e.g., based on a neural network) determines the context of the call from the corresponding text. Cognitive engine 345 reads text file 340. Furthermore, cognitive engine 345 writes to context register 350, which is read by quality enhancer 360; context register 350 stores an indication of the context of the call, along with one or more metadata defining it. The quality enhancer 360 also utilizes a speech synthesizer 355 (e.g., IBM's Watson text-to-speech, its trademark); the speech synthesizer 355 is used to synthesize enhanced speech from the enhanced text (by artificially generating a corresponding sound that mimics human speech).

[0034] Now for reference Figures 4A to 4B The diagram illustrates a flowchart describing the activities related to the implementation of embodiments of this disclosure. In this respect, each block may correspond to one or more executable instructions for implementing a specified logical function on a general-purpose client.

[0035] Specifically, the diagram illustrates an exemplary process that can be used to enhance the reproduction of voice during a call using method 400. The process begins at black start circle 403 and proceeds to box 406 once the user of the general client enters the call as a participant; for example, the user authenticates with the call manager (running on a server) using the call agent and then submits a corresponding request to it, or he / she uses the call agent to accept a corresponding request received from the call manager. In response, at box 409, the call agent begins to control the call, thereby allowing participants to communicate with other participants by speaking / listening and, possibly, sharing multimedia content (e.g., video, images, data, documents, etc.). For this purpose, the call agent can continuously send (input) call signals representing information provided by participants to the call manager (via a network driver) and can receive (output) call signals representing information provided by other participants to the call (e.g., both in MP4 format) from the call manager (via a network driver).

[0036] Call signals can be received in streaming, i.e., continuously during a call, to reproduce the corresponding information substantially in real time (and in any case, before the call signal for the entire call is received). As long as relevant to this disclosure, the call agent can instruct the microphone to record the input sound at the client's location into a corresponding input audio signal added to the input call signal (sent to the call manager); simultaneously, as described in detail below, the call agent can control the reproduction of the output audio signal included in the output call signal (received from the call manager), which has been generated by the call manager by mixing input audio signals received from the clients of other participants into the call.

[0037] For this purpose, if necessary, the call agent at block 412 can isolate the output audio signal (e.g., in MP3 format) from the output call signal. At block 415, the speech recognizer can continuously analyze the (output) audio signal to search for representations of the (output) speech of other participants in the call (e.g., by applying spectral analysis techniques). Once the speech recognizer detects speech, at block 418, the quality enhancer can determine sound segments of the audio signal, representing pauses in the speech (if any); the sound segment may begin after the end of a previous sound segment (initialized to the start of the call) and may end when speech is detected (no sound segment is determined when speech is detected immediately). At block 421, the quality enhancer can add the sound segment to the playback buffer for the next playback by the speaker driver.

[0038] In box 424, the speech recognizer can continuously analyze the audio signal using a general vocabulary to search for representations of words in the corresponding speech. Specifically, the speech recognizer can determine as quickly as possible the (longest) word whose pronunciation indicated in the general vocabulary best matches the speech in the audio signal, for example, with a degree of accuracy above a minimum (correctness) threshold (e.g., 20-40%). In response, the speech recognizer is able to add the recognized word to a text file. At box 427, the quality enhancer can calculate a quality indicator for the word; for example, the quality indicator can be based on the correctness of the word determined by the speech recognizer, an initial value of the correctness determined by the speech recognizer at the first iteration and the number of further iterations required to achieve the correctness threshold, a trend in the correctness of words already recognized before the call (e.g., the speech recognizer logging into the text file), and so on (e.g., normalized to a weighted sum from 0 to 1). At box 430, the quality enhancer can identify another sound segment of the audio signal representing a word in the speech; as described above, the sound segment can begin after the previous sound segment (initialized as the start of the call) ends and can end when the word is recognized.

[0039] The activity's flow branches at box 433 depending on whether the context of the call has been determined (as indicated in the corresponding register). If not, at box 436, the quality enhancer can add a sound segment to the playback buffer for the next playback by the speaker driver (independent of its quality indicator). Conversely, if the context of the call has been determined, the activity's flow branches at box 439 based on the word's quality indicator. If the word's quality indicator (potentially strictly) is above a threshold level considered acceptable (e.g., 0.5–0.7), at box 442, the quality enhancer can add a sound segment to the playback buffer for the next playback by the speaker driver.

[0040] Conversely, if the quality indicator of a word is (potentially strictly) below a threshold level, the quality enhancer can attempt to enhance the recognition of the word in the sound segment. For this purpose, the quality enhancer can select, at box 445, a context vocabulary (retrieved from the corresponding register) corresponding to the context of the call. At box 448, the speech recognizer can again analyze the sound segment by further using the (selected) context vocabulary to search for representations of words in the corresponding speech. Specifically, as described above, the speech analyzer can determine the (enhanced) word whose pronunciation, indicated in the general / context vocabulary, best matches the sound segment with a higher degree of accuracy than the previously recognized (original) word; for this purpose, the speech recognizer can also apply further enhancement techniques to determine the degree of accuracy, for example, by evaluating the number of pronunciation matches between the sound segment and the pronunciations of words in the general / context vocabulary. In response, the speech recognizer can replace the original word with the enhanced word from the text file (if necessary).

[0041] At box 451, the quality enhancer can calculate the quality indicator of the enhanced word as described above. At box 454, the speech synthesizer can synthesize enhanced speech corresponding to the enhanced word represented by the corresponding (enhanced) audio signal (e.g., mimicking the speech of the corresponding participant learned during a call). At box 457, the quality enhancer can extract a representation of the background from the sound segment, as defined by any sound different from the corresponding speech (e.g., by applying spectral analysis techniques). At box 460, the quality enhancer can calculate the degree of blending between the enhanced word and the background. The degree of blending can depend on the quality indicator of the enhanced word; a higher quality indicator results in a lower degree of blending (e.g., setting the complement of the quality indicator to 1).

[0042] Furthermore, the blending degree can also depend on the trend of the quality indicators of words already identified in the loop during the search for enhanced words, the trend of the quality indicators of (original / enhanced) words already identified before the call (e.g., logged into a text file by a speech recognizer), the trend of the blending degree already calculated before the call (e.g., also logged into a text file by a speech recognizer), and so on (such as by applying appropriate correction factors). In box 463, the quality enhancer can generate corresponding enhanced sound segments by blending the enhanced words with the background according to the blending degree (the higher the blending degree, the higher the weight of the enhanced speech).

[0043] At box 466, the quality enhancer can add an enhanced audio segment to the reproduction buffer for the next reproduction by the speaker driver (rather than the corresponding audio segment). In this way, the popularity of the enhanced speech in the enhanced audio segment (as defined by the degree of mixing) can correspond to the clarity of the enhanced words in the enhanced audio segment (as defined by its quality indicator); this can smooth the transition between any pair of adjacent audio segments and the enhanced audio segment (thus making the reproduction of the enhanced sound smoother).

[0044] In any case, at box 469, the activity stream can again be merged from boxes 436, 442, or 466. At this point, the cognitive engine can extract possible metadata from the already recognized text, which is used to incrementally determine the context of the speech (where the context register is updated accordingly).

[0045] The call agent can verify in box 472 whether the participant is still engaged in the call. If so, the process can return to box 412 to repeat the operation. Conversely, once the participant's engagement in the call has been terminated (e.g., in response to a manual command submitted by the participant to the call agent to leave the call or in response to a command received from the conference manager to close the call), the process can end at the concentric white / black stop circle 475.

[0046] Therefore, to meet local and specific requirements, those skilled in the art can apply numerous logical and / or physical modifications and alterations to this disclosure. More specifically, while this disclosure has been described with a degree of particularity with reference to one or more embodiments thereof, it should be understood that various omissions, substitutions, and changes in form and detail, as well as other embodiments, are possible. Specifically, different embodiments of this disclosure may be practiced even without the specific details (such as numerical values) set forth in the foregoing description to provide a more thorough understanding; conversely, well-known features may have been omitted or simplified to avoid unnecessary detail that obscures the description. Furthermore, it is expressly intended that specific elements and / or method steps described in connection with any embodiment of this disclosure may be incorporated in any other embodiment as a matter of general design choice. Moreover, items presented in the same group and in different embodiments, examples, or substitutions should not be construed as being factually equal to each other (but they are separate and autonomous entities). In any case, each numerical value should be understood to be modified according to applicable tolerances; in particular, unless otherwise indicated, the terms “substantially,” “about,” “approximately,” etc., should be understood to be within 10%, preferably 5%, and even more preferably 1%. Furthermore, each range of numerical values ​​should be intended to explicitly specify any possible number along a continuum within that range (including its endpoints). Ordinal numbers or other qualifiers serve only as labels to distinguish elements with the same name, but do not in themselves imply any priority, precedence, or order. Terms including, comprising, having, containing, relating, etc., should be intended to have an open, non-exhaustive meaning (i.e., not limited to the stated items), terms based on, depending on, according to, function, etc., should be intended to be non-exclusive (i.e., with other possible variables involved), the term a / a should be intended to be one or more items (unless otherwise expressly indicated), and the means to which the term is used (or any means plus function format) should be intended to be any structure adapted or configured to perform the relevant function.

[0047] For example, an embodiment provides a method for enhancing speech reproduction on a computing system. However, this method can be used in any context (e.g., during a video / audio call, the delivery of live / pre-recorded content, etc.).

[0048] In this embodiment, the method includes the following steps under the control of a computing system. However, the computing system can be of any type (see below).

[0049] In an embodiment, the method includes (by a computing system) receiving an audio signal representing sound including speech. However, the audio signal can be any format readable by the computing system (e.g., MP3, WMA, AC-3, etc.) and can be received in any manner (e.g., streaming, downloading, etc.); the audio signal can represent any sound including any speech (e.g., chat, lecture, etc.) (e.g., isolated speech, speech with music / noise in the background, etc.).

[0050] In one embodiment, the method includes recognizing text from an audio signal (by a computing system). However, the text can be recognized in any way (e.g., based on a hidden Markov model, machine learning techniques, neural networks, etc.).

[0051] In an embodiment, the text comprises a sequence of text segments identified from a corresponding sequence of sound segments. However, the text segments and sound segments can be of any type (e.g., representing words, morphemes, phrases, clauses, sentences, etc.).

[0052] In an embodiment, the method includes (by a computing system) determining a corresponding quality indicator for the identified text fragment. However, the quality indicators can be of any type (e.g., continuous / discrete, having any value, etc.) and they can be determined in any way (e.g., based on partial, different, and additional features related to those features mentioned above, individually or in any combination thereof).

[0053] In one embodiment, the method includes (by a computing system) generating a corresponding enhanced text segment from any text segment whose quality indicator has not reached a threshold level. However, this operation can be triggered in any way (e.g., once the quality indicator drops below any value, after this has happened for two or more consecutive text segments, etc.).

[0054] In one embodiment, an enhanced text segment is generated based on additional information about the corresponding audio segment. However, the enhanced text segment can be generated in any manner (e.g., based on the context of the speech, the current sentence, etc.).

[0055] In an embodiment, the method includes synthesizing (by a computing system) a corresponding enhanced speech segment from the enhanced text segment. However, the enhanced speech segment can be synthesized in any manner (e.g., based on techniques such as concatenation, unit selection, diphones, structural components, articulation, deep learning, etc.).

[0056] In an embodiment, the method includes (by a computing system) reproducing enhanced sound, which includes corresponding sound segments and corresponding enhanced sound segments of text segments whose quality indicators respectively reach and fail to reach threshold levels. However, the enhanced sound can be generated in any way (e.g., by simply cascading sound segments and enhanced sound segments, by smoothing the cascading between each pair of sound segments and enhanced sound segments, etc.), and it can be reproduced in any way (e.g., via a reproduction buffer, directly, etc.).

[0057] In this embodiment, the enhanced audio segment is based on a corresponding enhanced speech segment. However, the enhanced audio segment can be based on the corresponding enhanced speech segment in any way (e.g., by mixing the speech segment with a corresponding background segment, by setting them directly to the speech segment, etc.).

[0058] Further embodiments provide additional advantageous features; however, these additional advantageous features may be omitted entirely in the basic implementation.

[0059] Specifically, in an embodiment, the method includes (by a computing system) generating enhanced text fragments based on the context of the speech. However, the context can be determined in any way (e.g., in a predetermined context or freely, based on text, the title of the speech, manually, etc.), and it can be used in any way (e.g., by using appropriate contextual vocabulary, specific inference rules, etc.) to generate the enhanced text fragments.

[0060] In one embodiment, the method includes determining the context of the speech (by a computing system) based on the text. However, the context can be determined based on the text in any way (e.g., using cognitive, machine learning, artificial intelligence, etc.).

[0061] In one embodiment, the method includes (by a computing system) selecting one or more contextual words based on the context of the speech. However, the contextual words can be any number (e.g., predefined, determined at runtime, etc.) and of any type (e.g., locally stored, remotely accessed, etc.).

[0062] In one embodiment, the method includes generating enhanced text fragments (via a computing system) based on contextual vocabulary. However, contextual vocabulary can be used in any way to generate enhanced text fragments (e.g., independently, in addition to general vocabulary).

[0063] In this embodiment, sound segments and text segments represent corresponding words or pauses in speech. However, text segments can represent any entity (e.g., a word or pause, a word only, or more generally any other speech unit).

[0064] In one embodiment, the method includes (by a computing system) isolating a corresponding background segment from the sound segment. However, the background segment can be isolated in any way (e.g., using techniques such as spectral analysis or deep learning).

[0065] In this embodiment, the background segment represents a background different from the speech. However, the background can be of any type (e.g., noise, music, interference, etc.).

[0066] In one embodiment, the method includes generating an enhanced audio segment (by a computing system) by mixing corresponding enhanced speech segments and background segments. However, the enhanced speech segments and background segments can be mixed in any way (e.g., according to a variable degree of mixing, in a fixed manner, etc.).

[0067] In one embodiment, the method includes (by a computing system) determining an additional quality indicator corresponding to the enhanced text segment. However, the additional quality indicator can be determined in any manner (whether the quality indicator is the same or different from the original quality indicator).

[0068] In an embodiment, the method includes (by a computing system) setting the corresponding blending degree of the enhanced speech segment and the background segment according to an additional quality indicator of the corresponding enhanced text segment. However, the blending degree can be set in any way based on further quality indicators (e.g., based on partial, different, and additional features mentioned above, individually or in any combination thereof); or, the blending degree can be set in a similar way based on the quality indicator of the corresponding (original) word.

[0069] In one embodiment, the method includes generating an enhanced audio segment (by a computing system) by mixing a corresponding enhanced audio segment and a background segment according to a corresponding degree of mixing. However, the enhanced audio segment and the background segment can be mixed in any way according to the corresponding degree of mixing (e.g., modulating their volume level, frequency content, dynamics, etc.).

[0070] In an embodiment, the method includes (by a computing system) further setting each of the mixing degrees according to a trend in the mixing degree. However, the mixing degree can be set in any way according to this trend (e.g., by applying a corresponding correction factor, by using it in a weighted sum, etc.).

[0071] In an embodiment, the method includes (by a computing system) determining each quality indicator based on the current quality, quality trend, and / or number of iterations of the identified corresponding text segment. However, each quality indicator can be determined individually or in any combination thereof, based on partial, distinct, and additional features.

[0072] In one embodiment, the method includes (via a computing system) receiving a multimedia signal representing multimedia content containing sound. However, the multimedia content can be of any type (e.g., including video, images, data, documents, etc.).

[0073] In one embodiment, the method includes isolating audio signals from multimedia signals (by a computing system). However, audio signals can be isolated in any way, or this operation can be omitted entirely (e.g., providing audio signals separately in an audio call / conference when they are already provided separately, etc.).

[0074] In one embodiment, the method includes receiving an audio signal in a streaming manner over a telecommunications network (via a computing system). However, any type of streaming (e.g., multicast, unicast, pair, etc.) and over any telecommunications network (e.g., a global, wide area, or local area network based on wired, cellular, or satellite connections, etc.) can be used to receive the audio signal.

[0075] In one embodiment, the method includes receiving an audio signal (by a computing system) during a call on a telecommunications network. However, the call can be of any type (e.g., an audio / video call, a conference call, a web conference, etc.).

[0076] Generally, similar considerations apply if the same solution is achieved using equivalent methods (by using similar steps with more steps or parts thereof that have the same functionality, removing some unnecessary steps or adding further optional steps); in addition, steps can be performed in different orders, simultaneously or (at least partially) in an interleaved manner.

[0077] One embodiment provides a computer program configured to cause a computing system to perform the methods described above. The embodiment provides a computer program product comprising one or more computer-readable storage media having program instructions commonly stored thereon; these program instructions can be read by a computing system to cause the computing system to perform the same methods. However, the computer program can be implemented as a standalone module, as a plug-in to a pre-existing software application (e.g., a call agent), or directly therein. Furthermore, the computer program can execute on any computing system (see below). In any case, the solutions according to embodiments of this disclosure are suitable for implementation even with hardware structures (e.g., electronic circuitry integrated into one or more chips of semiconductor material) or with a combination of suitably programmed or otherwise configured software and hardware.

[0078] One embodiment provides a system including means configured to perform the steps of the methods described above. The embodiment provides a system including circuitry (i.e., any hardware appropriately configured by software) for performing each step of the methods described above. However, the system can be of any type (e.g., personal computer, smartphone, tablet computer, virtual machine, etc.) having any number and type of devices for reproducing enhanced sound (e.g., one or more speakers, headphones, earphones, etc.).

[0079] Generally, similar considerations apply if the system has a different structure or includes equivalent components, or if the system has other operational characteristics. In any case, each component can be separated into multiple elements, or two or more components can be combined together to form a single element; furthermore, each component can be replicated to support the parallel execution of corresponding operations. Moreover, unless otherwise specified, any interaction between different components generally does not need to be sequential, and it can be direct or indirect through one or more intermediaries.

[0080] This invention can be a system, method, and / or computer program product of any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (one or more media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention. The computer-readable storage medium may be a tangible means capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital universal disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination thereof. As used herein, computer-readable storage medium should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires. The computer-readable program instructions described herein can be downloaded from the computer-readable storage medium to the appropriate computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the appropriate computing / processing device. Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server.In the latter case, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) can be personalized to execute computer-readable program instructions by utilizing state information of computer-readable program instructions in order to perform aspects of the present invention. The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing apparatus to generate a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing device, and / or other device to operate in a particular manner. Thus, the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram. Computer-readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device to cause a series of operational steps to be performed on the computer, other programmable device, or other device to produce a computer-implemented process, such that the instructions that execute on the computer, other programmable device, or other device implement the functions / actions specified in one or more blocks of a flowchart and / or block diagram. The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or in a manner that overlaps partially or completely in time, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

Claims

1. A method for enhancing speech reproduction on a computing system, the method comprising: The computing system receives audio signals representing sounds, including speech. The computing system identifies text from the audio signal, the text comprising a sequence of text segments identified from a corresponding sound segment sequence of the sound; The computing system determines the quality indicator corresponding to the text segment sequence; The computing system generates enhanced text segments from any text segments in which the quality indicator does not reach a threshold level, the enhanced text segments being generated based on additional information about the corresponding audio segment sequence; The computing system synthesizes enhanced speech segments from the enhanced text segments; as well as The computing system reproduces the enhanced sound, which includes a portion of the corresponding sound segment sequence in which the quality indicator reaches a threshold level and an enhanced sound segment, which is obtained based on the enhanced speech segment.

2. The method according to claim 1, further comprising: The enhanced text fragment is generated by the computing system based on the context of the speech.

3. The method according to claim 2, further comprising: The computing system determines the context of the speech based on the text.

4. The method according to claim 2, further comprising: The computing system selects one or more contextual words based on the context of the speech; as well as The enhanced text fragment is generated by the computing system based on the one or more context words.

5. The method according to claim 1, wherein, The sound segments in the corresponding sound segment sequence, the enhanced sound segments, the text segments, and the enhanced text segments represent the corresponding words or pauses of the speech.

6. The method according to claim 1, further comprising: The computing system isolates the background segments of the sound from the corresponding sound segment sequences, wherein the background segments represent a background different from the speech. as well as The enhanced sound segment is generated by the computing system by mixing the enhanced speech segment and the background segment.

7. The method according to claim 6, further comprising: The computing system determines other corresponding quality indicators for the enhanced text fragment; The computing system sets the corresponding blending degree of the enhanced speech segment and the background segment according to the other quality indicators of the enhanced text segment; as well as The enhanced sound segment is generated by the computing system by mixing the enhanced speech segment and the background segment according to the corresponding mixing degree.

8. The method according to claim 7, further comprising: The computing system further sets the corresponding mixing degree based on the trend of the corresponding mixing degree.

9. The method according to claim 1, further comprising: The computing system determines the quality indicator based on at least one of the current quality, quality trend, or number of iterations of the text identified from the audio signal.

10. The method according to claim 1, further comprising: The computing system receives a multimedia signal representing multimedia content containing the sound. as well as The computing system isolates the audio signal from the multimedia signal.

11. The method according to claim 1, further comprising: The audio signal is received by the computing system via a telecommunications network in a streaming manner.

12. The method of claim 11, further comprising: During a call via the telecommunications network, the audio signal is received by the computing system.

13. A computer program product for enhancing speech reproduction, the computer program product comprising program instructions readable by a computing system to cause the computing system to perform a method comprising: Receive audio signals representing sounds, including speech; Text is identified from the audio signal, the text comprising a sequence of text segments identified from a corresponding sequence of sound segments of the sound; Determine the quality indicator corresponding to the text segment sequence; Enhanced text segments are generated from any text segments in which the quality indicator does not reach a threshold level, the enhanced text segments being generated based on additional information about the corresponding audio segment sequence; Synthesize enhanced speech segments from the enhanced text segments; as well as The enhanced sound is reproduced, the enhanced sound comprising a portion of the corresponding sound segment sequence in which the quality indicator reaches a threshold level and an enhanced sound segment, the enhanced sound segment being obtained based on the enhanced speech segment.

14. The computer program product according to claim 13, further comprising: The enhanced text fragment is generated by the computing system based on the context of the speech.

15. The computer program product according to claim 14, further comprising: The computing system determines the context of the speech based on the text.

16. The computer program product according to claim 14, further comprising: The computing system selects one or more contextual words based on the context of the speech; as well as The enhanced text fragment is generated by the computing system based on the one or more context words.

17. The computer program product according to claim 13, wherein, The sound segments in the corresponding sound segment sequence, the enhanced sound segments, the text segments, and the enhanced text segments represent the corresponding words or pauses of the speech.

18. The computer program product according to claim 13, further comprising: The computing system isolates the background segments of the sound from the corresponding sound segment sequences, wherein the background segments represent a background different from the speech. as well as The enhanced sound segment is generated by the computing system by mixing the enhanced speech segment and the background segment.

19. The computer program product according to claim 18, further comprising: The computing system determines other corresponding quality indicators for the enhanced text fragment; The computing system sets the corresponding blending degree of the enhanced speech segment and the background segment according to the other quality indicators of the enhanced text segment; as well as The enhanced sound segment is generated by the computing system by mixing the enhanced speech segment and the background segment according to the corresponding mixing degree.

20. A system for enhancing speech reproduction, wherein, The system includes: Circuitry for receiving audio signals representing sounds, including speech; Circuitry for recognizing text from the audio signal, the text comprising a sequence of text segments recognized from a corresponding sequence of sound segments of the sound; Circuitry for determining the quality indicator corresponding to the text segment sequence; Circuitry for generating enhanced text segments from any text segment in which the quality indicator does not reach a threshold level, the enhanced text segments being generated based on additional information about the corresponding audio segment sequence; Circuitry for synthesizing enhanced speech segments from the enhanced text segments; and Circuitry for reproducing enhanced sound, the enhanced sound comprising a portion of the corresponding sound segment sequence in which a quality indicator reaches a threshold level and an enhanced sound segment, the enhanced sound segment being obtained based on the enhanced speech segment.

Citation Information

Patent Citations

  • Systems and methods for providing non-lexical cues in synthesized speech

    CN106575502A

  • Method and device for generating synthetic voice, equipment and medium

    CN111862933A