Automatic leveling of speech content

Abstract

Embodiments for automatic leveling of speech content are disclosed. In embodiments, a method includes receiving, using one or more processors, frames of an audio recording comprising speech content and non-speech content; for each frame: determining, using the one or more processors, a speech probability; analyzing, using the one or more processors, a perceptual loudness of the frame; obtaining, using the one or more processors, a target loudness range for the frame; calculating, using the one or more processors, a gain to apply to the frame based on the target loudness range and the perceptual loudness analysis, wherein the gain comprises a frame-by-frame variation and a dynamic gain that is scaled based on the speech probability; and applying the gain to the frame such that a resulting loudness range of speech content in the audio recording conforms within the target loudness range.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 126,889, filed December 17, 2020; U.S. Provisional Application No. 63 / 032,158, filed May 29, 2020; and Spanish Patent Application No. P202000051, filed March 27, 2020, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to audio signal processing. Background Technology

[0004] Audio content is a crucial component of media transmission and consumption, with many content creators producing speech-based media such as podcasts, audiobooks, interviews, and lectures. In audio content production, the recording stage is typically followed by editing and processing. During this stage, the original recordings are polished and processed to meet the quality standards established in the broadcast industry, ensuring a suitable listening experience across various devices by guaranteeing clarity, loudness consistency, and proper timbre balance.

[0005] Editing and processing stages are typically performed manually using analog or digital processing equipment (hardware, software, or a combination of both). Skilled engineers apply gain control, equalization, compression, noise reduction, sibilance reduction, and other similar processing to the voice recordings. This workflow is very time-consuming because sound engineers need to consider the entire duration of the recording for processing. As more and more voice content is generated, it is valuable to provide tools for automating and accelerating the editing and processing stages while ensuring professional-quality naturalness of the voice without processing artifacts. One process commonly applied to voice is leveling, where gain control (manual or automatic) is applied to the audio to ensure a consistent loudness level, keeping soft parts of the voice clearly discernible and preventing loud parts from becoming overly loud.

[0006] Speech content leveling is typically accomplished in two ways: through manual gain adjustment (e.g., controlling the volume controller on a mixing console or plotting an automated gain curve on a digital audio workstation), or using a dynamic compressor with a set threshold level in which gain reduction is automatically applied to audio segments where the level exceeds the threshold. The first method usually yields the best results but is time-consuming. Furthermore, the manual leveling method cannot guarantee that the output audio will match the desired loudness range across all loudness ranges of the input audio. The second method, using one or more dynamic compressors to level speech content, is less efficient than the manual method and can lead to quality degradation when significant gain reduction is required. Summary of the Invention

[0007] An implementation method for automatic leveling of voice content is disclosed.

[0008] In one embodiment, a method includes: using one or more processors to receive frames of an audio recording comprising speech content and non-speech content; for each frame: using the one or more processors to determine speech probabilities; using the one or more processors to analyze the perceived loudness of the frame; using the one or more processors to obtain a target loudness range for the frame; using the one or more processors to calculate a gain to be applied to the frame based on the target loudness range and the perceived loudness analysis, wherein the gain includes a dynamic gain that varies frame-by-frame and is scaled based on the speech probabilities; and using the one or more processors to apply the gain to the frame such that the resulting loudness range of the speech content in the audio recording conforms to the target loudness range.

[0009] Other embodiments disclosed herein relate to systems, apparatuses, and computer-readable media. Details of the disclosed embodiments are set forth in the accompanying drawings and description below. Other features, objects, and advantages will be apparent from this specification, the drawings, and the claims.

[0010] The specific embodiments disclosed herein offer one or more of the following advantages. The loudness of the speech content is automatically leveled by applying time-varying gain, which enhances the softer parts of the speech and attenuates the louder parts. The quality of the result is comparable to that of manual leveling methods, with the advantage that the level of the output speech content conforms to the desired loudness range regardless of the loudness range of the input speech content. Furthermore, when a significant gain reduction is required, there is no noticeable degradation in the output speech content. Attached Figure Description

[0011] In the accompanying drawings, for ease of description, specific arrangements or orders of schematic elements are shown, such as those representing devices, units, instruction blocks, and data elements. However, those skilled in the art will understand that the specific order or arrangement of the schematic elements in the drawings does not imply a requirement for a particular processing sequence or order, or processing separation. Furthermore, the inclusion of schematic elements in the drawings does not mean that such elements are required in all embodiments, or that in some embodiments, the features represented by such elements may not be included in other elements or combined with other elements.

[0012] Furthermore, in drawings that use connecting elements such as solid or dashed lines or arrows to illustrate connections, relationships, or associations between two or more other schematic elements, the absence of any such connecting element does not imply the absence of connections, relationships, or associations. In other words, some connections, relationships, or associations between elements are not shown in the drawings to avoid obscuring this disclosure. Additionally, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, in cases where a connecting element represents communication of signals, data, or instructions, those skilled in the art will understand that such an element represents one or more signal paths that may be required to affect the communication.

[0013] Figure 1 This is a block diagram of a system for automatic leveling of voice content according to an embodiment.

[0014] Figure 2 The graph illustrates an example of instantaneous loudness, overall loudness, and target dynamic range boundary according to an embodiment.

[0015] Figure 3 This is a flowchart of an automatic leveling process for voice content according to an embodiment.

[0016] Figure 4 This is for implementation according to the embodiments. Figures 1 to 3 A block diagram of an example device architecture for the system and methods.

[0017] The same reference numerals used in all the figures indicate the same elements. Detailed Implementation

[0018] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the various embodiments described. It will be apparent to those skilled in the art that different implementations can be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail to avoid unnecessarily obscuring aspects of the embodiments. Several features are described below, each of which can be used independently of each other or in any combination with other features.

[0019] Naming conventions

[0020] As used herein, the term “comprising” and variations thereof should be understood as open-ended terms meaning “including but not limited to”. Unless the context explicitly states otherwise, the term “or” should be understood as “and / or”. The term “based on” should be understood as “at least partially based on”. The terms “one example implementation” and “example implementation” should be understood as “at least one example implementation”. The term “another implementation” should be understood as “at least one other implementation”. The term “determined, determines, or determining” should be understood as obtaining, receiving, calculating, estimating, predicting, or acquiring. Furthermore, in the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0021] System Overview

[0022] This disclosure describes a method for automatically leveling the loudness of speech content by applying time-varying gain, thereby boosting the soft parts of the speech and attenuating the loud parts. In an exemplary embodiment, there are six parts. The first part includes a voice activity detector (VAD) that modifies the gain as a function of the probability of speech presence. The VAD is particularly helpful in adapting the gain to avoid unnecessary boosting of soft and non-speech parts. The second part is an optional signal-to-noise ratio (SNR) estimator for automating some leveling parameters. The third part includes an optional speaker segmentation clustering (SD) for segmenting the content based on speaker identity. The leveling process is then applied independently to the segmentation of each speaker, which is particularly useful for multi-speaker content. The fourth part includes an optional denoising module for attenuating background noise before or after the leveling process. The fifth part is a loudness analysis stage that extracts loudness information at long-term and short-term time scales, including overall loudness, instantaneous loudness, and loudness range.

[0023] In the offline implementation, the entire recording is available and therefore the look-ahead is unlimited, as the algorithm makes local decisions based on the entire recording to ensure that all available information in the recording is used. In this implementation, the extracted information is saved as metadata for subsequent gain processing. In the real-time implementation, long-term information is updated online.

[0024] Part 6 utilizes analytical information from other parts to estimate the gain for each frame of the audio recording to keep the loudness within the target range. The gain is adjusted based on the VAD / SNR output to avoid boosting non-speech content.

[0025] Figure 1 This is a block diagram of an automated speech leveling system 100 according to an embodiment. System 100 includes an optional denoising unit 101, a VAD unit 102, an optional speaker segmentation and clustering unit 103, an optional SNR estimation unit 104, a loudness analyzer 105, a dynamic gain analyzer 106, a static gain analyzer 107, a leveling unit 108, and an optional denoising unit 109. System 100 operates on frames of an audio input file or audio input stream. In some embodiments, each frame of the audio input includes N ms of audio (e.g., 85 ms) with X% overlap (e.g., 50% overlap) with adjacent frames. In other embodiments, any suitable windowing function may be used.

[0026] refer to Figure 1 At the top, in some embodiments, an optional denoising unit 101 takes frames of the audio input file or audio input stream as input and removes background noise from said frames. Where the robustness of the VAD is sensitive to low SNR speech content, as shown, it is preferable to apply the denoising unit 101 to frames preceding the VAD 102. This will also allow the loudness analyzer 105 to perform more relevant loudness analysis. In some embodiments, an optional SNR estimation unit 104 generates an SNR estimate for the entire audio recording. In other embodiments, the optional SNR estimation unit 104 uses information provided by an optional speaker segmentation clustering unit 103 to adapt to each speaker. SNR estimation is used to drive parameters to ensure robust operation of the leveling unit 108 in the presence of noise, the robust operation including any of the following: adjusting the applied leveling amount such that leveling decreases as SNR decreases, thereby avoiding excessive amplitude modulation of background noise; adjusting the sensitivity of the VAD by post-processing the speech probability in a manner that increases the distinction between speech and non-speech when SNR indicates clean speech; and adjusting the target loudness range such that a smaller dynamic range is achieved only when the signal is sufficiently clean.

[0027] The denoised frames are then input to a VAD 102 configured to distinguish between speech and non-speech content. In some embodiments, VAD 102 returns the probability that the frame contains speech. In some embodiments, the probability output by VAD 102 is a real number in the range of 0.0 to 1.0, where 0.0 indicates that the frame does not contain speech and 1.0 indicates that the frame definitely contains speech. In some embodiments, these output values ​​are interpreted as the likelihood that the frame contains speech. In some embodiments, the VAD output values ​​are smoothed across frames using a moving average, a first-order recursive filter, or any other suitable filter. In some embodiments, if the output value is less than or greater than a threshold, the output value is converted to a binary value 0 or 1 (or a Boolean value). In some embodiments, the VAD output values ​​are stored in memory or written to a file for later use by the leveling unit 108 described below. In some embodiments, the VAD output values ​​are calculated in real time during the recording process and stored for use after the leveling unit 108 stage, thereby avoiding additional VADs. In some embodiments, speech probabilities are calculated using deep neural networks (e.g., recurrent neural networks (RNNs), long short-term memory (LSTMs)) trained on various types of speech content.

[0028] In some embodiments, the distinction between speech and non-speech is based on applying a threshold to the broadband energy level of a frame, such that frames with energy below the threshold are considered non-speech frames or background noise. Alternatively, the energy level of a frame can be limited bandwise rather than broadband, for example, considering only the typical frequency range of speech (e.g., 100 Hz to 4 kHz).

[0029] The output of VAD 102 is input to an optional speaker segmentation clustering unit 103. For multi-speaker content, speaker segmentation clustering facilitates the independent application of leveling to each speaker. Speaker segmentation clustering not only controls the loudness dynamics of each speaker but also allows the leveling parameters to be adapted to the different SNRs of each speaker's recording conditions. In some embodiments, speaker segmentation clustering unit 103 outputs a time index of active dominance for each speaker. Segments belonging to a speaker are treated as a whole, and the leveling process is applied independently to each speaker.

[0030] Loudness analyzer 105 receives VAD / SNR (a separate denoised frame stream for each identified speaker and the speech probability for each frame) of speaker adaptive information with optional speaker segmentation clustering unit 103, and analyzes the loudness of the content in the recording over time. The entire recording is analyzed, and instantaneous loudness is calculated and stored in memory or a file. Instantaneous loudness M(n) is a quantity expressed in loudness units at full scale (LUFS) and represents the perceived loudness of a 400ms segment of audio. The variable n refers to the nth frame of the audio. In some embodiments, instantaneous loudness is calculated in real time during the recording process and stored by leveling unit 108 for later use, thereby avoiding an additional loudness analysis phase.

[0031] The overall loudness I is also calculated, which represents the loudness of the entire file, as described in the following document: Algorithms to Measure Audio Programme Loudness and True-peak Audio Level, International Telecommunications Union-Radiocommunication Sector (ITU-R) BS.1770-4 (October 2015). Alternative loudness measures can be used, such as those described in BR.J. Moore, BR.R. Glasberg, and T. Baer, ​​“A Model for the Prediction of Thresholds, Loudness, and Partial Loudness,” J. Audio Eng. Soc., vol. 45, no. 4, pp. 224-240, (1997 April) [BR.J. Moore, BR.R. Glasberg, and T. Baer, ​​A Model for the Prediction of Thresholds, Loudness, and Partial Loudness, Journal of the Audio Engineering Society, Vol. 45, No. 4, pp. 224-240, (April 1997)], or Zwicker, E., Fasti, H., Widmann, U., Kurakata, K., Kuwano, S., & Namba, S. (1991). Program for calculating loudness according to DIN 45631 (ISO 532B). Journal of the Acoustical Society of Japan (E), 12(1), 39-42 [Zwicker, E., Fasti, H., Widmann, U., Kurakata, K., Kuwano, S. and Namba, S. (1991), Procedure for calculating loudness according to DIN 45631 (ISO 532B), Journal of the Acoustical Society of Japan (E), 12(1), 39-42] describes the loudness measure.

[0032] In the paper by Zwicker et al., partial loudness is estimated, which is a loudness estimation in the presence of noise. It is also envisioned that some scene segmentation could be added, and the overall loudness could be performed scene by scene at a time. In some embodiments, the loudness metric emphasizes the energy in a specific frequency band, such as a band containing speech frequencies, and can be as simple as measuring the energy in this band.

[0033] Dynamic gain analyzer 106 and static gain analyzer 107 calculate dynamic and static gains over time, respectively, and apply them to the content so that the resulting loudness range of the speech content in the audio recording conforms to the desired target loudness range. More specifically, static gain analyzer 107 determines the static gain G. s The static gain is constant across frames and is used to adjust the overall loudness to a target loudness T (e.g., -23 LUFS). The dynamic gain analyzer 106 determines the dynamic gain G. d The dynamic gain can be varied frame by frame and is used to adjust the loudness range to the desired target loudness range ΔL.

[0034] In some embodiments, the loudness range is expressed in decibels and is typically between 3 dB and 6 dB. The instantaneous loudness M(n) is segmented according to its local minimum and maximum values, wherein each segment includes all frames between a local maximum and the next local minimum, or between a local minimum and the next local maximum. Then, based on the value M(n) k For each fragment n k Define the target dynamic gain anchor point G in dB. d (n k As shown in equation [1]:

[0035] C d (n k )=I+ΔL / 2-M(n k If M(n) k )>I+ΔL / 2 [1]

[0036] G d (n k )=(I-ΔL / 2-M(n k ))·c(n k If M(n) k )<I-ΔL / 2

[0037] G d (n k ) = 0 otherwise

[0038] The factor c(n) in the above equation [1] k () depends on the frame n of VAD 102. k The weight of the VAD value output at c(n). k The purpose of c(n) is to avoid boosting non-speech components. In some embodiments, c(n) k ) is frame n k The probability of speech at a given location. In other embodiments, c(n) k) is a non-linear function of speech probability, such as the square root of the speech probability or an sigmoid function of the speech probability, where the former corresponds to giving higher weight to frames whose VAD output is closer to 1, and the latter corresponds to a soft threshold for the speech probability. In some embodiments, the optional SNR estimate 104 can be used to control the determination of c(n) k The parameters of a nonlinear function of the speech probability of ). For example, when using a sigmoid function to calculate c(n) k When the value of ) is compressed towards values ​​of 0 and 1, the SNR can control the amount of compression. When the signal is noisy, the speech probability determined by VAD 102 is used, while when the speech is clean, VAD detection becomes more discriminative, allowing the speech portion to be fully leveled.

[0039] In some embodiments, the dynamic gain is calculated as a continuous function of the distance between the loudness of each frame and the overall loudness. For example, the dynamic gain can be calculated as a continuous function mapping M(n) to the range [I-ΔL / 2, I+ΔL / 2]. An example is the sigmoid function given by equation [2]:

[0040]

[0041] Among them, the gain G d (n k ) is applied to n k With n k+1 The audio frames between.

[0042] In some embodiments, the vector G of the gain anchor point d (n) Smoothly adapts over time to avoid sudden changes (discontinuities) between adjacent sample frames. When a new gain anchor is received, the smooth gain is calculated over a defined time period by linear interpolation between the previous gain anchor and the new gain anchor, as shown in equation [3]:

[0043]

[0044] Where τ is the time constant, representing the interpolation gain value. To achieve the new target gain value G d How many seconds are needed. A typical value for τ is 0.1 seconds when the target gain is greater than the current gain (onset or boost condition), and 0.25 seconds when the target gain is less than the current gain (e.g., release or decay condition).

[0045] In some embodiments, linear interpolation is calculated for linear gain values ​​instead of dB values. In some embodiments, linear interpolation is replaced by a first-order recursive filter with an exponential time constant. If a target overall loudness is given, an additional gain factor, i.e., the static gain G, can be applied. s=TI. The leveling amount can be scaled by multiplying the dynamic gain by a factor between 0.0 and 1.0, where 1.0 corresponds to full effect, 0.0 corresponds to no leveling, and the intermediate value corresponds to an intermediate leveling amount. This amount can be controlled by the SNR estimate, such that the scaling factor decreases linearly as the SNR estimate decreases, to reduce potential artifacts caused by over-leveling of the noise content.

[0046] In some embodiments, the gain is applied broadband to the signal. In other embodiments, the gain is applied to a specific frequency band, such as a band containing speech frequencies. The leveling unit 108 applies a smooth gain to the audio by multiplying the sample value of each frame by the corresponding linear gain value. In some embodiments, limits are applied to the maximum gain boost and the maximum gain attenuation. Typical values ​​for the maximum gain are between 10 dB and 20 dB for both boost and attenuation. In cases where residual noise is significantly amplified due to the leveling process, an optional SNR denoising unit 109 can be added subsequently. The SNR estimate output by the optional SNR estimation unit 104 can also be used to determine the dry / wet mixing ratio between the input audio and the processed audio.

[0047] Figure 2 This is a graph illustrating an example of instantaneous loudness, overall loudness gain, and target loudness range according to an embodiment. Note that the target loudness range is the region between I+ΔL / 2 and I-ΔL / 2, where the overall loudness is at the center of the loudness range. In this embodiment, the target loudness range (e.g., 6 dB) is an input parameter set by the user. In embodiments where audio is streamed, the target loudness range may be set by the broadcaster and included in the metadata of the audio stream.

[0048] Example process

[0049] Figure 3 This is a flowchart of an automatic leveling process 300 for speech content according to an embodiment. Process 300 can be used... Figure 4 The device architecture shown is used for implementation.

[0050] Process 300 begins by dividing the audio recording, which contains both speech and non-speech content, into frames (301). For each frame, process 300 distinguishes between speech and non-speech content (302). For example, the VAD calculates the speech probability indicating whether a frame contains speech content, and the SNR estimator provides both a global estimate and a local time-varying estimate.

[0051] Then, process 300 analyzes the perceived loudness of the frame (303) and obtains the target loudness range of the frame (304).

[0052] Then, process 300 determines the gain (305) based on the target loudness range, perceived loudness analysis, and whether the frame includes speech or non-speech, and applies the gain to the speech content in the frame (306) so that the resulting loudness range of the speech content in the audio recording conforms to the target loudness range. For example, static gain and dynamic gain can be calculated. The speech probability and SNR information output by the VAD can be used to scale the dynamic gain to avoid boosting non-speech content, as referenced. Figure 1 As stated above.

[0053] In some embodiments, the gain includes a static gain applied to all frames and a dynamic gain that varies frame by frame, and the dynamic gain is calculated as a continuous function of the distance between the loudness of each frame and the overall loudness.

[0054] In some embodiments, the dynamic gain of frames within the desired loudness range relative to the overall loudness is uniform, and the dynamic gain applied to frames outside the desired loudness range is calculated as the difference between the loudness value of the frame and the nearest boundary of the desired loudness range.

[0055] In some embodiments, static gain is the difference between overall loudness and target loudness.

[0056] In some embodiments, the gain applied to a frame is the sum of the static gain and the dynamic gain.

[0057] In some embodiments, the dynamic gain is multiplied by a coefficient between 0.0 and 1.0.

[0058] In some embodiments, the coefficient is a function of SNR.

[0059] In some embodiments, the probability of speech is calculated for each frame, and the dynamic gain is scaled as a function of the speech probability.

[0060] In some embodiments, the speech probability is calculated by a neural network.

[0061] In some embodiments, the speech probability is a function of the broadband energy level for each frame.

[0062] In some embodiments, the speech probability is a function of the energy level of each frame in a specific frequency band.

[0063] In some embodiments, the speech probability can be modified using a sigmoid function, and the parameters of the sigmoid function can be manually fixed or automatically adapted based on the estimated SNR.

[0064] In some embodiments, the gain is smoothed over time by linear interpolation over a predefined duration.

[0065] In some embodiments, the gain is smoothed over time by adding a portion of the current value to a portion of the previous value.

[0066] In some embodiments, the loudness of each frame is calculated and stored during recording.

[0067] In some embodiments, loudness information is read from metadata that has been pre-calculated based on the audio.

[0068] In some embodiments, the probability of speech is calculated and stored at the recording time.

[0069] In some embodiments, loudness is instantaneous loudness.

[0070] In some embodiments, the energy in the frequency band is used as a horizontal measure.

[0071] In some embodiments, the gain is applied to a specific frequency band of the signal.

[0072] In some embodiments, the boost or attenuation provided by the gain is limited to a predefined maximum value.

[0073] In some embodiments, the speech content may contain multiple speakers with different SNRs, and speaker segmentation clustering (e.g., speaker segmentation clustering 103) is used to segment the content according to speaker identity before SNR estimation, wherein segments belonging to each speaker are processed separately from segments belonging to other speakers.

[0074] In some embodiments, the audio recording is preprocessed using the techniques described above, and the resulting leveling gain is included in the metadata or speech content frames in the bitstream being streamed to one or more decoding devices, wherein the decoder extracts the gain from the metadata and applies the gain to the speech content frames.

[0075] Example System Architecture

[0076] Figure 4 A block diagram of an example system 400 suitable for implementing exemplary embodiments of the present disclosure is shown. System 400 includes any device capable of playing audio, including but not limited to: smartphones, tablet computers, wearable computers, in-vehicle computers, game consoles, surround sound systems, and kiosks.

[0077] As shown, system 400 includes a central processing unit (CPU) 401 capable of executing various processes based on programs stored, for example, in read-only memory (ROM) 402 or loaded from, for example, storage unit 408 into random access memory (RAM) 403. RAM 403 also stores data required by the CPU 401 for executing various processes as needed. CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0078] The following components are connected to I / O interface 405: input unit 406, which may include a keyboard, mouse, etc.; output unit 407, which may include a display such as a liquid crystal display (LCD) and one or more speakers; storage unit 408, which includes a hard disk or another suitable storage device; and communication unit 409, which includes a network interface card such as a network card (e.g., wired or wireless).

[0079] In some implementations, the input unit 406 includes one or more microphones located at different locations (depending on the host device), which enable the capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0080] In some embodiments, the output unit 407 includes a system having a variety of numbers of speakers. For example... Figure 4 As illustrated, the output unit 407 (depending on the capabilities of the host device) can render audio signals in various formats, such as mono, stereo, immersive, binaural, and other suitable formats.

[0081] Communication unit 409 is configured to communicate with other devices (e.g., via a network). Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, flash drive, or other suitable removable media, is mounted on drive 410 so that computer programs read from it are installed into storage unit 408. Those skilled in the art will understand that although system 400 is described as including the components described above, in practice, some of these components may be added, removed, and / or replaced, and all such modifications or changes fall within the scope of this disclosure.

[0082] According to exemplary embodiments of this disclosure, the processes described above can be implemented as computer software programs or on computer-readable storage media. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing a method. In such embodiments, as... Figure 4 As shown, the computer program can be downloaded and installed from the network via the communication unit 809, and / or installed from the removable medium 411.

[0083] Typically, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry (e.g., control circuitry), software, logic, or any combination thereof. For example, the units discussed above can be implemented by control circuitry (e.g., with...) Figure 4 The control circuitry executes the actions described herein, which are performed by the CPU in combination with other components. Some aspects may be implemented in hardware, while others may be implemented in firmware or software (e.g., control circuitry) that can be executed by a controller, microprocessor, or other computing device. Although various aspects of the exemplary embodiments of this disclosure are illustrated and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers, other computing devices, or some combination thereof, as non-limiting examples.

[0084] Additionally, the various blocks shown in the flowchart can be viewed as method steps, and / or operations resulting from the operation of computer program code, and / or multiple coupled logic circuit elements configured to perform associated functions(s). For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code configured to perform the methods described above.

[0085] In the context of this disclosure, a machine-readable medium can be any tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be non-transitory and can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More specific examples of machine-readable storage media will include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0086] Computer program code used to perform the methods of this disclosure may be written in any combination of one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus having control circuitry, such that, when executed by the processor of the computer or other programmable data processing apparatus, the program code implements the functions / operations specified in the flowcharts and / or block diagrams. The program code may be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server, or distributed across one or more remote computers and / or servers.

[0087] While this document contains numerous details of specific implementations, these details should not be construed as limiting the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments. Certain features described herein in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially stated so, in some cases one or more features of a claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof. The logical flow depicted in the drawings does not require the specific order or ordered sequence shown to achieve the desired result. Additionally, other steps may be provided from the described flow, or steps may be deleted, and other components may be added to or removed from the described system. Therefore, other implementations are within the scope of the following claims.

Claims

1. A method for audio processing, comprising: One or more processors are used to receive frames of audio recordings, including both speech and non-speech components, wherein the speech comprises speech content from multiple speakers with different signal-to-noise ratios (SNR). Detect the speech frames in the speech content and the corresponding speech probabilities of the speech frames; The audio frames are segmented into separate streams based on the identities of the plurality of speakers, wherein the segmentation is based on a time index to indicate the dominant position of each of the plurality of speakers in the audio recording; For each frame in each of the individual streams: The one or more processors are used to analyze the perceived loudness of the frame; The one or more processors are used to obtain the target loudness range of the frame; The one or more processors are used to calculate a gain to be applied to the frame based on the target loudness range and the perceived loudness analysis, wherein the gain includes a static gain applied to all frames and a dynamic gain that varies frame-by-frame and is scaled based on the speech probability, wherein the static gain is the difference between the overall loudness and the target loudness, and the dynamic gain is calculated as a continuous function of the distance between the perceived loudness and the overall loudness for each frame; and The gain is applied to the frame using one or more processors such that the resulting loudness range of the speech content in the audio recording conforms to the target loudness range.

2. The method as described in claim 1, wherein, The gain applied to each frame is the sum of the static gain and the dynamic gain of that frame.

3. The method as described in claim 1, wherein, The dynamic gain of frames within the desired loudness range is uniform relative to the overall loudness, and the dynamic gain applied to frames outside the desired loudness range is calculated as the difference between the loudness value of the frame and the nearest boundary of the desired loudness range.

4. The method of claim 1, wherein, The dynamic gain is multiplied by a coefficient between 0.0 and 1.

0.

5. The method of claim 1, wherein, The speech probability is calculated using a neural network.

6. The method of claim 1, wherein, The speech probability is a function of the broadband energy level for each frame.

7. The method of claim 1, further comprising: Estimate the signal-to-noise ratio (SNR); as well as The speech probability is modified at least in part based on the estimated SNR.

8. The method of claim 7, wherein, The speech probability is determined by a voice activity detector (VAD), and the method further includes: When the estimated SNR indicates that the speech content is clean, the sensitivity of the VAD is adjusted to increase the distinction between speech and non-speech.

9. The method of claim 7, wherein, The dynamic gain is multiplied by a coefficient between 0 and 1, and the coefficient is a function of the SNR.

10. The method of claim 1, wherein, The speech probability can be modified by a sigmoid function, wherein the parameters of the sigmoid function are either manually fixed or automatically adapted based on the estimated SNR of the speech content.

11. The method of claim 1, wherein, The speech probability is a function of the energy level of each frame in a frequency band containing speech frequencies.

12. The method of claim 1, wherein, The gain is smoothed over time by linear interpolation over a predefined duration.

13. The method of claim 1, wherein, The gain is smoothed over time by adding a portion of the current value to a portion of the previous value.

14. The method of claim 1, wherein, The perceived loudness is calculated and stored for each frame during recording.

15. The method of claim 1, wherein, Loudness information is read from metadata that has been pre-calculated based on the audio.

16. The method of claim 1, wherein, The probability of the speech is calculated and stored during the recording time.

17. The method of claim 1, wherein, The loudness mentioned is instantaneous loudness.

18. The method of claim 1, wherein, The energy in the frequency band is used as a horizontal measure.

19. The method of claim 1, wherein, The gain is applied to the frequency band of the signal that includes speech frequencies.

20. The method of claim 1, wherein, The gain boost and attenuation are limited by a predefined maximum value.

21. A system for audio processing, comprising: One or more processors; as well as A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operation of the method as described in any one of claims 1 to 20.

22. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operation of the method as described in any one of claims 1 to 20.

23. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the method as described in any one of claims 1 to 20.

Images