Method and apparatus for speech / music classification and core encoder selection in a sound codec

Abstract

A two-stage speech / music classification device and method classifies an input sound signal and selects a core encoder for encoding the sound signal. A first stage classifies the input sound signal as one of a plurality of final classifications. A second stage extracts high-level features of the input sound signal and, in response to the extracted high-level features and the final classification selected in the first stage, selects a core encoder for encoding the input sound signal.

Description

Technical Field

[0001] This disclosure relates to sound codecs, and more specifically, to speech / music classification and core encoder selection, and especially, but not exclusively, to multichannel sound codecs capable of producing good sound quality at low bit rates and low latency, for example, in complex audio scenarios.

[0002] In this disclosure and the appended claims:

[0003] - The term "sound" can refer to speech, audio, and any other sound;

[0004] - The term "stereo" is an abbreviation of "stereophonic"; and

[0005] The term "mono" is an abbreviation of "monophonic". Background Technology

[0006] Historically, conversational phone calls were made using mobile phones with only one transducer, which output sound to only one ear of the user. Over the past decade, users have begun using their mobile phones with headphones to receive sound through both ears, primarily for music, but sometimes also for voice calls. However, when using a mobile phone to send and receive conversational voice, the content remains mono, but when using headphones, it is presented to both ears of the user.

[0007] With the latest 3GPP voice codec standards, such as EVS (Enhanced Voice Service) described in reference [1] (the entire contents of which are incorporated herein by reference), the quality of codeced sound (e.g., voice and / or audio) sent and received via portable mobile phones has been significantly improved. The next step is naturally to send stereo information so that the receiver is as close as possible to the real-life audio scene captured at the other end of the communication link.

[0008] In audio codecs, stereo information is typically transmitted, as described in reference [2] (the entire contents of which are incorporated herein by reference).

[0009] For conversational speech codecs, mono signals are standard. When transmitting stereo signals, the bit rate is typically doubled because both the left and right channels of the stereo signal are encoded using a mono codec. This works well in most situations, but it has the drawback of doubling the bit rate and failing to utilize any potential redundancy between the two channels (the left and right channels of the stereo signal). Furthermore, to keep the overall bit rate at a reasonable level, each of the left and right channels uses a very low bit rate, thus affecting the overall sound quality. To reduce the bit rate, efficient stereo codec techniques have been developed and used. As non-limiting examples, two stereo codec techniques that can be used effectively at low bit rates are discussed in the following paragraphs.

[0010] The first stereo encoding and decoding technique is called parametric stereo. Parametric stereo uses a common mono codec plus a certain amount of stereo side information (corresponding to stereo parameters) representing the stereo image to encode two inputs (left and right channels) into mono signals. The two inputs are downmixed into mono signals and then the stereo parameters are calculated. This is usually performed in the frequency domain (FD), such as in the discrete Fourier transform (DFT) domain. Stereo parameters are related to so-called binaural or inter-channel cues. Binaural cues (see, for example, reference [3], the entire contents of which are incorporated herein by reference) include binaural level difference (ILD), binaural time difference (ITD), and binaural correlation (IC). Depending on the characteristics of the audio signal, the stereo scene configuration, etc., some or all of the binaural cues are encoded and sent to the decoder. Information about what binaural cues are encoded and sent is sent as signaling information, which is usually part of the stereo side information. Furthermore, different encoding / decoding techniques can be used to quantize a given two-channel cue, resulting in the use of a variable number of bits. Then, in addition to the quantized two-channel cue, stereo side information can typically contain the quantized residual signal produced by downmixing at medium to high bit rates. The residual signal can be encoded using entropy coding techniques such as arithmetic encoders.

[0011] Another stereo coding technique operates in the time domain. This stereo coding technique mixes two inputs (left and right channels) into so-called master and sub-channels. For example, as described in reference [4] (the entire contents of which are incorporated herein by reference), time-domain mixing can be based on a mixing ratio that determines the respective contribution of the two inputs (left and right channels) in producing the master and sub-channels. The mixing ratio is derived from several metrics, such as the normalized correlation of the two inputs (left and right channels) relative to a mono signal, or the long-term correlation difference between the two inputs (left and right channels). The master channel can be encoded and decoded by a common mono codec, while the sub-channel can be encoded and decoded by a lower bit-rate codec. The encoding and decoding of the sub-channel can take advantage of the consistency between the master and sub-channels and can reuse some parameters of the master channel.

[0012] Furthermore, in recent years, the generation, recording, representation, encoding / decoding, transmission, and reproduction of audio have been evolving towards enhanced, interactive, and immersive listener experiences. An immersive experience can be described, for example, as a state of deep engagement or participation in a sound scene when sound comes from all directions. In immersive audio (also known as 3D audio), a sound image is reproduced in all three-dimensional space surrounding the listener, taking into account a wide variety of sound characteristics such as timbre, directionality, reverberation, transparency, and (auditory) spatial accuracy. Immersive audio is produced for a specific sound playback or reproduction system (such as a speaker-based system, an integrated reproduction system (soundboard), or headphones). The interactivity of the sound reproduction system can then include, for example, the ability to adjust sound levels, change sound positions, or select different languages ​​for reproduction.

[0013] There are three basic methods to achieve an immersive experience.

[0014] One approach to achieving an immersive experience is a channel-based audio method that uses multiple spaced-out microphones to capture sound from different directions, with each microphone corresponding to an audio channel in a specific speaker layout. Each recorded channel is then provided to a speaker in a given location. Examples of channel-based audio methods include stereo, 5.1 surround, 5.1+4, etc.

[0015] The second approach to achieving immersive experiences is scene-based audio, which represents the desired sound field in a local space as a function of time through a combination of dimensional components. The sound signal representing scene-based audio is independent of the location of the audio source, and the sound field is transformed at the renderer to fit a selected speaker layout. An example of scene-based audio is ambisonics.

[0016] The third approach to achieving immersive experiences is an object-based audio method, which represents the auditory scene as a set of individual audio elements (e.g., singer, drums, guitar, etc.) accompanied by information such as their location, so that they can be reproduced by a sound reproduction system in their intended positions. This provides great flexibility and interactivity to object-based audio methods because each object is discrete and can be manipulated individually.

[0017] Each of the audio methods described above for achieving immersive experiences has its advantages and disadvantages. Therefore, it is common to combine several audio methods rather than just one in complex audio systems to create immersive auditory scenes. Examples could be audio systems that combine scene-based or channel-based audio with object-based audio, such as ambient stereo with several discrete audio objects. Summary of the Invention

[0018] According to a first aspect, this disclosure provides a two-stage speech / music classification device for classifying input sound signals and selecting a core encoder for encoding the sound signals, comprising: a first stage for classifying the input sound signals into one of a plurality of final categories; and a second stage for extracting high-level features of the input sound signals and, in response to the extracted high-level features and the final category selected in the first stage, selecting a core encoder for encoding the input sound signals.

[0019] According to the second aspect, a two-stage speech / music classification method is provided for classifying input sound signals and selecting a core encoder for encoding the sound signals, comprising: in a first stage, classifying the input sound signal into one of a plurality of final categories; and in a second stage, extracting high-level features of the input sound signal and, in response to the extracted high-level features and the final category selected in the first stage, selecting a core encoder for encoding the input sound signal.

[0020] The foregoing and other objects, advantages and features of the sound codec including the two-stage speech / music classification device and method will become more apparent from the following non-limiting description of exemplary embodiments given by way of example only with reference to the accompanying drawings. Attached Figure Description

[0021] In the attached diagram:

[0022] Figure 1 It is a schematic block diagram of a sound processing and communication system, describing the possible implementation environment of a sound codec, including a two-stage speech / music classification device and method;

[0023] Figure 2 This is a schematic block diagram showing the first stage of a two-stage speech / music classification device and the first stage of the corresponding two-stage speech / music classification method.

[0024] Figure 3 This is a schematic block diagram showing the second stage of a two-stage speech / music classification device and the corresponding second stage of a two-stage speech / music classification method.

[0025] Figure 4 This is a schematic block diagram that simultaneously shows the state machine of the first stage of the two-stage speech / music classification device and the signal division operation of the first stage of the two-stage speech / music classification method;

[0026] Figure 5 This is a graph showing a non-limiting example of syllable onset / attack detection based on relative frame energy;

[0027] Figure 6 A histogram representing the selected features in the training database;

[0028] Figure 7 The figure shows a curve illustrating outlier feature detection based on histogram values;

[0029] Figure 8 This is a graph showing the Box-Cox transform curves for different power transform exponent values ​​λ;

[0030] Figure 9 The graphs showing the rising and falling edge detection behavior for calculating the forgetting factor of an adaptive IIR filter are presented as a non-limiting example.

[0031] Figure 10 The graph shows the distribution of the smoothed difference score wdlp(n) of the training database and the threshold for switching between SPEECH / NOISE, UNCLEAR and MUSIC classifications.

[0032] Figure 11 It is a graph showing the ranking of samples in the ENTRY state during the calculation of the weighted average of the difference scores;

[0033] Figure 12 It shows a complete set of rule category transformation diagrams for the conversion between SPEECH / NOISE, UNCLEAR, and MUSIC.

[0034] Figure 13 This is a schematic diagram illustrating segment start detection performed on multiple short segments in the current frame of the input audio signal;

[0035] Figure 14 It is a description of Figure 3A schematic diagram of the initial selection mechanism of the core encoder used by the initial selector of the second stage core encoder in a two-stage voice / music classification device;

[0036] Figure 15 This is a simplified block diagram of an example configuration of hardware components for implementing a sound codec, including a two-stage speech / music classification device and method. Detailed Implementation

[0037] In recent years, 3GPP (3rd Generation Partnership Project) has been working on developing a 3D (three-dimensional) sound codec based on the EVS codec (see reference [5], the entire contents of which are incorporated herein by reference).

[0038] This disclosure describes speech / music classification techniques and core encoder selection techniques within the IVAS codec framework. Both techniques are part of a two-stage speech / music classification approach, the result of which is core encoder selection.

[0039] Although the speech / music classification method and apparatus are based on EVS (see references [6] and [1], section 5.1.13.6, the entire contents of which are incorporated herein by reference), some improvements and developments have been implemented. Furthermore, in this disclosure, a two-stage speech / music classification method and apparatus are described by way of example only, with reference to the IVAS codec framework (or IVAS sound codec) that runs throughout this disclosure. However, incorporating such a two-stage speech / music classification method and apparatus into any other sound codec is also within the scope of this disclosure.

[0040] Figure 1 This is a schematic block diagram of a stereo processing and communication system 100, which describes a possible implementation environment for a sound codec (IVAS codec), including a two-stage speech / music classification device and method.

[0041] Figure 1 The stereo processing and communication system 100 supports the transmission of stereo signals via a communication link 101. The communication link 101 may include, for example, a wire or fiber optic link. Optionally, the communication link 101 may at least partially include a radio frequency (RF) link. RF links typically support simultaneous communication by multiple entities that need to share bandwidth resources (as can be found in cellular phones). Although not shown, the communication link 101 may be replaced by a storage device in a single device implementation of the system 100 that records and stores the encoded and decoded stereo signals for later playback.

[0042] Still referencing Figure 1For example, a pair of microphones 102 and 122 produce the left channel 103 and right channel 123 of the original analog stereo signal. As noted in the preceding description, the sound signal may, in particular but not exclusively, include speech and / or audio.

[0043] The left channel 103 and right channel 123 of the original analog stereo signal are provided to the analog-to-digital (A / D) converter 104 for converting them into the left channel 105 and right channel 125 of the original digital stereo signal. The left channel 105 and right channel 125 of the original digital stereo signal can also be recorded and provided from a storage device (not shown).

[0044] Stereo encoder 106 encodes the left channel 105 and right channel 125 of the original digital stereo signal to produce a set of encoded parameters, which are multiplexed in the form of bitstream 107 and transmitted to an optional error correction encoder 108. The optional error correction encoder 108 (when present) adds redundancy to the binary representation of the encoded parameters in bitstream 107 before transmitting the resulting bitstream 111 via communication link 101.

[0045] On the receiver side, an optional error correction decoder 109 utilizes the redundant information in the received bitstream 111 to detect and correct errors that may occur during transmission on the communication link 101, generating a bitstream 112 with the received encoding parameters. A stereo decoder 110 converts the received encoding parameters in the bitstream 112 to create the synthesized left channel 113 and right channel 133 of the digital stereo signal. The reconstructed left and right channels 113 of the digital stereo signal in the stereo decoder 110 are then converted into the synthesized left and right channels 114 and 134 of the analog stereo signal in a digital-to-analog (D / A) converter 115.

[0046] The synthesized left channel 114 and right channel 134 of the analog stereo signal are played back in a pair of speaker units or two-channel headphones 116 and 136, respectively. Optionally, the left channel 113 and right channel 133 of the digital stereo signal from the stereo decoder 110 can also be provided and recorded in a storage device (not shown).

[0047] For example, Figure 1 The stereo encoder 106 can be implemented by the encoder of the IVAS codec, including Figure 2 and 3 A two-stage voice / music classification device.

[0048] 1. Two-stage speech / music classification

[0049] As noted in the preceding description, this disclosure describes speech / music classification techniques and core encoder selection techniques within the IVAS codec framework. Both techniques are part of a two-stage speech / music classification approach (and corresponding devices), the result of which is the selection of a core encoder for encoding the main (dominant) channel (in the case of time-domain (TD) stereo coding) or the mixed mono channel (in the case of frequency-domain (FD) stereo coding). The foundation of this technique development is speech / music classification in the EVS codec (reference [1]). This disclosure describes modifications and improvements implemented therein and which are part of the baseline IVAS codec framework.

[0050] The first phase of the speech / music classification method and device in the IVAS codec is based on the Gaussian Mixture Model (GMM). The original model used in the EVS codec has been extended, improved, and optimized to be suitable for stereo signal processing.

[0051] Summarize:

[0052] The GMM model takes feature vectors as input and provides probability estimates for three categories, including speech, music, and background noise.

[0053] The parameters of the GMM model are trained on a large number of manually labeled sound signal feature vectors.

[0054] - The GMM model provides a probability estimate for each of the three categories in each frame (e.g., a 20-millisecond frame). Audio signal processing frames that include subframes are well known to those skilled in the art, but further information about such frames can be found, for example, in reference [1].

[0055] - The anomaly detection logic ensures that frames whose one or more features of the audio signal do not meet the normal distribution condition are processed correctly.

[0056] - Through logistic regression, the individual probability is transformed into a single, unrestricted score.

[0057] - Two-stage speech / music classification devices have their own state machine for classifying input signals into one of four states.

[0058] - Adaptive smoothing is applied to the output score, depending on the two-stage speech / music classification method and the current state of the device.

[0059] - The two-stage speech / music classification method and device achieve rapid response in rapidly changing content through syllable initiation / outitiation detection logic based on relative frame energy.

[0060] - The smoothed score is used to select from the following three signal types: pure speech, pure music, and speech with music.

[0061] Figure 2 This is a schematic block diagram showing the first stage 200 of the two-stage speech / music classification device and the first stage 250 of the corresponding two-stage speech / music classification method.

[0062] refer to Figure 2 The first stage of the two-stage voice / music classification device includes:

[0063] -State machine 201 for signal partitioning;

[0064] - Syllable start / start detector 202 based on relative frame energy;

[0065] -Feature extractor 203;

[0066] - Histogram-based outlier detector 204;

[0067] -Short-term feature vector filter 205;

[0068] - Nonlinear eigenvector transformer 206 (Box-Cox);

[0069] - Principal Component Analyzer (PCA) 207;

[0070] - Gaussian Mixture Model (GMM) Calculator 208;

[0071] - Adaptive smoother 209; and

[0072] - A classifier for classifying categories based on state 210.

[0073] The core encoder selection technique in the IVAS codec (the second stage of a two-stage speech / music classification device and method) builds upon the first stage of the two-stage speech / music classification device and method and passes the final output to perform the selection of a core encoder from ACELP (Algebraic Digital Excitation Linear Prediction), TCX (Transform Coded Excitation), and GSC (General Audio Signal Encoder), as described in reference [7], the entire contents of which are incorporated herein by reference. Other suitable core encoders may also be implemented within the scope of this disclosure.

[0074] Summarize:

[0075] -Then, the selected core encoder is applied to encode the main (dominant) channel (in the case of TD stereo encoding) or the downmixed mono channel (in the case of FD stereo encoding).

[0076] - The core encoder chooses to use additional high-level features computed on a window, which is typically longer than the window used in the first stage of a two-stage speech / music classification device approach.

[0077] - The core encoder uses its own optimizations to achieve seamless switching of the start / syllable initial detection logic. The output of the start / syllable detector differs from the output of the first-stage start / syllable detector.

[0078] The core encoder was initially selected based on the output of the first-stage dependent-state category classifier 210. This selection was then refined by examining additional high-level features and the output of the second-stage syllable start / start detector.

[0079] Figure 3 This is a schematic block diagram showing the second stage 300 of the two-stage speech / music classification device and the corresponding second stage 350 of the two-stage speech / music classification method.

[0080] refer to Figure 3 The second phase of the two-stage voice / music classification device includes:

[0081] - Extractor 301 with additional advanced features;

[0082] - The initial selector 302 of the core encoder; and

[0083] - Refiner 303 initially selected by the core encoder.

[0084] 2. The first stage of a two-stage speech / music classification device and method:

[0085] First, it should be mentioned that the GMM model was trained on a large, manually labeled training sample database using the Expectation-Maximization (EM) algorithm. The database contains mono items used in the EVS codec and some additional stereo items. The total size of the mono training database is approximately 650MB. The original mono files are converted to their corresponding binaural variants before being used as input to the IVAS codec. The total size of the additional stereo training database is approximately 700MB. The additional stereo database contains real recordings of speech signals from simulated dialogues, music samples downloaded from open-source sources on the internet, and some manually created items. The manually created stereo items were obtained by convolving mono speech samples with pairs of real binaural room impulse responses (BRIRs). These impulse responses correspond to typical room configurations such as small offices, seminar rooms, auditoriums, etc. Labels for the training items were created semi-automatically using voice activity detection (VAD) information extracted from the IVAS codec; this was not optimal, but given the size of the database, manual labeling frame by frame was not feasible.

[0086] 2.1 State Machine for Signal Division

[0087] refer to Figure 2 The first stage 250 of the two-stage speech / music classification method includes a signal division operation 251. To perform this operation, the first stage 200 of the two-stage speech / music classification device includes a state machine 201.

[0088] The concept of the state machine in the first stage is taken from the EVS codec. No major modifications were made to the IVAS codec. The purpose of state machine 201 is to divide the incoming audio signal into one of four states: INACTIVE, ENTRY, ACTIVE, and UNSTABLE.

[0089] Figure 4 This is a schematic block diagram showing the state machine 201 of the first stage 200 of the two-stage speech / music classification device and the signal division operation 251 of the first stage 250 of the two-stage speech / music classification method.

[0090] Figure 4 The schematic diagram also shows the transition conditions of the state machine 201 for transitioning the input sound signal from one state to another, and these transition conditions are related to the characteristics of the input sound signal.

[0091] The INACTIVE state 401, which indicates background noise, is selected as the initial state.

[0092] When the VAD flag 403 (see reference [1]) changes from “0” to “1”, state machine 201 switches from INACTIVE state 401 to ENTRY state 402. Any VAD detector or SAD (Sound Activity Detection) detector can be used to generate the VAD flag used by the first stage of the two-stage speech / music classification method and device. ENTRY state 402 marks the first syllable or the start of the input sound signal after a long period of silence.

[0093] After, for example, eight frames 405 in ENTRY state 402, state machine 201 enters ACTIVE state 404, which marks the start of a stable audio signal with sufficient energy (given energy level). If the energy 409 of the signal suddenly decreases while state machine 201 is in ENTRY state 402, state machine 201 transitions from ENTRY state to UNSTABLE state 407, corresponding to an input audio signal with an energy level close to background noise. Furthermore, if the VAD flag 403 changes from "1" to "0" while state machine 201 is in ENTRY state 402, state machine 201 returns to INACTIVE state 401. This ensures continuity of classification during brief pauses.

[0094] If the energy 406 of the stable signal (active state 404) suddenly drops to a level closer to the background noise, or the VAD flag 403 changes from "1" to "0", then state machine 201 switches from ACTIVE state 404 to UNSTABLE state 407.

[0095] After a period of time, such as 12 frames 410, in the UNSTABLE state 407, state machine 201 returns to the INACTIVE state 401. If, while state machine 201 is in the UNSTABLE state 407, the energy 408 of an unstable signal suddenly increases or the VAD flag 403 changes from "0" to "1", then state machine 210 returns to the ACTIVE state 404. This ensures the continuity of classification during brief pauses.

[0096] In the following description, the current state of state machine 201 is represented as f. SM The constants assigned to an individual state can be defined as follows:

[0097] INACTIVE f SM =-8

[0098] UNSTABLE f SM ∈<-7,-1>

[0099] ENTRY f SM ∈<0,7>

[0100] ACTIVE f SM =+8

[0101] In INACTIVE and ACTIVE states, f SM Corresponding to a single constant, while in the UNSTABLE and ENTRY states, depending on the progress of state machine 201, f SM Multiple values ​​can be presented. Therefore, in the UNSTABLE and ENTRY states, f SM It can be used as a short-term counter.

[0102] 2.2 Syllable onset / attack detector

[0103] refer to Figure 2 The first stage 250 of the two-stage speech / music classification method includes an operation 252 of syllable start / start detection based on relative frame energy. To perform this operation, the first stage 200 of the two-stage speech / music classification device includes a syllable start / start detector 202.

[0104] Syllable start / attack detector 202 and corresponding syllable start / attack detection operation 252 are suited for the purpose and function of speech / music classification in the IVAS codec. The objects particularly, but not exclusively, include the beginning of speech phonation (attack) and the localization of syllable starts in musical segments. These events are typically associated with sudden changes in the characteristics of the input sound signal. Successful detection of syllable starts and attacks after a period of signal inactivity allows for reduction of the influence of past information during fractional smoothing (described below). The syllable start / attack detection logic is implemented in state machine 201 (…). Figure 2 (playing a role in) Figure 4 The ENTRY state 402 plays a similar role. The difference between the two concepts is related to their input parameters. State machine 201 mainly uses VAD flag 403 from HE-SAD (High-Efficiency Sound Activity Detection) technology (see reference [1]). Figure 4 ), while the syllable start / start detector 252 uses the difference in relative frame energy.

[0105] Relative frame energy E r Frame energy can be calculated as the difference between frame energy and long-term average energy, expressed in dB. The frame energy in dB can be calculated using the following relationship:

[0106]

[0107] Where E CB (i) is the average energy per critical band (see reference [1]). The long-term average frame energy can be calculated using the following relationship:

[0108]

[0109] The initial value is The relative frame energy can be calculated as follows:

[0110]

[0111] The parameter used by the syllable start / start detector 252 is the cumulative sum of the differences between the relative energy of the input sound signal in the current frame and the relative energy of the input sound signal in the previous frame, updated in each frame. This parameter is initialized to 0 and is applied only if the relative energy E in the current frame is... r (n) is greater than the relative energy E in the previous frame. r It is only updated when (n-1) syllable start / start detector 252 uses, for example, the following relationship to update the cumulative sum v run (n):

[0112] v run (n)=v run (n-1)+(E r (n)-Er (n-1))

[0113] Where n is the index of the current frame. The syllable start / start detector 252 uses a cumulative sum v run (n) is used to update the counter v for the first / last syllable of a syllable. cnt The counter of the syllable start / start detector 252 is initialized to 0 and incremented by 1 in each frame of ENTRY state 402, where v run >5. Otherwise, it is reset to 0.

[0114] The output of the start / syllable detector 202 is a binary flag f. att For example, when 0 <v run When the value is less than 3, it is set to 1 to indicate that a syllable beginning / starting sound has been detected. Otherwise, the binary flag is set to 0 to indicate... No The first / starting sound of a syllable has been detected. This can be expressed as follows:

[0115]

[0116] As a non-restrictive example, by Figure 5 The graph illustrates the operation of the syllable initiation / starting syllable detector 202.

[0117] 2.3 Feature Extractor

[0118] refer to Figure 2 The first stage 250 of the two-stage speech / music classification method includes an operation 253 for extracting features from the input sound signal. To perform this operation, the first stage 200 of the two-stage speech / music classification device includes a feature extractor 203.

[0119] During the training phase of the GMM model, training samples are resampled to 16kHz, normalized to -26dBov (dBov is the dB level relative to the system overload point), and concatenated. The resampled and concatenated training samples are then fed into the encoder of the IVAS codec to collect features using feature extractor 203. For feature extraction purposes, the IVAS codec can operate in FD stereo coding mode, TD stereo coding mode, or any other stereo coding mode and at any bit rate. As a non-limiting example, feature extractor 203 operates in 16.4kbps TD stereo coding mode. Feature extractor 203 extracts the following features used in the GMM model for speech / music / noise classification:

[0120] Table 1: Features used in the GMM model

[0121]

[0122] Apart from the MFCC feature, all of the above features already exist in the EVS codec (see reference [1]).

[0123] Feature extractor 203 uses open-loop pitch T OL and speech measurement Further analysis was conducted. These measurements were calculated as the average of three adjacent signal segments of the input audio signal. Each segment was 10 ms long. The LSF (line spectral frequency) feature is a vector of M values, where M is the order of the LP (linear prediction) analysis. The residual energy from the LP analysis ∈ P It is a vector of M+1 values ​​calculated as part of the Levinson-Durbin algorithm. Short-term correlation plot C map It is a single-parameter characteristic calculated as a byproduct of harmonic spectrum analysis; it reflects the harmony and spectral stability of the input sound signal. Non-stationarity characteristic n sta This reflects the dynamics of spectral energy evolution. The dynamics in speech signals are typically higher than those in music signals.

[0124] MFCC characteristics are N corresponding to the Mel frequency cepstral coefficients. mel The vector of values, the Mel frequency cepstral coefficients are the result of the cosine transformation of the real logarithm of the short-term energy spectrum expressed on the Mel frequency scale (see reference [8], the entire contents of which are incorporated herein by reference).

[0125] The last two features P diff and P sta The calculations use, for example, normalized per-bin power spectra. Defined as

[0126]

[0127] Where P l This is the power spectrum per interval in the current frame calculated in the IVAS spectrum analysis routine (see reference [1]). It covers a frequency range corresponding to 150–3500 Hz. <k low ,k high Normalization is performed within the range >=<3,70>.

[0128] Power spectral difference P diff It can be defined as

[0129]

[0130] An index (n) has been added to explicitly represent the frame index.

[0131] The spectral stationarity characteristic P can be calculated from the sum of the ratios of the normalized power spectrum per interval and the power difference spectrum using the following relationship. sta :

[0132]

[0133] In frames containing frequency intervals that simultaneously have high amplitude and small spectral differences, spectral stationarity is generally high.

[0134] 2.4 Anomaly Detector Based on Individual Feature Histogram

[0135] refer to Figure 2 The first stage 250 of the two-stage speech / music classification method includes an operation 254 for detecting outlier features based on individual feature histograms. To perform operation 254, the first stage 200 of the two-stage speech / music classification device includes an outlier detector 204.

[0136] The GMM model is trained on a large training database using feature vectors collected from the IVAS codec. The accuracy of the GMM model is largely influenced by the statistical distribution of individual features. Optimal results are obtained when the features are normally distributed, for example, when N represents a statistical distribution X ~ N(μ,σ) with mean μ and variance σ. Figure 6 Histograms of selected features from a large training database are shown. It can be seen that... Figure 6 The histograms of some features do not show that they were drawn from a normal distribution.

[0137] GMM models can, to some extent, represent features of non-normal distributions. If the values ​​of one or more features differ significantly from their mean, the feature vector is identified as an outlier. Outliers often lead to incorrect probability estimates. Instead of discarding the feature vector, outlier features can be replaced with, for example, feature values ​​from previous frames, the average of multiple previous frames, or the global average of a large number of previous frames.

[0138] Detector 204 is based on the analysis of individual feature histograms computed on the training database (e.g., see the feature histograms shown). Figure 6 The figure shows a graph illustrating the detection of outliers based on histogram values. Figure 7 This involves detecting outliers in the first stage 200 of a two-stage speech / music classification device. For each feature, a lower bound H for outlier detection is calculated using, for example, the following relationship: low and upper limit H high :

[0139] For i = 0, ..., i max

[0140] For i = i max ,..,I

[0141] Where H(i) is a normalized feature histogram such that max(H(i)) = 1, and i is a frequency interval index ranging from 0 to I = 500 intervals, and i max It is the interval of the maximum values ​​in the histogram containing this feature. Threshold thr H Set to 1e -4 Threshold thr H This specific value can be interpreted as follows. If the true statistical distribution of the feature is a normal distribution with zero mean μ and variance σ, it can be rescaled so that its maximum value equals 1. In this case, the probability density function (PDF) can be expressed as...

[0142]

[0143] By using the threshold thr H Replace f xs (x|0,σ 2 Then rearrange the variables to obtain the following relation:

[0144]

[0145] For thr H =1e -4 The following results were obtained:

[0146]

[0147] Therefore, assuming the distribution is normal and scaled, a threshold 1e is applied. -4 This causes the probability density function to be adjusted to a range of ±2.83σ around the mean, making the probability density function f xs (0|0,σ 2 = 1. The probability that an eigenvalue lies outside the trimmed range is given by, for example, the following relation:

[0148]

[0149] Where erf(.) is the Gaussian error function known in statistical theory.

[0150] If the variance of the eigenvalues ​​is σ = 1, then the percentage of outliers detected is approximately 0.47%. The above calculation is only approximate because the true distribution of the eigenvalues ​​is not normally distributed. This is due to… Figure 6 Non-stationarity characteristics n staThe histogram illustrates this, where the tail to the right of the mean is "heavier" than the tail to the left of the mean. For example, if the sample variance σ is used as the basis for outlier detection, and the interval is set to, for example, ±3σ, many "good" values ​​to the right of the mean will be classified as outliers.

[0151] For the first stage of the two-stage speech / music classification method and device, calculate the lower bound H for each feature used in the 250 / 200 phase. low and upper limit H high The values ​​are stored in the memory of the IVAS codec. When the encoder of the IVAS codec is running, the outlier detector 204 will store the value X of each feature j in the current frame n. j (n) and the boundary H of this feature low and H high The comparison is performed, and features j with values ​​outside the corresponding range defined between the lower and upper limits are marked as outlier features. This can be represented as...

[0152]

[0153] Where F is the number of features. The outlier detector 204 includes a counter (not shown) for outlier features, c odv The number of detected outliers can be represented using, for example, the following relationship:

[0154]

[0155] If the number of outlier features is equal to or greater than, for example, 2, then the outlier detector 204 will set the binary flag f. out Set to 1. This can be represented as follows:

[0156]

[0157] The mark f out The feature vector used for signaling notification is an outlier. If the flag f out If the value is equal to 1, then the outlier feature X j (n) is replaced, for example, with a value from a previous frame, as shown below:

[0158] X j (n)=X j (n-1) For j = 1, ..., F, if f odv (j)=1

[0159] 2.5 Short-Term Eigenvector Filter

[0160] refer to Figure 2The first stage 250 of the two-stage speech / music classification method includes an operation 255 of filtering short-term feature vectors. To perform operation 255, the first stage 200 of the two-stage speech / music classification device includes a short-term feature vector filter 205 for smoothing the short-term vectors of the extracted features.

[0161] Speech / music classification accuracy is improved through feature vector smoothing. This can be achieved by applying a short-term infinite impulse response (IIR) filter, which serves as a short-term feature vector filter 205:

[0162] For j = 1, ..., F

[0163] in The feature representing short-term filtering in frame n, and α m =0.5 is the so-called forgetting factor.

[0164] In which f att =1 or f iut =1 Figure 4 In ENTRY state 402, feature vector smoothing (operation 255, filtering short-term feature vectors) is not performed in the frame to avoid the trailing effect on strong onsets or anomalous points at the beginning of ACTIVE signal segments where the information potential of the feature vectors in previous frames is limited. In frames classified as syllable beginnings / transients by the IVAS transient classification algorithm, in Figure 4 Smoothing is not performed in ACTIVE state 404 (stable signal) (see reference [1]). When the short-term eigenvector filtering operation 255 is not performed, the eigenvalues ​​X of the unfiltered vector are simply copied and used. j (n). This can be expressed by the following relation:

[0165] For j = 1, ..., F

[0166] In the following description, the eigenvalue X is used. j The original symbol of (n) is replaced That is, assumption

[0167] For j = 1, ..., F

[0168] 2.6 Nonlinear Eigenvector Transformation (Box-Cox)

[0169] refer to Figure 2 The first stage 250 of the two-stage speech / music classification method includes a nonlinear feature vector transformation operation 256. To perform operation 256, the first stage 200 of the two-stage speech / music classification device includes a nonlinear feature vector transformer 206.

[0170] like Figure 6 As shown in the histogram, many features used in speech / music classification are not normally distributed, and therefore cannot achieve the best accuracy of GMM. As a non-limiting example, the nonlinear feature vector transformer 206 can use the Box-Cox transform described in reference [9], the entire contents of which are incorporated herein by reference, to transform non-normal features into features with normal shapes. Box-Cox transform of feature X box It is a parameterized nonlinear power transform, defined as follows:

[0171]

[0172] Where λ is the exponent of the power transform, varying from -5 to +5 (see [link to relevant documentation]). Figure 8 Examples in; Figure 8 This is a graph showing the Box-Cox transform curves for different power transform exponent values ​​λ. The Box-Cox transform is only constrained for positive inputs. Therefore, a bias Δ is added to ensure that all input values ​​of the feature are positive. This bias corresponds to the smallest feature value in the entire training database.

[0173] For n = 0, ..., N

[0174] Where N is the number of samples of the features in the training database.

[0175] During the training process, the nonlinear eigenvector transformer 206 considers and tests all values ​​of the exponent λ to select the optimal value of the exponent λ based on a normality test. The normality test is based on the D'Agostino and Pearson method described in reference

[10] , the entire contents of which are incorporated herein by reference, and incorporates the skewness and kurtosis of the probability distribution function. The normality test produces the following skewness and kurtosis measurements r sk (SK Measurement):

[0176] r sk =s 2 +k 2

[0177] Where s is the z-score returned by the skewness test, and k is the z-score returned by the kurtosis test. For details on the skewness and kurtosis tests, see reference

[11] , the entire contents of which are incorporated herein by reference.

[0178] The normality test also returns the two-sided chi-square probability of the null hypothesis, i.e., the eigenvalues ​​drawn from a normal distribution. The optimal value of the exponent λ minimizes the SK measurement. This can be expressed by the following relationship:

[0179] For λ = -2, ..., 3

[0180] The subscript j indicates that the above minimization process is performed for each individual feature j = 1, ..., F.

[0181] In the encoder, the nonlinear eigenvector transform 206 applies the Box-Cox transform only to selected features that satisfy the following conditions related to SK measurement:

[0182] For j = 1, ..., F

[0183] Where r sk (j) is the SK measurement calculated for the j-th feature before the Box-Cox transformation, and r′ sk (j) is in the case of having the optimal exponential value λ j SK measurement after Box-Cox transformation. Optimal exponent value λ for the selected feature. j and related deviation Δ j It is stored in the memory of the IVAS codec.

[0184] In the following description, the eigenvalue X will be used. j The original symbol of (n) is used to replace X. box,j (n), that is, the assumption

[0185] X j (n)←X box,j (n) for the selected features

[0186] 2.7 Principal Component Analyzer

[0187] refer to Figure 2 The first stage 250 of the two-stage speech / music classification method includes principal component analysis (PCA) operation 257 to reduce the feature dimensionality of the sound signal and increase the discriminative power of the sound signal classification. To perform operation 257, the first stage 200 of the two-stage speech / music classification device includes a principal component analyzer 207.

[0188] Following the short-term eigenvector filtering operation 255 and the nonlinear eigenvector transformation operation 256, the principal component analyzer 207 standardizes the eigenvectors by removing the mean of the features and scaling them to unit variance. This can be achieved using the following relationship:

[0189] For j = 1, ..., F

[0190] in μ represents the standardized feature. j It is the mean and s j Features X in the training databasej The standard deviation, and as mentioned above, n represents the current frame.

[0191] Feature X j The average value μ j And deviations j The following can be calculated:

[0192]

[0193]

[0194] Where N represents the total number of frames in the training database.

[0195] In the following description, the eigenvalue X will be used. j The original symbol of (n) is used to replace That is, the assumption is:

[0196] For n = 1, ..., N

[0197] Then, the principal component analyzer 207 uses PCA to process the eigenvectors, where the dimensionality is reduced, for example, from F = 15 to F... PCA =12. PCA is an orthogonal transformation that converts a set of potentially correlated features into a set of linearly uncorrelated variables called principal components (see reference

[12] , the entire contents of which are incorporated herein by reference). In speech / music classification methods, analyzer 207 uses, for example, the following relation to transform the feature vector:

[0198] Y(n) = W T X(n)

[0199] Where X(n) is the F-dimensional column eigenvector, and W is the PCA loading matrix F×F PCA The column is X T The eigenvectors of (n)X(n) are given, where the superscript T denotes the vector transpose. The payload is found using singular value decomposition (SVD) of the feature samples in the training database. During the training phase, the payload is computed only for active frames, such as frames with the VAD flag set to 1. The computed payload is stored in the memory of the IVAS codec.

[0200] In the following description, the original notation of the feature vector X(n) will be used instead of Y(n), i.e., it is assumed that:

[0201] X(n)←Y(n)

[0202] 2.8 Gaussian Mixture Model (GMM)

[0203] refer to Figure 2The first stage 250 of the two-stage speech / music classification method includes an operation 258 of Gaussian Mixture Model (GMM) computation. To perform operation 258, the first stage 200 of the two-stage speech / music classification device includes a GMM calculator 208. As will be seen, the GMM calculator 208 estimates the decision bias parameter by maximizing the harmonic balance accuracy on the training database. The decision bias is a parameter added to the GMM due to insufficient training data to improve the accuracy of the "MUSIC" classification determination.

[0204] Multivariate GMMs are parameterized by a mixture of component weights, component means, and covariance matrices. The speech / music classification method uses three GMMs, each trained on its own training database: a "speech" GMM, a "music" GMM, and a "noise" GMM. In a GMM with K components, each component has its own mean μ. k The sum of the covariance matrix ∑ k In speech / music classification methods, three (3) GMMs are fixed with K = 6 components. Component weights are constrained. To represent φ k This is done so that the probability distribution is normalized. The probability p(X) generated by a given feature vector X from a GMM can be calculated using the following relation:

[0205]

[0206] In the above relationship, calculating the exponential function exp(...) is a complex operation. The parameters of the GMM are calculated using the Expectation-Maximization (EM) algorithm. It is well known that the Expectation-Maximization algorithm can be used for latent variables (variables that cannot be directly observed and are actually inferred from the values ​​of other observed variables) to predict their values ​​given a general form of the probability distributions governing these latent variables.

[0207] To reduce the complexity of probability calculations, the above relationship can be simplified by taking the logarithm of the inner terms of the summation term ∑, as shown below:

[0208]

[0209] The output of the simplified formula above is called a "score". A score is an unbounded variable proportional to the log-likelihood. The higher the score, the higher the probability that a given feature vector was generated by the GMM. The score is calculated by the GMM calculator 208 for each of the three GMMs. The score for the "speech" GMM. S (X) and the score of the "Music" GMM M (X) is combined into a single value Δ by calculating their difference. s (X), to obtain the difference fraction Δ s(X), for example, using the following relation:

[0210] Δ s (X) = score M (X)-score S (X)

[0211] A negative value of the difference fraction indicates that the input sound signal is a speech signal, while a positive value indicates that the input sound signal is a music signal. The difference fraction dlp(X, b) can be calculated by adding non-negative values ​​to the difference fraction using the following relationship: s Introducing decision bias b into the calculation of ) s :

[0212] dlp(X, b) s ) = score M (X)-score S (X)+b s

[0213] Decision bias b s The value is found based on an ensemble of difference scores computed on the training database. The decision bias b is found. s The process of obtaining the value can be described as follows.

[0214] Let X t Let yt be the matrix representing the feature vectors from the training database. Let yt be the corresponding label vector. Let the value of the real SPEECH frame in this vector be +1.0, and the value of other frames be 0. The total number of ACTIVE frames in the training database is denoted as N. act .

[0215] After EM training, i.e., when the parameters of GMM are known, the difference score dlp(X, b) can be calculated in the active frames of the training database. s Then, it may be possible to use, for example, the following relation to predict the label y in the active frames of the training database. pred (n):

[0216] y pred (n)=0.5*[sign[-1.0dlp(X(n),b s =0)]+1.0]

[0217] Where sign[.] is the sign function, and dlp(X(n), b s =0) means that under the assumption b s The difference fraction calculated when y = 0. The resulting value of the label y. pred (n) equal to +1.0 indicates SPEECH, or equal to 0 indicates MUSIC or NOISE.

[0218] The accuracy of such binary predictions can be summarized using the following four statistical methods:

[0219]

[0220]

[0221]

[0222]

[0223] Where E r It is the relative frame energy used as a sample weighting factor. Statistical measurement has the following meanings: c tp It refers to the number of true positives, i.e., the number of hits in the SPEECH classification, c fp It is the number of false positives, that is, the number of frames misclassified by the MUSIC classification, c tn It refers to the number of true negatives, i.e., the number of hits in the MUSIC / NOISE classification, and c fn It represents the number of false negatives, i.e., the number of frames that are misclassified in the SPEECH classification.

[0224] The statistics defined above can be used to calculate the true positive rate, commonly known as the recall rate.

[0225]

[0226] Furthermore, the true negative rate, often referred to as specificity, is...

[0227]

[0228] By taking the harmonic mean of TPR and TNR and using the following relationship, recall (TPR) and specificity (TNR) can be combined into a single number:

[0229]

[0230] The result is called harmonic balance accuracy.

[0231] By maximizing the use of the label / predictor y pred (n) The harmonic balance accuracy defined above is used to find the decision bias b. s The value of b, where b s The selection is made from the interval (-2, 2) of the consecutive steps. The space of candidate values ​​for the decision bias is approximately logarithmic, with a higher concentration of values ​​near 0.

[0232] Use the discovered decision bias b s The difference fraction dlp(X, b) calculated from the value of the difference fraction. sThe value is limited to, for example, a range of (-30.0, +30.0). This applies when the VAD flag is 0, or when the total frame energy E... tot When it is below 10dB, or when the speech / music classification method is in ENTRY state 402 and f att or f out When the value is 1, the difference fraction dlp(X, b) s ) was reset to 0.

[0233] 2.9 Adaptive Smoother

[0234] refer to Figure 2 The first stage 250 of the two-stage speech / music classification method includes an adaptive smoothing operation 259. To perform operation 259, the first stage 200 of the two-stage speech / music classification device includes an adaptive smoother 209.

[0235] The adaptive smoother 209 includes, for example, an adaptive IIR filter to smooth the differential fraction dlp(X, b) of frame n identified as dlp(n) from the GMM calculator 208. s Adaptive smoothing and filtering operation 259 can be described using the following operations:

[0236] wdlp(n)=wght(n)·wdlp(n-1)+(1-wght(n))·dlp(n)

[0237] Where wdlp(n) is the obtained smoothed difference score, wght(n) is the so-called forgetting factor of the adaptive IIR filter, and n represents the frame index.

[0238] The forgetting factor is the product of three independent parameters, as shown in the following formula:

[0239] wght(n)=wrelE(n)·wdrop(n)·wrise(n)

[0240] The parameter wrelE(n) is related to the relative energy E of the current frame. r (n) is linearly proportional and can be calculated using the following relationship:

[0241]

[0242] The parameter wrelE(n) is limited to, for example, the interval (0.9, 0.99). The constants used in the above relationship have the following interpretation. When the relative energy is above 15 dB, the parameter wrelE(n) reaches the upper threshold of 0.99. Similarly, when the relative energy is below -15 dB, the parameter wrelE(n) reaches the lower threshold of 0.9. The value of the parameter wrelE(n) affects the forgetting factor wght(n) of the adaptive IIR filter of the smoother 209. Smoothing is stronger in segments with weaker energy, where the desired features carry less relevant information about the input signal.

[0243] The parameter wdrop(n) is proportional to the derivative of the difference fraction dlp(n). First, the short-term mean dlp of the difference fraction dlp(n) is calculated using, for example, the following relationship. ST (n):

[0244] dlp ST (n) = 0.8·dlp ST (n-1)+0.2·dlp(n)

[0245] The parameter wdrop(n) is set to 0 and is modified only in frames that meet the following two conditions:

[0246] dlp(n) < 0

[0247] dlp(n) < dlp ST (n)

[0248] Therefore, the adaptive smoother 209 only updates the parameter wdrop(n) when the differential score dlp(n) has a decreasing trend and when it indicates that the current frame belongs to the SPEECH category. In the first frame, when both conditions are met and if dlp ST When (n) > 0, the parameter wdrop(n) is set to

[0249] wdrop(n) = -dlp(n)

[0250] Otherwise, the adaptive smoother 209 steadily increases the parameter wdrop(n) using, for example, the following relationship:

[0251] wdrop(n) = wdrop(n-1) + (dlp ST (n-1)-dlp(n))

[0252] If the two conditions defined above are not met, the parameter wdrop(n) is reset to 0. Therefore, the parameter wdrop(n) reacts to a sudden drop in the differential fraction dlp(n) indicating a level below zero at the beginning of the latent syllable. The final value of the parameter wdrop(n) is linearly mapped to, for example, the interval (0.7, 1.0), as shown in the following relation:

[0253]

[0254] Please note that in the formula above, the value of wdrop(n) is "overwritten" to simplify the notation.

[0255] The adaptive smoother 209 similarly calculates the parameter wrise(n), except that the parameter wdrop(n) reacts to a sudden increase in the differential fraction dlp(n) indicating the beginning of a potential musical syllable. The parameter wrise(n) is set to 0, but is modified in frames that meet the following conditions:

[0256] f SM (n) = 8 (ACTIVE)

[0257] dlp ST (n)>0

[0258] dlp Sr (n)>dlp ST (n-1)

[0259] Therefore, when the difference score dlp(n) shows an increasing trend and when the difference score dlp(n) indicates that the current frame n belongs to the MUSIC category, the adaptive smoother 209 only updates the parameter wrise(n) in the ACTIVE state 404 of the input audio signal (see...). Figure 4 ).

[0260] In the first frame, when the above three (3) specified conditions are met, and if the short-term mean dlp ST If (n-1) < 0, then the third parameter wrise(n) is set to:

[0261] wrise(n) = -dlp ST (n)

[0262] Otherwise, the adaptive smoother 209 steadily increases the parameter wrise(n) according to, for example, the following relationship:

[0263] wrise(n)=wrise(n-1)+(dlp ST (n)-dlp ST (n-1))

[0264] If the above three (3) conditions are not met, the parameter wrise(n) will be reset to 0. Therefore, the third parameter wrise(n) reacts to a sudden increase in the differential fraction dlp(n) above the zero level indicating the beginning of a potential musical syllable. The final value of the parameter wrise(n) is linearly mapped to, for example, the interval (0.95, 1.0), as shown below:

[0265]

[0266] Note that in the formula above, the value of the parameter wrise(n) is "overridden" to simplify the notation.

[0267] Figure 9 The graph shown is a non-limiting example illustrating the behavior of parameters wdrop(n) and wrise(n) on a short segment of a speech signal with background music. The peak of parameter wdrop(n) is typically located near the beginning of a speech syllable, while the peak of parameter wrise(n) is typically located where the speech gradually fades and the background music begins to dominate the signal content.

[0268] The forgetting factor wght(n) of the adaptive IIR filter in the adaptive smoother 209 decreases in response to strong SPEECH or MUSIC signal content. To this end, the adaptive smoother 209 analyzes the long-term mean of the difference fraction dlp(n). and long-term variance Calculate using, for example, the following relation:

[0269]

[0270]

[0271] In ENTRY state 402 (for input sound signals) Figure 4 (), long-term mean and long-term variance When the absolute value of the difference score dlp(n) is high and the variation around its mean is low, the speech / music classification device is likely to be deterministic about its content. This can be represented by the following ratio of long-term mean to long-term variance:

[0272]

[0273] expression r m2v (n) corresponds to the long-term standard deviation of the difference fraction. Using, for example, the following relationship, where r... m2v In frames where (n)>15, the forgetting factor wght(n) of the adaptive IIR filter of adaptive smoother 259 decreases:

[0274] wght(n) ← 0.9 · wght(n)

[0275] The final value of the forgetting factor wght(n) of the adaptive IIR filter of the adaptive smoother 209 is limited to, for example, a range of (0.01, 1.0). In the total frame energy E tot In frames where n is below 10 dB, the forgetting factor wght(n) is set to, for example, 0.92. This ensures proper smoothing of the difference fraction dlp(n) during the silent period.

[0276] As described below, the filtered and smoothed difference score wdlp(n) is a parameter used for category decision in speech / music classification methods.

[0277] 2.10 State-dependent classifiers

[0278] refer to Figure 2 The first stage 250 of the two-stage speech / music classification method includes operation 260, which classifies the input sound signal into a category based on the dependent state in response to a threshold of differential fraction distribution and dependency direction. To perform operation 260, the first stage 200 of the two-stage speech / music classification device includes a category classifier 210 based on the dependent state.

[0279] Operation 260 is the final operation of the first stage 250 of the two-stage speech / music classification method, and includes classifying the input sound signal into the following three final categories:

[0280] ·SPEECH / NOISE(0)

[0281] ·UNCLEAR(1)

[0282] ·MUSIC(2)

[0283] Above, the numbers in parentheses are numerical constants related to the last three categories. The set of categories above differs slightly from the categories of difference scores discussed so far. The first difference is that the SPEECH and NOISE classes are combined. This is to facilitate the core encoder selection mechanism (described below), where the ACELP encoder core is typically selected for encoding both the speech signal and background noise. A new category, UNCLEAR, has been added to the set. Frames falling into this category typically appear in speech segments with high-level additional background music. The smoothed difference score wdlp(n) of frames in the UNCLEAR category is mostly close to 0. Figure 10 The graph shows the distribution of the smoothed difference scores wdlp(n) of the training database and their relationship with the final classifications SPEECH / NOISE, UNCLEAR, and MUSIC.

[0284] Let d SMC (n) represents the final classification selected by the classifier 210 of the dependent state.

[0285] When the input audio signal is in ENTRY state 402 in the current frame (see...) Figure 4 When the state-dependent classifier 210 selects the final classification d based on the weighted average of the difference scores dlp(n) calculated in frames belonging to ENTRY state 402 and preceding the current frame. SMC (n). The weighted average can be calculated using the following formula:

[0286] For n = n ENTRY ,..,n ENTRY +7

[0287] Where n ENTRY The start (frame) of ENTRY state 402 is marked, and α k (nn ENTRY ) represents the weights of the samples dlp(n) in the ENTRY state. Therefore, the weighted average wdlp depends on the position of the current frame relative to the start (frame) of the ENTRY state. ENTRY The number of samples used in (n) ranges from 0 to 7. This is in Figure 11 As shown in the figure, Figure 11 A graph showing the weights sorted in descending order, such that the most recent frame is associated with the highest weight. Table 2 below shows examples of weights that can be used for this averaging.

[0288] Table 2: Weights used for averaging in the ENTRY state

[0289]

[0290]

[0291] If the absolute frame energy E tot If the value is below, for example, 10 dB in the current frame, then the state-dependent classifier 210 will ultimately classify d. SMC (n) is set to SPEECH / NOISE, without considering the difference fraction dlp(n). This is to avoid misclassification during the silent period.

[0292] If the weighted average of the difference fractions in the ENTRY state is wdlp ENTRY If (n) is less than, for example, 2.0, then the state-dependent classifier 210 will ultimately classify d. SMC (n) is set to SPEECH / NOISE.

[0293] If the weighted average of the difference fractions in the ENTRY state is wdlp ENTRY If (n) is higher than, for example, 2.0, then the state-dependent classifier 210 sets the final classification d based on the non-smooth difference score dlp(n) in the current frame. SMC (n). If dlp(n) is higher, for example, higher than 2.0, then the final classification is MUSIC. Otherwise, it is UNCLEAR.

[0294] In other states of input sound signal (see...) Figure 4 The state-dependent classifier 210 selects a final classification in the current frame based on a smoothed difference score wdlp(n) and the final classification selected in the previous frame. The final classification in the current frame is first initialized with the classification from the previous frame, i.e.

[0295] d SMC (n)=d SMC (n-1)

[0296] If the smoothed difference score wdlp(n) crosses a different classification threshold than the classification selected in the previous frame (see Table 3), the state-dependent classifier 210 can change its decision. The transitions between these classifications are as follows: Figure 10 As shown. For example, if the final category d selected in the previous frame... SMC If (n) is SPEECH / NOISE, and the smoothing difference score wdlp(n) in the current frame is higher than, for example, 1.0, then the final classification d in the current frame is... SMC (n) will be changed to UNCLEAR. Figure 10 The graph shows a histogram of the smoothed difference scores wdlp(n) for the SPEECH / NOISE final classification and the MUSIC final classification computed on the training database, excluding INACTIVE frames. Figure 10 The graph shows two sets of thresholds: one for the SPEECH / NOISE->UNCLEAR->MUSIC conversion, and the other for the reverse direction, MUSIC->UNCLEAR->SPEECH / NOISE conversion. Final classification d SMC (n) There is no direct switch from SPEECH to MUSIC, or vice versa. The value of the decision threshold indicates whether the classifier 210, depending on the state, tends to classify the final SPEECH / NOISE. Table 3 below summarizes examples of transitions between classifications and related thresholds.

[0297] Table 3: Thresholds for Category Transition arrive

[0298]

[0299] As mentioned above, the transition between categories is driven not only by the value of the smooth difference score wdlp(n), but also by the final category selected in the previous frame. Figure 12 The class transition diagram shows a complete set of transition rules between categories.

[0300] Figure 12 The arrows indicate the direction in which the classification might change when (or more) conditions within the corresponding rhombus are met. When multiple conditions exist within the rhombus, it is assumed that a logical "AND" exists between them, meaning that all conditions must be met for a transformation to occur. If the arrow indicates a condition of "≥X frames," it means that the classification can only change after at least X frames. This adds a brief lag to some transformations.

[0301] exist Figure 12 In the middle, the symbol f sp The short pitch flag is a byproduct of the stable high pitch analysis module of the IVAS codec (see reference [1]). In a pre-selected active frame, the short pitch flag is set to 1, indicating a high value in the speech measurement. The speech measurement is calculated as the average normalized correlation value of the three adjacent half-frames of the current frame as follows:

[0302]

[0303] in It is the normalized autocorrelation function in the current frame, and the upper index [k] refers to the position of the half-frame window. The normalized autocorrelation function is calculated as part of the open-loop pitch analysis module of the IVAS codec (see reference [1], section 5.1.11.3.2).

[0304] short pitch marker f sp The following settings can be configured in a pre-selected frame:

[0305]

[0306] in It is a long-term measurement of the maximum high pitch correlation value in the current frame (see reference [1]), and This is a long-term speech measurement in the current frame. Long-term measurement and The following formula can be used for calculation:

[0307]

[0308]

[0309] and It is the maximum high pitch correlation value calculated as part of the stable high pitch analysis module of the IVAS codec (see reference [1]).

[0310] exist Figure 12 In the middle, parameter c VAD This is the counter for the ACTIVE frame. Counter c VAD It is initialized to zero and reset to zero in all frames where the VAD flag is 0. Counter c VAD Only increment the VAD flag by 1 in frames where it is 1, until it reaches a threshold (e.g., 50) or until the VAD flag returns to zero.

[0311] Parameter v run (n) is defined in Section 2.2 (Syllable Beginning / Start Detection) of this disclosure.

[0312] 3. Core Encoder Selection

[0313] Figure 3 This is a schematic block diagram showing the second stage 300 of the two-stage speech / music classification device and the corresponding second stage 350 of the two-stage speech / music classification method.

[0314] In the second stage 350 / 300 of the two-stage speech / music classification method and device, the final classification d is selected by the state-dependent category classifier 210. SMC (n) is “mapped” to one of the three core encoder technologies of the IVAS codec: ACELP (Algebraic Digital Excited Linear Prediction), GSC (General Audio Signal Coding), or TCX (Transform Coded Excited). This is known as a three-way classification. It does not guarantee that the chosen technology will be used as the core encoder, as other factors, such as bit rate or bandwidth limitations, influence the decision. However, for common types of input audio signals, the initial choice of core encoder technology is generally accepted.

[0315] In addition to the category d selected by the category classifier 210 based on the dependent state in the first stage SMC In addition to (n), the core encoder selection mechanism also considers some additional advanced features.

[0316] 3.1 Additional Advanced Feature Extractor

[0317] refer to Figure 3 The second stage 350 of the two-stage speech / music classification method includes an operation 351 for extracting additional high-level features from the input sound signal. To perform operation 351, the second stage 300 of the two-stage speech / music classification device includes an additional high-level feature extractor 301.

[0318] In the first stage 200 / 250 of the two-stage speech / music classification device and method, most features are typically computed on short segments (frames) of the input audio signal, typically no more than 80 ms. This allows for rapid responses to events such as the beginning or offset of speech syllables in the presence of background music. However, this also leads to a relatively high misclassification rate. Adaptive smoothing mitigates misclassification to some extent, as described in Section 2.9 above, but it is not sufficiently effective for certain types of signals. Therefore, as part of the second stage 300 / 350 of the two-stage speech / music classification device and method, the classification method can be modified to select the most suitable core encoder technique for certain types of signals. SMC (n). To detect this type of signal, the detector typically computes additional high-level features and / or flags on longer segments of the input signal.

[0319] 3.1.1 Long-term signal stability

[0320] Long-term signal stability is a characteristic of the input sound signal that can be used for successful discrimination in vocal and operatic music. In the context of core encoder selection, signal stability is understood as the long-term stationarity of segments with high autocorrelation. An additional advanced feature extractor 301 is based on “speech” measurements. This can be used to estimate long-term signal stability characteristics. Speech parameters calculated over multiple (e.g., 10) frames can be used. variance vor var (n), the long-term signal stability in the current frame n is estimated using the following relationship:

[0321]

[0322]

[0323] In the above formula, It is the average speech over 10 frames.

[0324] For greater robustness, IIR filters are used to smooth the speech parameters in the current frame n according to, for example, the following formula:

[0325]

[0326] If the smooth speech parameter vor LT (n) is high enough, and the variance of the speech parameters vor car If (n) is low enough, then the input signal is considered "stable" for the purpose of core encoder selection. This is achieved by adjusting the value cor. LT (n) and cor var(n) is compared with a predefined threshold and measured using binary flags set according to rules such as the following:

[0327]

[0328] binary flag f STAB (n) is an indicator of long-term signal stability and is used in the selection of the core encoder, which is discussed later in this disclosure.

[0329] 3.1.2 Segment Start Detection

[0330] Extractor 301 extracts segment onset features from multiple (e.g., 32) segments of the current frame n, such as... Figure 13 As shown.

[0331] In each segment, the additional advanced feature extractor 301 uses, for example, the following relationship to calculate the energy E. ata (k):

[0332]

[0333] Where s(n) is the input audio signal in the current frame n, k is the index of the segment, and i is the index of the sample in the segment. Then, the start position is calculated as the index of the segment with the maximum energy, as shown below:

[0334]

[0335] The additional advanced feature extractor 301 extracts features from the starting note of the current frame n (segment k = k). ata The energy E of the input sound signal s(n) from the end (segment 31) ata The average value of (k) (the numerator of the following equation) and the energy E of the input signal s(n) from the beginning of the current frame n (segment 0) to 3 / 4 (segment 24) ata The intensity of the attack is estimated by comparing the average value of (k) (the denominator of the following formula). ata Strength str ata This estimation is performed using, for example, the following relation:

[0336]

[0337] If the value is str ata If the pitch is higher than, for example, 8, then the onset is considered strong enough, and the fragment k ata It is used as an indicator to signal the start position within the current frame n. Otherwise, the indicator k ataSetting it to 0 indicates that no attack was detected. Attacks are only detected in GENERIC frame types notified by IVAS frame type selection logic signaling (see reference [1]). To reduce false attack detections, the segment k = k that has been detected as an attack is selected. ata Energy E ata (k ata ) and the previous frame of the current frame n The energy E of the segments (segments 2 to 21) ata (k) compares (str) 3_4 (k)), for example, using the following relation:

[0338]

[0339] If for the segment k = 2, ..., 21, k ≠ k ata Any comparison value of the fragment str 3_4 If (k) is less than, for example, 2, then k ata A value set to 0 indicates that no attack was detected. In other words, the energy of the segment containing the attack must be at least equal to the energy of the preceding segment in the current frame. It has twice the energy of other segments.

[0340] The above mechanism ensures that the onset of the sound mainly occurs after the current frame. They were detected in the medium, which makes them suitable for encoding using ACELP or GSC technologies.

[0341] For unvoiced frames classified as UNVOICED_CLAS, UNVOICED_TRANSITION, or ONSET by the IVAS FEC classification module (see reference [1]), the advanced feature extractor 301 extracts the starting segment k = k ata Energy E ata (k ata (The numerator of the following equation) and the energy E in the first 32 segments before the onset of sound. ata The intensity of the attack is estimated by comparing the average value of (k) (the denominator of the following formula). ata Use, for example, the following relation:

[0342]

[0343] In the above formula, the negative indices in the denominator refer to the segment energy E in the previous frame. ata The value of (k). If the strength str is calculated using the formula above... ata Above, for example, 16, the onset is strong enough, and k ata It is used to signal the position of the start of a tone within the current frame. Otherwise, k ataA threshold of 0 indicates that no attack was detected. If the last frame is classified as UNVOICED_CLAS by the IVAS FEC classification module, then the threshold is set to, for example, 12 instead of 16.

[0344] For unvoiced frames classified as UNVOICED_CLAS, UNVOICED_TRANSITION, or ONSET by the IVAS FEC classification module (see reference [1]), another condition must be met to consider the detected onset sufficiently strong. This condition is related to the long-term average energy. In comparison, the energy E of the initial sound ata (k) must be high enough, and this long-term average energy can be calculated as follows:

[0345]

[0346] in For example, if the ratio of the following formula is greater than 20, then the onset is strong enough:

[0347]

[0348] Otherwise, k ata A value of 0 indicates that no attack sound was detected.

[0349] If an attack has been detected in a previous frame, then in the current frame n, k ata It was reset to 0 to prevent the trailing effect of the sound.

[0350] For other frame types (excluding UNVOICED and GENERIC frames as described above), the additional advanced feature extractor 301 extracts the segment containing the onset sound k = k according to, for example, the following ratio. ata Energy E ata (k ata Energy E in other segments ata (k) comparison:

[0351] And k≠k ata

[0352] And if for k = 2, ..., 21, k ≠ k ata Any comparison value str other If (k) is below, for example, 1.3, then the onset is considered weak, and k ata Set to 0. Otherwise, fragment k ata It will be used to signal the position of the tone in the current frame.

[0353] Therefore, the final output of the additional advanced feature detector 301 regarding the detection of the attack sound in a segment is a signal containing the attack sound or k. ata The index of 0 is k=kata The segment. If the index is positive, the attack sound is detected. Otherwise, no attack sound is detected.

[0354] 3.1.3 Signal Pitch Estimation

[0355] In the second stage of the two-stage speech / music classification device and method, the pitch of the input sound signal is represented as a pitch binary symbol, which reflects the spectral stability and consonance of the input signal in the lower frequency range up to 4 kHz. An additional advanced feature extractor 301 extracts features based on the correlation plot S. map (n, k) calculates the binary flag of the tone. The correlation plot is a byproduct of the tone stability analysis in the IVAS encoder (see reference [1]).

[0356] Correlation plots are a measure of signal stability and cotone. They are calculated from the first, for example, 80 intervals of the residual energy spectrum in the logarithmic domain, E... dB，res (k), k = 0, ..., 79 (see reference [1]). Correlation plots were calculated for segments with peaks in the residual energy spectrum. These segments are determined by parameter i. min (p) is defined, where p = 1, ..., N min It is a fragment index, and N min It represents the total number of segments.

[0357] Define the set of indices belonging to a specific fragment x as

[0358] PK(p)={i|i≥i min (p), and i < i min (p+1), and i < 80}

[0359] Then, the correlation plot can be calculated as follows.

[0360]

[0361] Use an IIR filter to examine the correlation plot M cor (PK(p)) is smoothed and summed over intervals within the frequency range k = 0, ..., 79 to produce a single number, for example, using the following two relations:

[0362]

[0363]

[0364] Where n represents the current frame and k represents the frequency interval. The weight β(n) used in the above equation is called the soft VAD parameter. It is initialized to 0 and can be updated in each frame.

[0365] β(n) = 0.95·β(n-1) + 0.05·f VAD (n)

[0366] Where f VAD (n) is the binary VAD flag of the IVAS encoder (see reference [1]). The weight β(n) is limited to, for example, the range of (0.05, 0.95). Extractor 301 extracts the weight β(n) by passing S... mass With adaptive threshold thr mass Comparison to set pitch marker ft on Threshold thr mass It is initialized to, for example, 0.65, and increments or decrements in each frame by, for example, 0.01. If S mass If the threshold is greater than 0.65, then the threshold thr mass Increase by 0.01, otherwise decrease by 0.01. Threshold thr mass The upper limit is, for example, 0.75, and the lower limit is, for example, 0.55. This gives the pitch symbol f ton A small lag has been added.

[0367] If S mass Higher than thr mass Then the tone mark f ton It is set to 1. Otherwise, it is set to 0.

[0368] 3.1.4 Peak-to-Average Ratio of the Spectrum

[0369] Another advanced feature used in the core encoder selection mechanism is the peak-to-average power ratio (PAPR). This feature is a measure of the spectral sharpness of the input audio signal s(n). Extractor 301 calculates this advanced feature, S, from the power spectrum of the input signal s(n) in the logarithmic domain. LT (n, k), k = 0, ..., 79, for example, in the range from 0 to 4 kHz. However, the power spectrum S is first smoothed using an IIR filter. LT (n, k), for example, using the following relation:

[0370]

[0371] Where n represents the current frame and k represents the frequency interval. The peak-to-average power ratio (PAPR) is calculated using, for example, the following relationship:

[0372]

[0373] 3.2 Core Encoder Initial Selector

[0374] refer to Figure 3The second stage 350 of the two-stage speech / music classification method includes an operation 352 for initial selection of the core encoder. To perform operation 352, the second stage 300 of the two-stage speech / music classification device includes a core encoder initial selector 302.

[0375] Selector 302 makes the initial selection of the core encoder and encoder based on (a) the relative frame energy E. r (b) The final classification selected in the first stage of the two-stage speech / music classification device and method. SMC (n), and (c) additional high-level features r p2a (n), S mass and thr mass (as described above). In Figure 14 The schematic diagram illustrates the selection mechanism used by the core encoder initial selector 302.

[0376] Let d core ∈{0,1,2} means Figure 14 The core encoder technology selected in the mechanism is categorized as follows: "0" represents ACELP technology, "1" represents GSC technology, and "2" represents TCX technology. Therefore, the initial selection of the core encoder technology closely follows the final classification from the first stage of the two-stage speech / music classification device and method. SMC (n) Assignment. Exceptions are related to strong tone signals; TCX technology is chosen because it provides better quality.

[0377] 3.3 Core Encoder Selection Refiner

[0378] refer to Figure 3 The second stage 350 of the two-stage speech / music classification method includes a refinement operation 353 for the initial selection of the core encoder. To perform operation 353, the second stage 300 of the two-stage speech / music classification device includes a core encoder selection refiner 303.

[0379] When d core =1, meaning that when the GSC core encoder is initially selected for core encoding, the core encoder selection refiner 303 can change the core encoder technology. This can occur, for example, in music projects classified as having low energy below 400Hz. The affected segments of the input signal can be identified by analyzing the following energy ratios:

[0380]

[0381] Where E bin (k), k = 0, ..., 127 is the power spectrum of the input signal at each frequency interval k in the linear domain, E tot It is the total energy of a signal segment (frame).

[0382] The summation in the molecules extends over the first eight frequency intervals of the energy spectrum, corresponding to the frequency range of 0 to 400 Hz. The core encoder selects refiner 303 to calculate and analyze the energy ratio in frames previously classified as MUSIC with considerable accuracy. LF For example, the core encoder technology changes from GSC to ACELP under the following conditions:

[0383]

[0384] For signals with very short and stable pitch periods, GSC is not the optimal core encoder technique. Therefore, as a non-limiting example, when f sp =1, When the core encoder selection refiner 303 changes the core encoder technology from GSC to ACELP or TCX, the following applies:

[0385]

[0386] Highly correlated signals with low energy variations are another type of signal for which the GSC core encoder technology is unsuitable. For these signals, the core encoder selection refiner 303 switches the core encoder technology from GSC to TCX. As a non-limiting example, this change in the core encoder is made when the following conditions are met:

[0387]

[0388] in It is the absolute pitch value of the first half of the open-loop pitch analysis in the current frame n (see reference [1]).

[0389] Finally, in a non-limiting example, the core encoder selection refiner 303 can change the initial core encoder selection from GSC to ACELP in the frame where the attack is detected if the following condition is met:

[0390]

[0391] Mark f bo_GSC It is an indicator that changes to the core encoder technology have been enabled.

[0392] The above conditions ensure that this change from GSC to ACELP in the core encoder only occurs in segments with rising energy. If the above conditions are met, and simultaneously, the frame counter TC is switched in the IVAS codec. cnt If it has already been set to 1 (Reference [1]), then the core encoder selection refiner 303 will change the core encoder to ACELP. That is:

[0393]

[0394] Furthermore, when the core encoder technology is changed to ACELP, the frame type will be set to TRANSITION. This means that the intro will be encoded using the TRANSITION mode of the ACELP core encoder.

[0395] If an attack is detected through the segment attack detection process of the additional advanced feature detection operation 351, as described in Section 3.1.2 above, then the index k of the attack is further examined. ata (Position). If the detected onset position is in the last subframe of frame n, the core encoder selection refiner 303 changes the core encoder technique to ACELP, for example, when the following conditions are met:

[0396]

[0397] In addition, when the core encoder technology was changed to ACELP, the frame type was set to TRANSITION, and a new onset "sign" f ata Set as follows

[0398] f ata =k ata +1

[0399] This means that the initial sound will be encoded using the TRANSITION mode of the ACELP core encoder.

[0400] If the detected attack location is not in the last subframe, but at least outside the first quarter of the first subframe, the core encoder selection remains unchanged, and the attack will be encoded using the GSC core encoder. Similar to the previous case, the new attack "signature" f... ata It can be set as follows:

[0401] f ata =k ata +1 if f no_GSC =1ANDTC cnt ≠1ANDk ata >4

[0402] parameter k ata The purpose is to reflect the position of the detected onset of a sound, hence the onset marker f. ata It's somewhat redundant. However, its use in this disclosure is for consistency with other documents and the source code of the IVAS codec.

[0403] Finally, the core encoder selection refiner 303 changes the frame type in the speech frame from GENERIC to TRANSITION, for which the ACELP core encoder technique was selected during the initial selection. This only occurs in active frames where the local VAD flag has been set to 1, and where the attack has been detected by the segment attack detection process of the additional advanced feature detection operation 351 described in Section 3.1.2, i.e., where k ata >0.

[0404] Then, the starting note is similar to the previous case. That is:

[0405] f ata =k ata +1

[0406] 4. Example configuration of hardware components

[0407] Figure 15 This is a simplified block diagram of an example configuration of the hardware components that form the aforementioned IVAS codec, including a two-stage speech / music classification device.

[0408] The IVAS codec, including a two-stage voice / music classification device, can be implemented as part of a mobile terminal, a portable media player, or any similar device. (The text repeats itself here: "Including a two-stage voice / music classification device...") Figure 15 The IVAS codec (marked as 1500) includes input 1502, output 1504, processor 1506, and memory 1508.

[0409] Input 1502 is configured to receive an input audio signal s(n), such as the left and right channels of an input stereo signal in digital or analog form, in the case of an encoder in an IVAS codec. In the case of an encoder in an IVAS codec, output 1504 is configured to provide an encoded multiplexed bitstream. Input 1502 and output 1504 can be implemented in a common module, such as a serial input / output device.

[0410] Processor 1506 is operatively connected to input 1502, output 1504, and memory 1508. Processor 1506 is implemented as one or more processors for running code instructions that support various elements and operations of the IVAS codec described above, including the two-stage speech / music classification apparatus and method as shown in the accompanying drawings and / or as described in this disclosure.

[0411] Memory 1508 may include non-transient memory for storing code instructions executable by processor 1506, specifically processor-readable memory storing non-transient instructions that, when executed, enable the processor to implement elements and operations of the IVAS codec, including a two-stage speech / music classification device and method. Memory 1508 may also include random access memory or buffers(s) for storing intermediate processing data from various functions performed by processor 1506.

[0412] Those skilled in the art will recognize that the description of the IVAS codec including a two-stage speech / music classification device and method is merely illustrative and not intended to be limiting in any way. Other embodiments will readily conceive of those skilled in the art upon receiving this disclosure. Furthermore, the disclosed IVAS codec including a two-stage speech / music classification device and method can be customized to provide valuable solutions to existing needs and problems in encoding and decoding sound (e.g., stereo).

[0413] For clarity, not all the conventional features of an implementation of the IVAS codec including a two-stage speech / music classification device and method are shown and described. It should be understood, of course, that the development of any such practical implementation of the IVAS codec, including a two-stage speech / music classification device and method, may require making many implementation-specific decisions to achieve the developer's specific goals, such as compliance with application, system, network, and business-related constraints, and these specific goals will vary depending on the implementation and the developer. Furthermore, it should be understood that the development work may be complex and time-consuming, but it remains a routine engineering task for those skilled in the art of sound processing who will benefit from this disclosure.

[0414] According to this disclosure, the elements, processing operations, and / or data structures described herein can be implemented using various types of operating systems, computing platforms, network devices, computer programs, and / or general-purpose machines. Furthermore, those skilled in the art will recognize that less general-purpose devices, such as hardwired devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., can also be used. Where a method comprising a series of operations and sub-operations is implemented by a processor, computer, or machine, and these operations and sub-operations can be stored as a series of non-transitory code instructions readable by the processor, computer, or machine, they can be stored on tangible and / or non-transitory media.

[0415] The elements and processing operations of the IVAS codec, including the two-stage speech / music classification device and method described herein, may include software, firmware, hardware, or any combination of software, firmware, or hardware suitable for the purposes described herein.

[0416] In the IVAS codec, which includes two-stage speech / music classification devices and methods, various processing operations and sub-operations can be executed in various orders, and some processing operations and sub-operations are optional.

[0417] Although the present disclosure has been described above by way of non-limiting illustrative embodiments, these embodiments may be modified freely within the scope of the appended claims without departing from the spirit and essence of the present disclosure.

[0418] References

[0419] This disclosure references the following sources, the entire contents of which are incorporated herein by reference:

[0420] [1] 3GPP TS 26.445, v.12.0.0, “Codecs for Enhanced Voice Services (EVS)”; detailed algorithm description”, September 2014.

[0421] [2] M. Neuendorf, M. Multrus, N. Rettelbach, G. Fuchs, J. Robillard, J. Lecompte, S. Wilde, S. Bayer, S. Disch, C. Helmrich, R. Lefevbre, P. Gournay, et al. "ISO / MPEG Unified Speech and Audio Coding Standards - Consistent High Quality for All Content Types and Bitrates", J. Audio Eng. Soc., Vol. 61, No. 12, pp. 956-977, December 2013.

[0422] [3] F. Baumgarte and C. Faller, “Binocular cue coding - Part 1: psychoacoustic basis and design principles”, IEEE Trans. Speech and Audio Processing, Vol. 11, pp. 509-519, November 2003.

[0423] [4] Tommy Vaillancourt, “Method and system for temporally downmixing stereo signals into main and sub-channels using long-term correlation difference between the left and right channels”, PCT application WO2017 / 049397A1.

[0424] [5] 3GPP SA4 Submission S4-170749 “New WID for EVS Codec Extensions for Immersive Voice and Audio Services”, SA4 Meeting #94, June 26-30, 2017 http: / / www.3gpp.org / ftp / tsg_sa / WG4_ CODEC / TSGS4_94 / Docs / S4-170749.zip

[0425] [6] V. Malenovsky, T. Vaillancourt, W. Zhe, K. Choo and V. Atti, “Two-stage speech / music classifier with decision smoothing and sharpening in EVS codec”, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2015, Brisbane, QLD, 2015, pp. 5718-5722.

[0426] [7] T. Vaillancourt and M. Jelinek, “Encoding a general audio signal with low bit rate and low latency”, U.S. Patent No. 9,015,038B2.

[0427] [8] KSRao and AKVuppala, Speech Processing in Mobile Environments, Appendix A: MFCC Features, Springer International Publishing, 2014

[0428] [9] Box, GEP and Cox, DR (1964). Transformation Analysis, Proceedings of the Royal Statistical Society, Series B, 26, 211-252.

[0429]

[10] D'Agostino, R. and Pearson, ES (1973), "Tests for deviation from normality", Biometrics, 60, 613-622.

[0430]

[11] D'Agostino, AJ Belanger and RBD'Agostino Jr., “Recommendations on the use of powerful and informative normality tests”, American Statistician, Vol. 44, pp. 316-321, 1990.

[0431]

[12] I. Jolliffe, Principal Component Analysis. New York: Springer Publishers, 2002.

Claims

1. A two-stage speech / music classification device for classifying input sound signals and selecting a core encoder for encoding the sound signals, comprising: The first stage is used to classify the input sound signal into one of a plurality of final categories, wherein the first stage for classifying the input sound signal includes an extractor of features of the input sound signal; as well as The second stage is used to extract additional high-level features of the input audio signal, and in response to the extracted additional high-level features and the final classification selected in the first stage, the core encoder for encoding the input audio signal is selected. The first stage for classifying the input audio signal includes an anomaly detector for: (a) detecting anomaly features in the features extracted in the first stage based on a histogram of the extracted features; (b) identifying the vector as an anomaly based on the number of anomaly features detected in the vector of features extracted in the first stage; and (c) instead of discarding the anomaly vector, replacing the anomaly features in the vector with feature values ​​obtained from at least one previous frame.

2. The two-stage speech / music classification device according to claim 1, wherein the first stage includes a syllable start / start detector in the input sound signal based on relative frame energy.

3. The two-stage speech / music classification device according to claim 2, wherein, The syllable start / start detector updates the cumulative sum of the differences between the relative energy of the input sound signal in the current frame and the relative energy of the input sound signal in the previous frame in each frame.

4. The two-stage speech / music classification device according to claim 3, wherein, To update the cumulative sum in the current frame, the syllable start / start detector adds (a) the cumulative sum updated in the previous frame and (b) the difference between the relative energy of the input sound signal in the current frame and the relative energy of the input sound signal in the previous frame.

5. The two-stage speech / music classification device according to claim 3 or 4, wherein, The syllable start / start detector updates the cumulative sum in the current frame only when the relative energy of the input sound signal in the current frame is greater than the relative energy of the input sound signal in the previous frame.

6. The two-stage speech / music classification device according to any one of claims 3 to 4, wherein, The syllable start / start detector uses the accumulated sum to update the counter of the syllable start / start frame, and wherein, in the ENTRY state of the input sound signal determined by the state machine, the counter is incremented in each frame if the accumulated sum is greater than a given value, otherwise it is reset.

7. The two-stage speech / music classification device according to any one of claims 3 to 4, wherein, If the cumulative sum is within a given range, the syllable start / start detector output is set to a first value of a binary flag to indicate that a syllable start / start has been detected; otherwise, the binary flag is set to a second value to indicate that no syllable start / start has been detected.

8. The two-stage speech / music classification device according to claim 1, wherein, The feature extractor in the first stage extracts Mel frequency cepstral coefficient features as features of the input sound signal.

9. The two-stage speech / music classification device according to claim 1, wherein the feature extractor of the first stage extracts at least one feature from the following group of features of the input sound signal: (a) Open-loop fundamental tone characteristics; (b) Voice measurement characteristics; (c) Characteristics related to line spectrum frequencies from LP analysis; (d) Characteristics related to residual energy from LP analysis; (e) Characteristics of short-term correlation plots; (f) Non-stationarity characteristics; (g) Mel frequency cepstral coefficient characteristics; (h) Power spectral difference characteristics; and (i) Spectral stationarity characteristics.

10. The two-stage speech / music classification device according to any one of claims 1, 8 and 9, wherein the outlier detector calculates a lower limit and an upper limit for each feature extracted in the first stage, compares the feature value with the lower limit and the upper limit, and marks features whose values ​​fall outside the range defined between the lower limit and the upper limit as outlier features.

11. The two-stage speech / music classification device according to claim 10, wherein, The outlier detector uses a normalized version of the histogram of the feature, an index of the frequency interval containing the maximum value of the histogram of the feature, and a threshold to calculate the lower and upper limits.

12. The two-stage speech / music classification device according to any one of claims 1, 8 and 9, wherein the outlier detector includes a counter of detected outlier features, and sets a flag to a given value to indicate that a vector of the features is an outlier when the number of detected outlier features is equal to or greater than a given value.

13. The two-stage speech / music classification device according to any one of claims 1, 8 and 9, wherein the first stage includes a filter for smoothing short-term vectors of the extracted features.

14. The two-stage speech / music classification device of claim 13, wherein the filter is an infinite impulse response filter using a forgetting factor.

15. The two-stage speech / music classification device according to claim 13, wherein, The filter does not perform feature vector smoothing in frames of the ENTRY or ACTIVE state of the input audio signal determined by the state machine, and where, when feature vector smoothing is not performed, the feature values ​​of the unfiltered vector are used.

16. The two-stage speech / music classification device according to any one of claims 1, 8 and 9, wherein the first stage includes a nonlinear feature vector transformer for transforming non-normal features extracted from the input sound signal into features having a normal shape.

17. The two-stage speech / music classification device according to claim 16, wherein the nonlinear feature vector transformer uses Box-Cox transformation to transform non-normal features into features with normal shapes.

18. The two-stage speech / music classification device of claim 17, wherein the Box-Cox transform performed by the nonlinear feature vector transformer uses a power transform with an exponent, and wherein different values ​​of the exponent define different Box-Cox transform curves, and wherein the nonlinear feature vector transformer selects the exponent value of the Box-Cox transform based on a normality test.

19. The two-stage speech / music classification device of claim 17, wherein the Box-Cox transformation performed by the nonlinear feature vector transformer uses bias to ensure that all input values ​​of the extracted features are positive.

20. The two-stage speech / music classification device of claim 18, wherein the normality test generates skewness and kurtosis measurements, and wherein the nonlinear feature vector transformer applies only the Box-Cox transformation to features that satisfy the conditions associated with the skewness and kurtosis measurements.

21. The two-stage speech / music classification device according to any one of claims 1, 8, and 9, wherein, The first stage includes a principal component analyzer to reduce the feature dimensionality of the sound signal and increase its classification discriminativeness. The principal component analyzer performs an orthogonal transformation to convert a set of potentially related features extracted from the input sound signal into a set of linearly unrelated variables that form the principal components.

22. The two-stage speech / music classification device according to claim 21, wherein, The principal component analyzer standardizes the vector by removing the mean of the vector of extracted features from the input audio signal and scaling the vector to unit variance.

23. The two-stage speech / music classification device according to claim 22, wherein, Principal Component Analysis (PCA) uses the following relation to transform eigenvectors: in It is a column eigenvector, and It is the matrix of principal component analysis loadings, and the superscript T indicates the vector transpose.

24. The two-stage speech / music classification device of claim 1, wherein the first stage includes a Gaussian mixture model (GMM) calculator to determine a first score proportional to the probability that a given feature vector extracted from the input sound signal is generated by a speech GMM, and a second score proportional to the probability that a given feature vector is generated by a music GMM, wherein the GMM calculator combines the first score and the second score by calculating the difference between these first scores and the second score to produce a difference score.

25. The two-stage speech / music classification device of claim 24, wherein a negative differential score indicates that the input sound signal is speech, and a positive differential score indicates that the input sound signal is music.

26. The two-stage speech / music classification device of claim 24, wherein the GMM calculator uses decision bias when calculating the difference between the first score and the second score.

27. The two-stage speech / music classification device according to claim 26, wherein, The GMM calculator subtracts the first score from the second score to calculate the difference between the first and second scores, wherein the decision bias is a non-negative value added to the difference.

28. The two-stage speech / music classification device of claim 26, wherein the GMM calculator predicts labels indicating that an input sound signal is speech, music, or noise in active frames of a training database, and wherein the GMM calculator uses the labels to find decision biases.

29. The two-stage speech / music classification device according to claim 26, wherein, The GMM calculator uses decision bias to calculate difference scores, where the difference scores have values ​​limited to a given range.

30. The two-stage speech / music classification device of claim 24, wherein the first stage includes an adaptive smoother for the difference scores from the GMM calculator.

31. The two-stage speech / music classification device of claim 30, wherein the adaptive smoother includes an infinite impulse response (IIR) filter using a forgetting factor.

32. The two-stage speech / music classification device of claim 31, wherein the adaptive smoother uses a first parameter that is linearly proportional to the relative energy of the current frame and constrained within a given interval to calculate the forgetting factor.

33. The two-stage speech / music classification device of claim 31, wherein the adaptive smoother uses a parameter proportional to the derivative of the difference fraction to calculate the forgetting factor.

34. The two-stage speech / music classification device of claim 33, wherein the adaptive smoother uses at least one operation from the following group of operations to calculate the parameters: (a) Calculate the short-term mean of the difference fractions; (b) If the difference score is less than 0 and less than the short-term mean, then set the parameter to 0; (c) In the first frame, if the difference score is less than 0 and less than the short-term mean, and the short-term mean is greater than 0, then the parameter is set to subtract the difference score; otherwise, the adaptive smoother increases the parameter. (d) If the difference fraction is not less than 0 and the short-term mean, then reset the parameter to 0; and (e) Finally, the parameters are mapped to a given interval.

35. The two-stage speech / music classification device of claim 31, wherein the adaptive smoother uses parameters that react to a sudden increase in the differential score indicating the beginning of a potential musical syllable to calculate a forgetting factor.

36. The two-stage speech / music classification device of claim 35, wherein the adaptive smoother uses at least one of the following operations to calculate the parameters: (a) Calculate the short-term mean of the difference fractions; (b) Set the parameter to 0, but modify the parameter if (a) the input audio signal is in the ACTIVE state determined by the state machine, (b) the short-term mean is greater than 0, and (c) the short-term mean in the current frame is greater than the short-term mean in the previous frame; (c) In the first frame, if (a) the input audio signal is in the ACTIVE state, (b) the short-term mean in the current frame is greater than 0, (c) the short-term mean in the current frame is greater than the short-term mean in the previous frame, and (d) the short-term mean in the previous frame is less than 0, then the parameter is set to subtract the short-term mean; otherwise, the adaptive smoother increases the parameter. (d) If (a) the input audio signal is not in the ACTIVE state, (b) the short-term mean is not greater than 0, and (c) the short-term mean in the current frame is not greater than the short-term mean in the previous frame, then reset the parameter to 0; and (e) Finally, map the third parameter to the given interval.

37. The two-stage speech / music classification device according to any one of claims 31 to 36, wherein, The adaptive smoother analyzes the long-term mean and long-term variance of the differenced fractions and performs at least one of the following operations: (a) If the state machine determines that the input sound signal is in the ENTRY state, then set the long-term mean and long-term variance of the differential fraction to 0. (b) Calculate the long-term mean-long-term variance ratio corresponding to the long-term standard deviation of the difference fraction; (c) Reduce the forgetting factor in frames with a long-term standard deviation greater than a given value; as well as (d) Limit the forgetting factor of the IIR filter to a given range.

38. The two-stage speech / music classification device according to claim 1, wherein, The multiple final categories include a first final category related to speech, a second final category related to music, and a third final category related to speech with background music.

39. The two-stage speech / music classification device according to any one of claims 24 to 36, wherein the first stage includes a category classifier of a dependent state that classifies an input sound signal into one of three final categories, the three final categories being SPEECH / NOISE, MUSIC, and UNCLEAR, wherein the final category UNCLEAR is associated with speech with background music.

40. The two-stage speech / music classification device of claim 39, wherein when the input sound signal is in the ENTRY state determined by the state machine in the current frame, the state-dependent category classifier selects one of three final categories, SPEECH / NOISE, MUSIC, and UNCLEAR, based on a weighted average of the differential scores calculated in the frames of the ENTRY states preceding the current frame.

41. The two-stage speech / music classification device according to claim 40, wherein, If the absolute frame energy is below a given value in the current frame, the state-dependent classifier will set the final classification to SPEECH / NOISE.

42. The two-stage speech / music classification device according to claim 40, wherein, If the weighted average of the differential scores in the frames of the ENTRY state of the input audio signal is less than 2.0, the state-dependent classifier will set the final classification to SPEECH / NOISE.

43. The two-stage speech / music classification device according to claim 40, wherein, If the weighted average of the difference scores in the frames of the ENTRY state of the input audio signal is higher than 2.0, the state-dependent classifier will set the final classification to MUSIC if the difference score in the current frame is higher than 2.0, and set it to UNCLEAR if the difference score in the current frame is not higher than 2.

0.

44. The two-stage speech / music classification device according to claim 40, wherein, In states other than the ENTRY of the input sound signal determined by the state machine, the state-dependent category classifier selects the final category SPEECH / NOISE, MUSIC, or UNCLEAR based on a smoothed version of the differential score and the final category SPEECH / NOISE, MUSIC, or UNCLEAR selected in the previous frame.

45. The two-stage speech / music classification device of claim 39, wherein the state-dependent category classifier first initializes the final classification in the current frame to the classification SPEECH / NOISE, MUSIC, or UNCLEAR set in the previous frame.

46. ​​The two-stage speech / music classification device according to claim 44, wherein the state-dependent category classifier first initializes the final classification in the current frame to the final classification SPEECH / NOISE, MUSIC, or UNCLEAR set in the previous frame, and wherein, In the current frame, in response to a smoothed differential score crossing a given threshold, the state-dependent classifier is converted from the final classification SPEECH / NOISE, MUSIC, or UNCLEAR set in the previous frame to another final classification.

47. The two-stage speech / music classification device according to claim 39, wherein, After a given number of frames, the state-dependent category classifier is switched from the final category SPEECH / NOISE, MUSIC, or UNCLEAR set in the previous frame to another of these categories.

48. The two-stage speech / music classification device according to claim 46, wherein, If the counter of the ACTIVE frame is below the first threshold, the cumulative sum of the differential frame energy is equal to zero, and the smooth differential score is greater than the second threshold, then the state-dependent classifier transitions from the final classification SPEECH / NOISE set in the previous frame to the final classification UNCLEAR.

49. The two-stage speech / music classification device according to claim 39, wherein, If the short pitch flag, a byproduct of open-loop pitch analysis of the input sound signal, is equal to a given value, and the smoothed version of the difference fraction is greater than a given threshold, then the state-dependent classifier transitions from the final classification SPEECH / NOISE set in the previous frame to the final classification UNCLEAR.

50. The two-stage speech / music classification device of claim 39, wherein the state-dependent category classifier does not perform a direct conversion between the SPEECH / NOISE and the final MUSIC classification.

51. The two-stage speech / music classification device according to any one of claims 8, 9, 24, and 38, wherein the second stage includes an extractor of the additional high-level features of the input sound signal in the current frame, wherein the additional high-level features include at least one feature from the following feature group: (a) The pitch of the input sound signal; (b) Long-term stability of the input audio signal, wherein an extractor with additional high-level features generates a flag indicating the long-term stability of the input audio signal; (c) The segment attack in the input audio signal, wherein the extractor with additional high-level features generates (a) an indicator of the position of the segment attack in the current frame of the input audio signal or (b) an indicator that the segment attack does not exist; and (d) Peak-to-average power ratio (PAPR) is a measurement of the spectral sharpness of an input audio signal, calculated from the power spectrum of the input audio signal.

52. The two-stage speech / music classification device of claim 1, wherein the second stage includes an extractor of the additional high-level features of the input sound signal in the current frame, wherein the additional high-level features include at least one feature from the following feature group: (a) The pitch of the input sound signal; (b) Long-term stability of the input audio signal, wherein an extractor with additional high-level features generates a flag indicating the long-term stability of the input audio signal; (c) A segment attack in the input audio signal, wherein an extractor with additional high-level features generates an indicator of (a) the position of the segment attack in the current frame of the input audio signal or (b) the absence of the segment attack; and (d) Peak-to-average power ratio (PAPR) is a measurement of the spectral sharpness of an input audio signal, calculated from the power spectrum of the input audio signal.

53. The two-stage speech / music classification device according to claim 52, wherein the pitch of the input sound signal is represented by a pitch marker that reflects both spectral stability and harmony of the input sound signal in the lower frequency range up to a given frequency.

54. The two-stage speech / music classification device of claim 53, wherein an extractor of additional high-level features uses a correlation map to calculate pitch signatures, the correlation map being formed in a plurality of first frequency intervals in the lower frequency range of the residual energy spectrum of the input sound signal, and calculated in segments of the residual energy spectrum where peaks exist.

55. The two-stage speech / music classification device of claim 54, wherein the extractor of additional high-level features applies smoothing of the correlation graph and calculates a weighted sum of the correlation graphs across frequency intervals in the lower frequency range of the input sound signal in the current frame to produce a single digit.

56. The two-stage speech / music classification device of claim 55, wherein the extractor of additional high-level features sets a pitch marker by comparing the individual digit with an adaptive threshold.

57. The two-stage speech / music classification device according to any one of claims 1, 8, 9, 24, 38 and 52, wherein the second stage includes a core encoder initial selector for making an initial selection of the core encoder using (a) relative frame energy, (b) the final classification of the input sound signal in the first stage, and (c) the extracted additional high-level features.

58. The two-stage speech / music classification device of claim 56, wherein the final classification includes SPEECH / NOISE, MUSIC, and UNCLEAR, wherein the final classification UNCLEAR is associated with speech with background music, and wherein the second stage includes a core encoder initial selector for initial selection of the core encoder using the following conditions: (a) If the relative frame energy is higher than the first value, the peak-to-average power ratio of the spectrum is higher than the second value, and the single number is higher than the adaptive threshold, then the TCX core encoder is initially selected. (b) If condition (a) is not met, and the final classification of the input audio signal in the first stage is SPEECH / NOISE, then the ACELP core encoder is initially selected. (c) If conditions (a) and (b) are not met, and the final classification of the input audio signal in the first stage is UNCLEAR, then the GSC core encoder is initially selected; and (d) If conditions (a), (b) and (c) are not met, then the TCX core encoder is initially selected.

59. The two-stage speech / music classification device according to any one of claims 24 to 36, wherein the second stage includes a core encoder initial selector for initial selection of the core encoder in response to extracted additional high-level features and a final classification selected in the first stage, and includes a refiner for the initial core encoder selection if the core encoder initial selector initially selects the GSC core encoder.

60. The two-stage speech / music classification device according to claim 59, wherein, If (a) the ratio of energy in multiple first frequency intervals of a signal segment to the total energy of the signal segment is lower than a first value, and (b) the short-term mean of the differential fraction is higher than a second value, then the refiner of the initial core encoder selection will change the initial selection of the GSC core encoder to the selection of the ACELP core encoder.

61. The two-stage speech / music classification device according to claim 59, wherein, For an input audio signal with a short and stable pitch period, the refiner of the initial core encoder selection changes the initial selection of the GSC core encoder to (a) selecting the ACELP core encoder if the smoothed version of the differential fraction is below a given value, or (b) selecting the TCX core encoder if the smoothed differential fraction is greater than or equal to a given value.

62. The two-stage speech / music classification device of claim 59, wherein the refiner for the initial core encoder selection changes the initial selection of the GSC core encoder to (a) the selection of the TCX core encoder in response to the long-term stability of the input sound signal, and (b) an open-loop pitch greater than a given value.

63. The two-stage speech / music classification device according to claim 59, wherein, Assuming the indicator for enabling core encoder selection change has a first value and the transition frame counter has a second value, if a segment attack is detected in the input audio signal, the refiner of the initial core encoder selection will change the initial selection of the GSC core encoder to the selection of the ACELP core encoder.

64. The two-stage speech / music classification device according to claim 59, wherein, Assuming the indicator for enabling the change of core encoder selection has a first value, the transition frame counter does not have a second value, and the indicator that identifies the segment corresponding to the start position in the current frame is greater than a third value, if the start of a segment is detected in the input audio signal, the refiner of the initial core encoder selection will change the initial selection of the GSC core encoder to the selection of the ACELP core encoder.

65. A two-stage speech / music classification device for classifying input sound signals and selecting a core encoder for encoding the sound signals, comprising: At least one processor; as well as A memory coupled to the processor and storing non-transitory instructions that, when executed, cause the processor to implement: The first stage is used to classify the input sound signal into one of a plurality of final categories, wherein the first stage for classifying the input sound signal includes an extractor of features of the input sound signal; as well as The second stage is used to extract additional high-level features of the input audio signal, and in response to the extracted additional high-level features and the final classification selected in the first stage, the core encoder for encoding the input audio signal is selected. The first stage for classifying the input audio signal includes an anomaly detector for: (a) detecting anomaly features in the features extracted in the first stage based on a histogram of the extracted features; (b) identifying the vector as an anomaly based on the number of anomaly features detected in the vector of features extracted in the first stage; and (c) instead of discarding the anomaly vector, replacing the anomaly features in the vector with feature values ​​obtained from at least one previous frame.

66. A two-stage speech / music classification device for classifying input sound signals and selecting a core encoder for encoding the sound signals, comprising: At least one processor; as well as A memory coupled to the processor and storing non-transitory instructions, which, when executed, cause the processor to: In the first stage, the input sound signal is classified into one of several final categories, wherein the first stage for classifying the input sound signal extracts features of the input sound signal; as well as In the second stage, additional high-level features of the input audio signal are extracted, and in response to the extracted additional high-level features and the final classification selected in the first stage, a core encoder for encoding the input audio signal is selected. The first stage for classifying the input audio signal includes (a) detecting outlier features based on the histogram of the extracted features in the features extracted in the first stage, (b) identifying the vector as an outlier based on the number of outlier features detected in the vector of features extracted in the first stage, and (c) instead of discarding the outlier vector, replacing the outlier features in the vector with feature values ​​obtained from at least one previous frame.

67. A two-stage speech / music classification method for classifying input sound signals and selecting a core encoder for encoding the sound signals, comprising: In the first stage, the input sound signal is classified into one of several final categories, wherein classifying the input sound signal in the first stage includes extracting features of the input sound signal; as well as In the second stage, additional high-level features of the input audio signal are extracted, and in response to the extracted additional high-level features and the final classification selected in the first stage, a core encoder for encoding the input audio signal is selected. In the first stage, classifying the input audio signal includes: (a) detecting outlier features based on the histogram of the extracted features in the features extracted in the first stage; (b) identifying the vector as an outlier based on the number of outlier features detected in the vector of features extracted in the first stage; and (c) instead of discarding the outlier vector, replacing the outlier features in the vector with feature values ​​obtained from at least one previous frame.

68. The two-stage speech / music classification method according to claim 67, comprising, in a first stage, detecting the first / starting syllable of an input sound signal based on relative frame energy.

69. The two-stage speech / music classification method of claim 68, wherein detecting the first / second syllable of an input sound signal comprises updating the cumulative sum of the differences between the relative energy of the input sound signal in the current frame and the relative energy of the input sound signal in the previous frame in each frame.

70. The two-stage speech / music classification method of claim 69, wherein detecting the first / second syllable of an input sound signal comprises, in order to update the cumulative sum in the current frame, adding (a) the cumulative sum updated in the previous frame and (b) the difference between the relative energy of the input sound signal in the current frame and the relative energy of the input sound signal in the previous frame.

71. The two-stage speech / music classification method according to claim 69 or 70, wherein detecting the first / second syllable of an input sound signal includes updating the cumulative sum in the current frame only when the relative energy of the input sound signal in the current frame is greater than the relative energy of the input sound signal in the previous frame.

72. The two-stage speech / music classification method according to claim 69 or 70, wherein detecting the first / second syllable of an input sound signal includes updating a counter for the first / second syllable frame using a cumulative sum, and in the ENTRY state of the input sound signal determined by the state machine, if the cumulative sum is greater than a given value, the counter is incremented in each frame; otherwise, the cumulative sum is reset.

73. The two-stage speech / music classification method according to claim 69 or 70, wherein detecting the first / starting syllable in the input sound signal includes generating a binary flag, wherein if the cumulative sum is within a given range, the binary flag is set to a first value to indicate that the first / starting syllable has been detected, otherwise, the binary flag is set to a second value to indicate that the first / starting syllable has not been detected.

74. The two-stage speech / music classification method according to claim 67, comprising, in a first stage, extracting features of the input sound signal including Mel-frequency cepstral coefficient features.

75. The two-stage speech / music classification method according to claim 67, comprising, in the first stage, extracting at least one feature from the following feature group of the input sound signal: (a) Open-loop fundamental tone characteristics; (b) Voice measurement characteristics; (c) Characteristics related to line spectrum frequencies from LP analysis; (d) Characteristics related to residual energy from LP analysis; (e) Characteristics of short-term correlation plots; (f) Non-stationarity characteristics; (g) Mel frequency cepstral coefficient characteristics; (h) Power spectral difference characteristics; and (i) Spectral stationarity characteristics.

76. The two-stage speech / music classification method according to any one of claims 67, 74 and 75, wherein detecting outlier features includes calculating a lower limit and an upper limit for each feature, comparing the feature value with the lower limit and the upper limit, and marking features whose values ​​fall outside the range defined between the lower limit and the upper limit as outlier features.

77. The two-stage speech / music classification method according to claim 76, wherein detecting outlier features includes calculating a lower limit and an upper limit using a normalized version of the feature histogram, an index of the frequency interval containing the maximum value of the feature histogram, and a threshold.

78. The two-stage speech / music classification method according to any one of claims 67, 74 and 75, wherein detecting outlier features includes counting the detected outlier features, and setting a flag to a given value to indicate that the feature vector is an outlier when the number of detected outlier features is equal to or greater than a given value.

79. The two-stage speech / music classification method according to any one of claims 67, 74 and 75, comprising, in a first stage, smoothing the short-term vectors of the extracted features.

80. The two-stage speech / music classification method according to claim 79, wherein smoothing the short-term vector of the extracted features includes filtering the short-term vector of the extracted features using an infinite impulse response filter with a forgetting factor.

81. The two-stage speech / music classification method of claim 79, comprising not performing feature vector smoothing in frames of the ENTRY or ACTIVE state of the input sound signal determined by the state machine, and using the feature values ​​of the unfiltered vector when feature vector smoothing is not performed.

82. The two-stage speech / music classification method according to any one of claims 67, 74 and 75, comprising, in a first stage, nonlinearly transforming the non-normal features extracted from the input sound signal into features having a normal shape.

83. The two-stage speech / music classification method according to claim 82, wherein the nonlinear transformation includes using Box-Cox transformation to transform non-normal features into features with a normal shape.

84. The two-stage speech / music classification method according to claim 83, wherein the Box-Cox transform includes using a power transform with an exponent, wherein different values ​​of the exponent define different Box-Cox transform curves, and the exponent value of the Box-Cox transform is selected based on a normality test.

85. The two-stage speech / music classification method of claim 83, wherein the Box-Cox transform includes using bias to ensure that all input values ​​of the extracted features are positive.

86. The two-stage speech / music classification method of claim 84, wherein the normality test produces skewness and kurtosis measurements, and wherein the Box-Cox transform is applied only to features that satisfy conditions associated with the skewness and kurtosis measurements.

87. The two-stage speech / music classification method according to any one of claims 67, 74 and 75, comprising, in a first stage, analyzing principal components to reduce the feature dimensionality of the sound signal and increase the discriminative power of the sound signal classification, wherein the principal component analysis includes an orthogonal transformation to convert a set of potentially correlated features extracted from the input sound signal into a set of linearly uncorrelated variables forming the principal components.

88. The two-stage speech / music classification method according to claim 87, wherein the principal component analysis includes standardizing the vector by removing the mean of the vector of extracted features of the input sound signal and scaling the vector to unit variance.

89. The two-stage speech / music classification method according to claim 88, wherein the principal component analysis includes transforming the feature vector using the following relations: in It is a column eigenvector, and It is the matrix of principal component analysis loadings, and the superscript T indicates the vector transpose.

90. The two-stage speech / music classification method of claim 67, comprising, in a first stage, a Gaussian mixture model (GMM) is computed to determine a first score proportional to the probability that a given feature vector extracted from an input sound signal is generated by a speech GMM, and a second score proportional to the probability that a given feature vector is generated by a music GMM, wherein the GMM computation includes combining the first score and the second score by computing the difference between these first scores and the second score to produce a difference score.

91. The two-stage speech / music classification method according to claim 90, wherein a negative difference score indicates that the input sound signal is speech, and a positive difference score indicates that the input sound signal is music.

92. The two-stage speech / music classification method of claim 90, wherein the GMM calculation includes using decision bias when calculating the difference between the first score and the second score.

93. The two-stage speech / music classification method of claim 92, wherein the GMM calculation includes subtracting the first score from the second score to calculate the difference between the first score and the second score, and wherein the decision bias is a non-negative value added to the difference.

94. The two-stage speech / music classification method of claim 92, wherein the GMM computation predicts labels in active frames of the training database that indicate that the input sound signal is speech, music, or noise, and wherein the GMM computation includes using the labels to find decision biases.

95. The two-stage speech / music classification method of claim 92, wherein the GMM calculation includes calculating a difference score using decision bias, wherein the difference score has a value limited to a given range.

96. The two-stage speech / music classification method according to claim 90, comprising adaptive smoothing of the difference scores in the first stage.

97. The two-stage speech / music classification method of claim 96, wherein adaptive smoothing includes using an infinite impulse response (IIR) filter itself with a forgetting factor.

98. The two-stage speech / music classification method of claim 97, wherein adaptive smoothing includes calculating a forgetting factor using parameters that are linearly proportional to the relative energy of the current frame and constrained within a given interval.

99. The two-stage speech / music classification method of claim 97, wherein adaptive smoothing includes calculating a forgetting factor using a parameter proportional to the derivative of the difference fraction.

100. The two-stage speech / music classification method of claim 99, wherein adaptive smoothing comprises calculating the parameters using at least one of the following groups of operations: (a) Calculate the short-term mean of the difference fractions; (b) If the difference score is less than 0 and less than the short-term mean, then set the parameter to 0; (c) In the first frame, if the difference score is less than 0 and less than the short-term mean, and the short-term mean is greater than 0, then the parameter is set to subtract the difference score; otherwise, the adaptive smoothing increases the parameter. (d) If the difference fraction is not less than 0 and the short-term mean, then reset the parameter to 0; and (e) Finally, the parameters are mapped to a given interval.

101. The two-stage speech / music classification method of claim 97, wherein adaptive smoothing includes calculating a forgetting factor using parameters that respond to a sudden increase in the differential score indicating the beginning of a potential musical syllable.

102. The two-stage speech / music classification method of claim 101, wherein the adaptive smoothing comprises calculating the parameters using at least one of the following operations: (a) Calculate the short-term mean of the difference fractions; (b) Set the parameter to 0, but modify the parameter if (a) the input audio signal is in the ACTIVE state determined by the state machine, (b) the short-term mean is greater than 0, and (c) the short-term mean in the current frame is greater than the short-term mean in the previous frame; (c) In the first frame, if (a) the input audio signal is in the ACTIVE state, (b) the short-term mean in the current frame is greater than 0, (c) the short-term mean in the current frame is greater than the short-term mean in the previous frame, and (d) the short-term mean in the previous frame is less than 0, then the parameter is set to subtract the short-term mean; otherwise, the parameter is adaptively and smoothly increased. (d) If (a) the input audio signal is not in the ACTIVE state, (b) the short-term mean is not greater than 0, and (c) the short-term mean in the current frame is not greater than the short-term mean in the previous frame, then reset the parameter to 0; and (e) Finally, the parameters are mapped to a given interval.

103. The two-stage speech / music classification method according to any one of claims 97 to 102, wherein, Adaptive smoothing involves analyzing the long-term mean and long-term variance of the differenced fractions and performing at least one of the following operations: (a) If the state machine determines that the input sound signal is in the ENTRY state, then set the long-term mean and long-term variance of the differential fraction to 0. (b) Calculate the long-term mean-long-term variance ratio corresponding to the long-term standard deviation of the difference fraction; (c) Reduce the forgetting factor in frames with a long-term standard deviation greater than a given value; as well as (d) Limit the forgetting factor of the IIR filter to a given range.

104. The two-stage speech / music classification method according to claim 67, wherein the plurality of final classifications includes a first final classification related to speech, a second final classification related to music, and a third final classification related to speech with background music.

105. The two-stage speech / music classification method according to any one of claims 90 to 102, comprising, in a first stage, classifying the category of the input sound signal dependent state into one of three final categories, including SPEECH / NOISE, MUSIC, and UNCLEAR, wherein the final category UNCLEAR is associated with speech with background music.

106. The two-stage speech / music classification method according to claim 105, wherein when the input sound signal is in the ENTRY state determined by the state machine in the current frame, the state-dependent category classification includes selecting one of three final categories SPEECH / NOISE, MUSIC, and UNCLEAR based on a weighted average of the differential scores calculated in frames with ENTRY states prior to the current frame.

107. The two-stage speech / music classification method according to claim 106, wherein, If the absolute frame energy is below a given value in the current frame, the state-dependent category classification includes setting the final classification to SPEECH / NOISE.

108. The two-stage speech / music classification method according to claim 106, wherein, If the weighted average of the differential scores in the frames of the ENTRY state of the input audio signal is less than 2.0, then the state-dependent category classification includes setting the final classification to SPEECH / NOISE.

109. The two-stage speech / music classification method according to claim 106, wherein, If the weighted average of the difference scores in the frames of the ENTRY state of the input audio signal is higher than 2.0, the state-dependent category classification includes setting the final category to MUSIC if the difference score in the current frame is higher than 2.0, and setting the final category to UNCLEAR if the difference score in the current frame is not higher than 2.

0.

110. The two-stage speech / music classification method according to claim 106, wherein, In states other than the ENTRY of the input sound signal determined by the state machine, the state-dependent category classification includes a smoothed version based on the differential fraction and the final category SPEECH / NOISE, MUSIC, or UNCLEAR selected in the previous frame to select the final category SPEECH / NOISE, MUSIC, or UNCLEAR.

111. The two-stage speech / music classification method according to claim 105, wherein the state-dependent category classification includes first initializing the final category in the current frame to the category SPEECH / NOISE, MUSIC, or UNCLEAR set in the previous frame.

112. The two-stage speech / music classification method according to claim 110, wherein the state-dependent category classification includes first initializing the final category in the current frame to the category SPEECH / NOISE, MUSIC, or UNCLEAR set in the previous frame, and wherein, In the current frame, the state-dependent category classification includes a smooth differential score crossing a given threshold, which is converted from the final category SPEECH / NOISE, MUSIC, or UNCLEAR set in the previous frame to another final category.

113. The two-stage speech / music classification method according to claim 105, wherein, The state-dependent category classification includes, after a given number of frames, transitioning from the final category SPEECH / NOISE, MUSIC, or UNCLEAR set in the previous frame to another of these categories.

114. The two-stage speech / music classification method according to claim 112, wherein, The state-dependent category classification includes: if the counter of the ACTIVE frame is below the first threshold, the cumulative sum of the differential frame energy is equal to zero, and the smooth differential score is greater than the second threshold, then the final classification SPEECH / NOISE set in the previous frame is switched to the final classification UNCLEAR.

115. The two-stage speech / music classification method according to claim 105, wherein, The category classification based on the dependent state includes: if the short pitch flag, which is a byproduct of the open-loop pitch analysis of the input sound signal, is equal to a given value, and the smoothed version of the difference fraction is greater than a given threshold, then the final classification SPEECH / NOISE set in the previous frame is switched to the final classification UNCLEAR.

116. The two-stage speech / music classification method of claim 105, wherein the state-dependent category classification includes not performing a direct conversion between the SPEECH / NOISE and the final MUSIC classification.

117. The two-stage speech / music classification method according to any one of claims 74, 75, 90, and 104, comprising, in a second stage, extracting the additional high-level features of the input sound signal in the current frame, wherein the additional high-level features include the following feature groups At least one feature: (a) The pitch of the input sound signal; (b) Long-term stability of the input audio signal, wherein extracting the additional high-level features includes generating a flag indicating the long-term stability of the input audio signal; (c) The segment attack in the input audio signal, wherein extracting the additional high-level features includes generating an indicator that (a) the position of the segment attack in the current frame of the input audio signal or (b) the segment attack does not exist; and (d) The peak-to-average power ratio (PAPR) forms a measure of the spectral sharpness of the input audio signal, wherein extracting the additional advanced features includes calculating the PAPR from the power spectrum of the input audio signal.

118. The two-stage speech / music classification method according to claim 67, comprising, in a second stage, extracting the additional high-level features of the input sound signal in the current frame, wherein the additional high-level features include the following feature groups At least one feature: (a) The pitch of the input sound signal; (b) Long-term stability of the input audio signal, wherein extracting the additional high-level features includes generating a flag indicating the long-term stability of the input audio signal; (c) The segment attack in the input audio signal, wherein extracting the additional high-level features includes generating an indicator that (a) the position of the segment attack in the current frame of the input audio signal or (b) the segment attack does not exist; and (d) The peak-to-average power ratio (PAPR) forms a measure of the spectral sharpness of the input audio signal, wherein extracting the additional advanced features includes calculating the PAPR from the power spectrum of the input audio signal.

119. The two-stage speech / music classification method according to claim 118, wherein extracting the pitch of the input sound signal includes representing the pitch by a pitch marker that reflects both the spectral stability and harmony of the input sound signal in a lower frequency range up to a given frequency.

120. The two-stage speech / music classification method of claim 119, wherein extracting pitch markers includes using a correlation map to calculate pitch markers, the correlation map being formed by measuring signal stability and consonance in a plurality of first frequency intervals in the lower frequency range of the residual energy spectrum of the input sound signal, and being calculated in segments of the residual energy spectrum where peaks exist.

121. The two-stage speech / music classification method of claim 120, wherein extracting pitch markers includes applying smoothing to a correlation map and calculating a weighted sum of correlation maps across frequency intervals in the lower frequency range of the input sound signal in the current frame to produce a single digit.

122. The two-stage speech / music classification method of claim 121, wherein extracting the pitch marker includes setting the pitch marker by comparing the single digit with an adaptive threshold.

123. The two-stage speech / music classification method according to any one of claims 67, 74, 75, 90, 104 and 118, comprising, in the second stage, using an initial selection of the core encoder as follows: (a) relative frame energy, (b) the final classification of the input sound signal in the first stage, and (c) additional high-level features extracted.

124. The two-stage speech / music classification method according to claim 122, wherein, The final classification includes SPEECH / NOISE, MUSIC, and UNCLEAR, where the final classification UNCLEAR is associated with speech with background music, and in the second stage, the two-stage speech / music classification method includes initial selection of the core encoder using the following criteria: (a) If the relative frame energy is higher than the first value, the peak-to-average power ratio of the spectrum is higher than the second value, and the single number is higher than the adaptive threshold, then the TCX core encoder is initially selected. (b) If condition (a) is not met, and the final classification of the input audio signal in the first stage is SPEECH / NOISE, then the ACELP core encoder is initially selected. (c) If conditions (a) and (b) are not met, and the final classification of the input audio signal in the first stage is UNCLEAR, then the GSC core encoder is initially selected; and (d) If conditions (a), (b) and (c) are not met, then the TCX core encoder is initially selected.

125. The two-stage speech / music classification method according to any one of claims 90 to 102, comprising, in the second stage, initially selecting a core encoder in response to the extracted additional high-level features and the final classification selected in the first stage, and refining the initial core encoder selection if the initial core encoder selection initially selected the GSC core encoder.

126. The two-stage speech / music classification method according to claim 125, wherein, Refining the initial core encoder selection includes: if (a) the ratio of energy in multiple first frequency intervals of a signal segment to the total energy of the signal segment is lower than a first value, and (b) the short-term mean of the differential fraction is higher than a second value, then the initial selection of the GSC core encoder is changed to the selection of the ACELP core encoder.

127. The two-stage speech / music classification method of claim 125, wherein refining the initial core encoder selection includes, for an input sound signal having a short and stable pitch period, changing the initial selection of the GSC core encoder to (a) selecting the ACELP core encoder if the smoothed version of the difference fraction is below a given value, or (b) selecting the TCX core encoder if the smoothed difference fraction is greater than or equal to a given value.

128. The two-stage speech / music classification method according to claim 125, wherein refining the initial core encoder selection includes changing the initial selection of the GSC core encoder to (a) the selection of the TCX core encoder in response to the long-term stability of the input sound signal, and (b) an open-loop pitch greater than a given value.

129. The two-stage speech / music classification method according to claim 125, wherein refining the initial core encoder selection includes, assuming that the indicator for enabling the change of core encoder selection has a first value and the transition frame counter has a second value, changing the initial selection of the GSC core encoder to the selection of the ACELP core encoder if a segment start is detected in the input audio signal.

130. The two-stage speech / music classification method according to claim 125, wherein refining the selection of the initial core encoder includes: Assuming the indicator for enabling the change of core encoder selection has a first value, the transition frame counter does not have a second value, and the indicator that identifies the segment corresponding to the start position in the current frame is greater than a third value, if the start of a segment is detected in the input audio signal, the initial selection of the GSC core encoder is changed to the selection of the ACELP core encoder.

Images