An audio compensation method and device, electronic equipment and storage medium

Abstract

The method comprises the following steps: receiving an audio signal, extracting and storing feature parameters of an audio frame in the audio signal; wherein the feature parameters comprise a spectrum feature and a pitch cycle sequence; the pitch cycle sequence is sequentially composed of pitch cycles of sub-frames in the audio frame; in response to detecting that the audio frame is lost, predicting the feature parameters of the lost frame according to the feature parameters of a previous audio frame of the lost frame; and determining an audio signal of the lost frame according to the feature parameters of the lost frame. The method can realize audio compensation at a receiving end and ensure audio playing quality.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an audio compensation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In network transmission of audio signals, signal loss can occur due to poor network conditions. While strategies such as retransmission and redundant transmission can mitigate some of these issues, these methods are unsuitable for extremely poor network conditions and can increase network load, leading to increased bandwidth consumption at the sending end. Therefore, there is an urgent need for audio compensation methods applicable to the receiving end to ensure audio playback quality. Summary of the Invention

[0003] This disclosure provides an audio compensation method, apparatus, electronic device, and storage medium that can achieve audio compensation at the receiving end and ensure audio playback quality.

[0004] In a first aspect, embodiments of this disclosure provide an audio compensation method, including:

[0005] The system receives an audio signal and extracts and stores feature parameters from the audio frames in the audio signal; wherein the feature parameters include spectral features and a pitch period sequence; the pitch period sequence is formed by the pitch periods of the subframes in the audio frame in sequence;

[0006] In response to detecting the loss of the audio frame, the feature parameters of the lost frame are predicted based on the feature parameters of the audio frame preceding the lost frame;

[0007] The audio signal of the lost frame is determined based on the characteristic parameters of the lost frame.

[0008] Secondly, embodiments of this disclosure also provide an audio compensation device, comprising:

[0009] The feature processing module is used to receive audio signals and extract and store feature parameters of audio frames in the audio signals; wherein, the feature parameters include spectral features and pitch period sequence; the pitch period sequence is formed by the pitch periods of the subframes in the audio frame in sequence;

[0010] A feature prediction module is configured to, in response to detecting the loss of the audio frame, predict the feature parameters of the lost frame based on the feature parameters of the preceding audio frame of the lost frame;

[0011] A signal determination module is used to determine the audio signal of the lost frame based on the characteristic parameters of the lost frame.

[0012] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0013] One or more processors;

[0014] Storage device for storing one or more programs.

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio compensation method as described in any of the embodiments of this disclosure.

[0016] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the audio compensation method as described in any of the embodiments of this disclosure.

[0017] Fifthly, this disclosure also provides a computer program product, characterized in that the computer program product includes a computer program, which, when executed by a processor, implements the audio compensation method as described in any of the embodiments of this disclosure.

[0018] In the technical solution of this disclosure embodiment, an audio signal can be received, and feature parameters of audio frames in the audio signal can be extracted and stored. The feature parameters include spectral features and a pitch period sequence. The pitch period sequence is formed by the pitch periods of subframes in the audio frame in sequence. In response to detecting audio frame loss, feature parameters of the lost frame are predicted based on the feature parameters of the preceding audio frames. The audio signal of the lost frame is determined based on the feature parameters of the lost frame. The receiving end of the audio signal can predict the feature parameters of the lost frame based on the feature parameters of the audio signal, thereby achieving compensation for the audio signal of the lost frame. Furthermore, by predicting the pitch period of subframes in the audio signal, finer-grained fundamental frequency information can be provided, making the compensated audio signal more realistic and natural, and ensuring audio playback quality. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 This is a schematic flowchart of an audio compensation method provided in an embodiment of the present disclosure;

[0021] Figure 2 This is a data flow diagram of an audio compensation method provided in an embodiment of the present disclosure;

[0022] Figure 3This diagram illustrates the construction and testing process of a prediction model in an audio compensation method provided in this embodiment of the present disclosure.

[0023] Figure 4 This is a schematic diagram of forward rolling prediction in an audio compensation method provided in an embodiment of the present disclosure;

[0024] Figure 5 This is a schematic diagram of a sequence-to-sequence model in an audio compensation method provided in an embodiment of this disclosure;

[0025] Figure 6 This is a schematic diagram illustrating the sequence-to-sequence model construction process in an audio compensation method provided in this embodiment of the present disclosure;

[0026] Figure 7 This is a schematic diagram illustrating the generation of a pitch period correction sequence based on context information in an audio compensation method provided in this embodiment of the present disclosure.

[0027] Figure 8 This is a schematic diagram of the structure of an audio compensation device provided in an embodiment of the present disclosure;

[0028] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0030] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0033] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0034] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0035] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0036] Figure 1 This is a schematic flowchart illustrating an audio compensation method provided in an embodiment of this disclosure. This embodiment is applicable to situations where audio signal compensation is performed at an audio signal receiving end. The method can be executed by an audio compensation device, which can be implemented in software and / or hardware and can be configured in an electronic device, such as an audio signal receiving device.

[0037] like Figure 1 As shown, the audio compensation method provided in this embodiment may include:

[0038] S110. Receive an audio signal and extract and store the feature parameters of the audio frames in the audio signal; wherein, the feature parameters include spectral features and pitch period sequence; the pitch period sequence is composed of the pitch periods of the subframes in the audio frame in sequence.

[0039] In this embodiment, the audio signal receiver can store the received audio signal in a buffer register for buffering, and can read the audio signal from it according to the storage sequence for signal processing and audio playback. The signal within each preset time interval of the audio signal can be called an audio frame, and the preset time is related to the encoding format of the audio signal. The audio frame can be pre-divided into a preset number of subframes; the preset number is at least two, and can be set according to specific business scenarios (e.g., four, etc.); the subframe division method can also be set according to specific business scenarios, for example, the audio frame can be divided evenly.

[0040] The receiving end, upon receiving the audio signal, can extract feature parameters from the audio frames. These feature parameters can include spectral features, which may include, but are not limited to, spectral envelope, harmonic amplitude, and phase information. The audio frames can be converted from time-domain to frequency-domain signals using time-domain to frequency-domain algorithms (such as short-time Fourier transform), allowing for the extraction of spectral features. Furthermore, the feature parameters can also include a pitch period sequence. This pitch period sequence can be constructed sequentially from the pitch periods of the subframes within the audio frame. Existing pitch period extraction methods can be used to extract the pitch periods of the subframes. The pitch periods can then be concatenated according to the temporal order of the subframes to form the pitch period sequence of the audio frame to which the subframe belongs.

[0041] For example, the pitch period of a subframe can be extracted using an autocorrelation method, such as defining an autocorrelation function R[k]: Determine the location of the first positive peak of the autocorrelation function R[k]: Wherein, the fundamental period T0 is the delay corresponding to the peak position: T0 = peak; correspondingly, the fundamental frequency is f0 = F. s / T0, where F s Indicates the sampling frequency.

[0042] For example, Figure 2 This is a schematic block diagram of the data flow for an audio compensation method provided in an embodiment of this disclosure. See also... Figure 2 After receiving the audio signal, the receiving end can detect whether there are any missing audio frames in the audio signal using existing detection algorithms before playing the audio signal. For example, it can determine whether audio frames are missing by detecting whether packet identifiers or audio frame identifiers are continuous. If no audio frames are missing, the feature parameters of the audio frames can be extracted and stored in a preset storage space.

[0043] S120, In response to detecting the loss of an audio frame, predict the feature parameters of the lost frame based on the feature parameters of the preceding audio frame of the lost frame.

[0044] See you again Figure 2 If audio frame loss is detected in the audio signal, the audio signal of the lost frame can be predicted based on frame-level historical information. In this embodiment, the feature parameters of the lost frame can be predicted based on the feature parameters of the preceding audio frame. The preceding audio frame of the lost frame can include at least one audio frame that is temporally preceding and adjacent to the lost frame.

[0045] The prediction model can predict the spectral characteristics of the lost frame based on the spectral characteristics of the preceding audio frame, or it can predict the spectral characteristics based on the pitch period sequence. The spectral characteristic prediction model and the pitch period sequence prediction model can be the same or different models.

[0046] For example, Figure 3 This diagram illustrates the construction and testing process of a prediction model in an audio compensation method provided in this embodiment of the present disclosure. Figure 3 The construction and testing process shown can represent both the construction and testing process of a prediction model for spectral features and a prediction model for pitch period sequences. See also Figure 3 The process of building a prediction model may include: extracting historical audio frames from sample frames (such as...) Figure 3 The feature parameters of the TP frame to T-1 frame in the prediction model are input into the neural network of the prediction model; the target frame (e.g., ...) is output by the prediction model. Figure 3 The prediction model is constructed by predicting the feature parameters of the T-frame (in the target frame); and by constructing a loss function based on the predicted and true feature parameters of the target frame to adjust the weights of the neural network. Correspondingly, the testing process of the prediction model can include: inputting the feature parameters of the preceding audio frame of the lost frame into the prediction model; and predicting the feature parameters of the lost frame using the prediction model.

[0047] S130. Determine the audio signal of the lost frame based on the characteristic parameters of the lost frame.

[0048] In this embodiment, existing techniques for generating time-domain audio signals by combining frequency domain features and pitch period can be used to determine the audio signal of the lost frame based on its spectral features and pitch period sequence. For example, a preset harmonic model can be used to re-superimpose each harmonic and other component in the spectrum corresponding to the spectral features in the time domain to obtain a time-domain waveform signal. During the spectral superposition process, random noise or transient signal spectra can be added using a noise model to assist in generating a more realistic time-domain waveform signal. Furthermore, a filter model (such as a channel filter) can be used to filter the pitch and excitation signal according to the pitch period sequence to more closely approximate the original timbre. See also... Figure 2 If no audio frame loss is detected in the audio signal, the audio signal can be directly obtained from the buffer register and processed and played in the time domain. Alternatively, the extracted audio frame feature parameters can be converted into a time domain audio signal and played to improve the audio playback quality.

[0049] In some alternative implementations, the lost frame includes at least one frame; the feature parameters of the previous audio frame include stored feature parameters and / or predicted feature parameters.

[0050] For example, Figure 4 This diagram illustrates the forward rolling prediction in an audio compensation method provided in this embodiment. Filled audio frames represent unlost audio frames, while blank audio frames represent lost frames. For a lost frame T, the preceding audio frames may include frames T-2 and T-1. Feature parameters of the lost frame T can be predicted by extracting and storing the feature parameters of T-2 and T-1. For a lost frame T+1, the preceding audio frames may include frames T-1 and T. Feature parameters of the lost frame T+1 can be predicted by using the stored feature parameters of T-1 and the predicted feature parameters of frame T.

[0051] Understandably, in these alternative implementations, in the event of network anomalies such as jitter or instability, since the duration of the network anomaly is not fixed, the stored and / or predicted feature parameters can be used as feature parameters of the previous audio frame to perform forward rolling prediction on consecutive lost frames. Furthermore, if the audio signal of a lost frame is recovered and the lost frame has not been played, the predicted audio signal can be directly replaced with the received audio signal to ensure the authenticity of the audio.

[0052] In some alternative implementations, when the lost frames include multiple consecutive frames, after determining the audio signal of the lost frame, the method may further include attenuating the audio signal of the determined lost frame according to the order of the lost frames.

[0053] When predicting audio signals from multiple missing frames in a rolling manner, the predicted audio signal may exhibit mechanical noises as the number of frames increases. In these alternative implementations, the continuously predicted audio signal can be attenuated to avoid mechanical noises. For example, the predicted audio signal x[T] can be attenuated using linear attenuation, where the linear attenuation factor α(T) can be expressed as α(T) = 1 - T / K; where T represents the order of the missing frames, and K represents a preset parameter. The attenuated audio signal can be expressed as y[T] = α(T)·x(T) = (1 - T / K)·x(T). For the predicted audio signal result x(T) of the missing frames, the attenuation factor α(T) can be applied to the audio signal of the missing frames to obtain the final output.

[0054] In the technical solution of this disclosure embodiment, an audio signal can be received, and feature parameters of audio frames in the audio signal can be extracted and stored. The feature parameters include spectral features and a pitch period sequence. The pitch period sequence is formed by the pitch periods of subframes in the audio frame in sequence. In response to detecting audio frame loss, feature parameters of the lost frame are predicted based on the feature parameters of the preceding audio frames. The audio signal of the lost frame is determined based on the feature parameters of the lost frame. The receiving end of the audio signal can predict the feature parameters of the lost frame based on the feature parameters of the audio signal, thereby achieving compensation for the audio signal of the lost frame. Furthermore, by predicting the pitch period of subframes in the audio signal, finer-grained fundamental frequency information can be provided, making the compensated audio signal more realistic and natural, and ensuring audio playback quality.

[0055] This embodiment can be combined with various optional schemes in the audio compensation method provided in the above embodiments. The audio compensation method provided in this embodiment describes in detail the prediction steps of the feature parameters of the lost frame. The prediction of the pitch period sequence in the feature parameters can be achieved based on a sequence-to-sequence model, and a scheduled sampling strategy can be adopted. By probabilistically controlling the ratio of autoregressive construction mode and teacher construction mode during the sequence-to-sequence model construction process, not only can errors be prevented from accumulating during the sequence-to-sequence model prediction process, but model convergence can also be accelerated.

[0056] In the audio compensation method provided in this embodiment, predicting the feature parameters of the lost frame based on the feature parameters of the preceding audio frame can include: predicting the pitch period sequence of the lost frame based on the pitch period sequence of the preceding audio frame using a sequence-to-sequence model; wherein, the sequence-to-sequence model is constructed based on the true pitch period sequence of the sample frame.

[0057] In this embodiment, the sequence-to-sequence (Seq2Seq) model can be modeled frame-by-frame using the ground truth sequence of the pitch period of the sample frames. The sequence-to-sequence model can include a recurrent neural network (RNN) model, such as a gated recurrent unit (GRU). Using a GRU as the sequence-to-sequence model not only alleviates the problems of long-term dependencies and gradient explosion (or vanishing) but also consumes less computational power while achieving the same performance. Specifically, in the prediction phase, the sequence-to-sequence model can predict the pitch period sequence of the lost frames based on the pitch period sequence of the previous audio frames.

[0058] For example, Figure 5 This is a schematic diagram of a sequence-to-sequence model in an audio compensation method provided in an embodiment of this disclosure. Figure 5In this context, a sequence-to-sequence model can include a GRU, and a GRU can include an encoder and a decoder. Figure 5 The GRU can be expanded according to the time sequence, that is... Figure 5 Each encoder in the GRU is essentially the same encoder, and each decoder is essentially the same decoder in the GRU. The encoder and decoder can be repeatedly displayed based on the input / output data of the encoder and decoder at different time steps.

[0059] Figure 5 In this model, each audio frame can be divided into four subframes. T-P1 to T-P14 represent the pitch period sequence from the input sequence to the sequence model, and T1 to T4 represent the pitch period sequence from the input sequence to the output frame to be predicted. For example, the pitch periods X = [T-P1, T-P2, T-P3, T-P4] of the four consecutive input subframes can be encoded into a hidden state for prediction of T1 to T4.

[0060] For example, the update formula for GRU for each time step t may include:

[0061] z t =σ(W z x t +U z h t-1 +b z );

[0062] r t =σ(W r x t +U r h t-1 +b r );

[0063]

[0064] Among them, z t It can represent an update gate, r t This can indicate a door reset. h can represent candidate hidden states. t σ can represent the current hidden state, σ can represent the Sigmoid function, and ⊙ can represent the element-wise product.

[0065] The encoder state update process in GRU may include: for each input subframe (TP) i ), h i =GRU(h i-1 ,TP iThe final encoder state is h4 = GRU(h3, T-P4). In the GRU, the decoder receives the encoder's final state h4 as its initial hidden state s0 and sequentially outputs the pitch periods from T1 to T4. For each time step t, the decoder's output can be expressed as:

[0066] s t =GRU(s) t-1 ,y t-1 );

[0067] y t =OutputLayer(s t );

[0068] Among them, y t-1 It can represent the output of the previous time step, y t This can represent the output at the current time step. For example, the encoder can output the prediction results for each subframe sequentially, as shown below:

[0069] s1 = GRU(s0, y0);

[0070] T1 = OutputLayer(s1);

[0071] s2 = GRU(s1, T1);

[0072] T2 = OutputLayer(s2);

[0073] s3 = GRU(s2, T2);

[0074] T3 = OutputLayer(s3);

[0075] s4 = GRU(s3, T3);

[0076] T4 = OutputLayer(s4).

[0077] In some optional implementations, the sequence-to-sequence model construction process may include: using the true pitch period values ​​in the true pitch period sequence as the input of the current time step of the sequence-to-sequence model with a first probability to predict the next pitch period; or using the pitch period output by the previous time step of the sequence-to-sequence model as the input of the current time step of the sequence-to-sequence model with a second probability to predict the next pitch period; wherein the sum of the first probability and the second probability is 1, and the first probability gradually decreases as the time step increases.

[0078] For example, Figure 6 This is a schematic diagram illustrating the sequence-to-sequence model construction process in an audio compensation method provided in this embodiment of the present disclosure. Figure 6In this context, the construction pattern of predicting the next pitch period based on the pitch period output from the previous time step can be called the autoregressive construction pattern; the construction pattern of predicting the next pitch period based on the pitch period truth values ​​in the pitch period truth value sequence can be called the teacher construction pattern. The sequence-to-sequence model can be constructed using a scheduled sampling strategy, and the mixed input x of the sequence-to-sequence model... t It can be represented as:

[0079]

[0080] At each time step t, the probability can be 1-ε. t Using the fundamental period truth value y from the fundamental period truth value sequence t-1 Perform the next fundamental period prediction, or with the second probability ε t Use the fundamental period output from the previous time step Predict the next fundamental period. Where ε t The value can start from 0 (i.e., using the true pitch period completely) and gradually increase to 1 (i.e., using the pitch period predicted by the model completely); where ε can be... t An exponential scheduling strategy is used to gradually adjust ε t Make adjustments, for example: ε t =min(1,α) t ), where α may include a predefined attenuation rate less than 1.

[0081] Among these optional implementations, continuously using an autoregressive modeling approach during sequence-to-sequence model construction may gradually amplify errors and lead to cumulative errors, especially in the early stages of construction, making model convergence difficult. Combining this with a teacher modeling approach can correct the sequence-to-sequence model during construction, reducing divergence and accelerating convergence. By probabilistically controlling the ratio of autoregressive to teacher models, overcorrection of model parameters can be effectively prevented.

[0082] The technical solution of this disclosure provides a detailed description of the prediction steps for the feature parameters of lost frames. The prediction of the pitch period sequence in the feature parameters can be achieved based on a sequence-to-sequence model. A scheduled sampling strategy can be employed, which controls the ratio of autoregressive and teacher modes during the sequence-to-sequence model construction process. This not only prevents error accumulation during the sequence-to-sequence model prediction process but also accelerates model convergence. The audio compensation method provided in this disclosure belongs to the same disclosed concept as the audio compensation method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.

[0083] This embodiment can be combined with various optional schemes in the audio compensation method provided in the above embodiments. The audio compensation method provided in this embodiment supplements the prediction step of feature parameters. By combining the context information of the last lost frame in the lost frames to generate a pitch period correction sequence, the pitch period sequence of the last lost frame is auxiliaryly corrected, thereby enabling a smoother connection between the compensation signal of the last lost frame and the subsequent unlost audio signal, improving the sound quality of the compensation signal and ensuring audio playback quality.

[0084] In the audio compensation method provided in this embodiment, in response to the detection of audio frame loss, it may further include: determining the pitch period correction sequence of the last lost frame based on the pitch period sequence of the preceding audio frame and the pitch period sequence of the following audio frame in the last lost frame; determining the pitch period target sequence of the last lost frame based on the pitch period correction sequence and the predicted pitch period sequence of the last lost frame; wherein, the pitch period target sequence is a feature parameter.

[0085] For example, Figure 7 This is a schematic diagram illustrating the generation of a pitch period correction sequence based on context information in an audio compensation method provided in this embodiment of the present disclosure. See also... Figure 7 Filled audio frames represent unlost audio frames, while blank audio frames represent lost frames. When the lost frames include one frame T, the last lost frame is T. In this case, the pitch period correction sequence of T can be determined based on the pitch period sequence of the preceding audio frame T-1 and the following audio frame T+1. When the lost frames include multiple frames from T to T+n-1, the last lost frame is T+n-1. In this case, the pitch period correction sequence of T+n-1 can be determined based on the pitch period sequence of the preceding audio frame T+n-2 and the following audio frame T+n.

[0086] Specifically, existing interpolation algorithms can be used to interpolate the pitch period sequence of the last lost frame based on the pitch period sequences of the preceding and following audio frames, resulting in a corrected pitch period sequence. The target pitch period sequence of the last lost frame can then be determined by combining the corrected pitch period sequence with the predicted pitch period sequence. For example, a weighted approach can be used to combine the corrected pitch period sequence and the predicted pitch period sequence. Based on this, the receiver can determine the lost audio signal according to the spectral characteristics of the lost frame and the target pitch period sequence.

[0087] In this embodiment of the disclosure, when the network recovers and stabilizes, the previous lost frame can be corrected in a timely manner using the pitch period sequence of the acquired audio frame, which is conducive to the maximum restoration of the audio signal.

[0088] In some optional implementations, determining the pitch period correction sequence of the last lost frame based on the pitch period sequence of the preceding audio frame and the pitch period sequence of the following audio frame in the last lost frame can include: constructing a spline interpolation model of the subframes in the audio frame based on the pitch period sequence of the preceding audio frame and the pitch period sequence of the following audio frame in the last lost frame; inputting the frame number of the lost frame into the spline interpolation model to determine the pitch period of the subframes in the last lost frame; and determining the pitch period correction sequence of the last lost frame based on the pitch period of the subframes in the last lost frame.

[0089] The spline interpolation model can be considered a data model built based on spline functions. The spline function, for example, can be a cubic spline function, which not only guarantees the interpolation accuracy of the pitch period sequence but also reduces computational consumption to some extent. For instance, the pitch period sequence of the preceding audio frames of the last lost frame can be represented as S... T-1,1 ,S T-1,2 ,S T-1,3 ,S T-1,4 The pitch period sequence of the subsequent audio frames is represented as S. T+1,1 ,S T+1,2 ,S T+1,3 ,S T+1,4 A cubic spline function can be used to construct the spline interpolation model S for each subframe i, i = 1, 2, 3, 4 in the audio frame. i (t), and S i (t) can be expressed, for example, by the following formula:

[0090] S i (t)=a i (t-(T-1)) 3 +b i (t-(T-1)) 2 +c i (t-(T-1))+di ;

[0091] Among them, a i ,b i ,c i ,d i These are the model coefficients to be determined.

[0092] The coefficient a can be determined by solving the following system of equations. i ,b i ,c i ,d i :

[0093]

[0094] Solving the first two formulas of the system of equations ensures the correctness of the spline interpolation function; solving the last two formulas ensures the smoothness of the spline interpolation function.

[0095] After solving the equations, the spline interpolation model S for each subframe position i in the audio frame is obtained. i (t). Then through S i (t) is used to calculate the pitch period S of the subframe of the lost frame T. T,i =S i (T). Then, the pitch periods of the subframes of the lost frame can be used to form a pitch period correction sequence for the lost frame in a temporal order.

[0096] At this point, the pitch period target sequence of the last lost frame can be determined by the following formula:

[0097]

[0098] in, It can represent the pitch period sequence of the predicted last lost frame; It can represent a fundamental periodic correction sequence; α can be preset, for example, it can be 0.2, etc.

[0099] Among these optional implementations, a spline interpolation model can be constructed using spline functions to interpolate the pitch period sequence of the last lost frame based on the pitch period sequence of the preceding audio frame and the pitch period sequence of the following audio frame, thereby obtaining a pitch period correction sequence.

[0100] The technical solution of this disclosure supplements the prediction step of feature parameters. By combining the context information of the last lost frame in the lost frames to generate a pitch period correction sequence, the pitch period sequence of the last lost frame is auxiliaryly corrected. This allows the compensation signal of the last lost frame to be more smoothly connected with the subsequent unlost audio signal, improving the sound quality of the compensation signal and ensuring audio playback quality. The audio compensation method provided in this disclosure belongs to the same concept as the audio compensation method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.

[0101] Figure 8 This is a schematic diagram of an audio compensation device provided in an embodiment of this disclosure. The audio compensation device provided in this embodiment is suitable for situations where audio signal compensation is performed at the receiving end of an audio signal.

[0102] like Figure 8 As shown, the audio compensation device provided in this embodiment may include:

[0103] The feature processing module 810 is used to receive audio signals and extract and store feature parameters of audio frames in the audio signals; wherein, the feature parameters include spectral features and pitch period sequence; the pitch period sequence is composed of the pitch periods of the subframes in the audio frame in sequence;

[0104] The feature prediction module 820 is used to predict the feature parameters of the lost frame based on the feature parameters of the preceding audio frame in response to the detection of audio frame loss.

[0105] The signal determination module 830 is used to determine the audio signal of the lost frame based on the characteristic parameters of the lost frame.

[0106] In some alternative implementations, the feature prediction module can be used for:

[0107] Using a sequence-to-sequence model, the pitch period sequence of the lost frame is predicted based on the pitch period sequence of the preceding audio frames.

[0108] The sequence-to-sequence model is constructed based on the true pitch period sequence of the sample frames.

[0109] In some alternative implementations, the audio compensation device may further include:

[0110] Model building: A model used to construct sequence-to-sequence models through the following process:

[0111] With the first probability, the true pitch period value in the true pitch period sequence is used as the input to the current time step of the sequence model to predict the next pitch period;

[0112] Alternatively, with a second probability, the pitch period output from the previous time step of the sequence model can be used as the input to the current time step of the sequence-to-sequence model to predict the next pitch period;

[0113] The sum of the first probability and the second probability is 1, and the first probability gradually decreases as the time step increases.

[0114] In some alternative implementations, the lost frame includes at least one frame; the feature parameters of the previous audio frame include stored feature parameters and / or predicted feature parameters.

[0115] In some alternative implementations, the feature prediction module can also be used for:

[0116] In response to the detection of audio frame loss, the pitch period correction sequence of the last lost frame is determined based on the pitch period sequence of the audio frame preceding the last lost frame and the pitch period sequence of the audio frame following the last lost frame.

[0117] Based on the pitch period correction sequence and the predicted pitch period sequence of the last lost frame, the pitch period target sequence of the last lost frame is determined; whereby the pitch period target sequence is a feature parameter.

[0118] In some alternative implementations, the feature prediction module can be used for:

[0119] Based on the pitch period sequence of the preceding audio frame and the pitch period sequence of the following audio frame in the last lost frame, a spline interpolation model of the subframe in the audio frame is constructed.

[0120] Input the frame number of the lost frame into the spline interpolation model to determine the pitch period of the subframe in the last lost frame;

[0121] Based on the pitch period of the subframes in the last lost frame, determine the pitch period correction sequence of the last lost frame.

[0122] In some alternative implementations, the audio compensation device may further include:

[0123] The signal attenuation module is used to attenuate the audio signal of the lost frame according to the order of the lost frames after determining the audio signal of the lost frame, when the lost frame includes multiple consecutive frames.

[0124] The audio compensation device provided in this disclosure can execute the audio compensation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0125] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0126] The following is for reference. Figure 9 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 9 The diagram below shows the structure of the terminal device or server 900. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0127] like Figure 9 As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0128] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0129] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the audio compensation method of embodiments of this disclosure.

[0130] The electronic device provided in this embodiment and the audio compensation method provided in the above embodiments belong to the same disclosed concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0131] This disclosure provides a storage medium for computer-executable instructions, which, when executed by a computer processor, can be used to perform the audio compensation method provided in the above embodiments.

[0132] It should be noted that the storage medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory (FLASH), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores executable instructions that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable executable instructions. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit executable instructions for use by or in connection with an instruction execution system, apparatus, or device. Executable instructions contained on the storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0133] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0134] The aforementioned storage medium may be included in the aforementioned electronic device; or it may exist independently and not be assembled into the electronic device.

[0135] The aforementioned storage medium carries one or more executable instructions, which, when executed by the electronic device, cause the electronic device to:

[0136] The system receives audio signals and extracts and stores feature parameters of audio frames within the audio signals. These feature parameters include spectral features and a pitch period sequence. The pitch period sequence is formed by the pitch periods of the subframes within the audio frame in sequence. In response to the detection of audio frame loss, the system predicts the feature parameters of the lost frame based on the feature parameters of the preceding audio frames. Based on the feature parameters of the lost frame, the system determines the audio signal of the lost frame.

[0137] Executable instructions for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The executable instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the audio compensation method provided in any embodiment of this disclosure.

[0139] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0141] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units and modules do not, in certain circumstances, constitute a limitation on the unit or module itself.

[0142] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0143] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0144] According to one or more embodiments of this disclosure, an audio compensation method is provided, the method comprising:

[0145] The system receives an audio signal and extracts and stores feature parameters from the audio frames in the audio signal; wherein the feature parameters include spectral features and a pitch period sequence; the pitch period sequence is formed by the pitch periods of the subframes in the audio frame in sequence;

[0146] In response to detecting the loss of the audio frame, the feature parameters of the lost frame are predicted based on the feature parameters of the audio frame preceding the lost frame;

[0147] The audio signal of the lost frame is determined based on the characteristic parameters of the lost frame.

[0148] According to one or more embodiments of this disclosure, an audio compensation method is provided, further comprising:

[0149] In some optional implementations, predicting the feature parameters of the lost frame based on the feature parameters of the preceding audio frame of the lost frame includes:

[0150] Using the sequence-to-sequence model, the pitch period sequence of the lost frame is predicted based on the pitch period sequence of the preceding audio frames of the lost frame;

[0151] The sequence-to-sequence model is constructed based on the true pitch period sequence of the sample frames.

[0152] According to one or more embodiments of this disclosure, an audio compensation method is provided, further comprising:

[0153] In some optional implementations, the sequence-to-sequence model construction process includes:

[0154] With a first probability, the true pitch period value in the true pitch period sequence is used as the input of the sequence to the current time step of the sequence model to predict the next pitch period;

[0155] Alternatively, with a second probability, the pitch period output from the previous time step of the sequence model is used as the input to the current time step of the sequence-to-sequence model to predict the next pitch period;

[0156] The sum of the first probability and the second probability is 1, and the first probability gradually decreases as the time step increases.

[0157] According to one or more embodiments of this disclosure, an audio compensation method is provided, further comprising:

[0158] In some alternative implementations, the lost frame includes at least one frame; the feature parameters of the preceding audio frame include stored feature parameters and / or predicted feature parameters.

[0159] According to one or more embodiments of this disclosure, an audio compensation method is provided, further comprising:

[0160] In some optional implementations, in response to detecting the audio frame loss, the method further includes:

[0161] Based on the pitch period sequence of the preceding audio frame and the pitch period sequence of the following audio frame of the last lost frame, a pitch period correction sequence of the last lost frame is determined.

[0162] Based on the pitch period correction sequence and the predicted pitch period sequence of the last lost frame, the pitch period target sequence of the last lost frame is determined; wherein the pitch period target sequence belongs to the feature parameter.

[0163] According to one or more embodiments of this disclosure, an audio compensation method is provided, further comprising:

[0164] In some optional implementations, determining the pitch period correction sequence of the last lost frame based on the pitch period sequence of the preceding audio frame and the pitch period sequence of the following audio frame in the lost frames includes:

[0165] Based on the pitch period sequence of the preceding audio frame and the pitch period sequence of the following audio frame in the last lost frame, a spline interpolation model of the subframe in the audio frame is constructed.

[0166] The frame number of the lost frame is input into the spline interpolation model to determine the pitch period of the subframe in the last lost frame;

[0167] Based on the pitch period of the subframe in the last lost frame, determine the pitch period correction sequence of the last lost frame.

[0168] According to one or more embodiments of this disclosure, an audio compensation method is provided, further comprising:

[0169] In some alternative implementations, when the lost frames comprise multiple consecutive frames, after determining the audio signal of the lost frames, the method further includes:

[0170] The audio signals of the identified lost frames are attenuated according to the order of the lost frames.

[0171] According to one or more embodiments of this disclosure, an audio compensation device is provided, the device comprising:

[0172] The feature processing module is used to receive audio signals and extract and store feature parameters of audio frames in the audio signals; wherein, the feature parameters include spectral features and pitch period sequence; the pitch period sequence is formed by the pitch periods of the subframes in the audio frame in sequence;

[0173] A feature prediction module is configured to, in response to detecting the loss of the audio frame, predict the feature parameters of the lost frame based on the feature parameters of the preceding audio frame of the lost frame;

[0174] A signal determination module is used to determine the audio signal of the lost frame based on the characteristic parameters of the lost frame.

[0175] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0176] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0177] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An audio compensation method, characterized in that, include: The system receives an audio signal and extracts and stores feature parameters from the audio frames in the audio signal; wherein the feature parameters include spectral features and a pitch period sequence; the pitch period sequence is formed by the pitch periods of the subframes in the audio frame in sequence; In response to detecting the loss of the audio frame, the feature parameters of the lost frame are predicted based on the feature parameters of the audio frame preceding the lost frame; The audio signal of the lost frame is determined based on the characteristic parameters of the lost frame.

2. The method according to claim 1, characterized in that, The step of predicting the feature parameters of the lost frame based on the feature parameters of the preceding audio frame of the lost frame includes: Using the sequence-to-sequence model, the pitch period sequence of the lost frame is predicted based on the pitch period sequence of the preceding audio frames of the lost frame; The sequence-to-sequence model is constructed based on the true pitch period sequence of the sample frames.

3. The method according to claim 2, characterized in that, The sequence-to-sequence model construction process includes: With a first probability, the true pitch period value in the true pitch period sequence is used as the input of the sequence to the current time step of the sequence model to predict the next pitch period; Alternatively, with a second probability, the pitch period output from the previous time step of the sequence model is used as the input to the current time step of the sequence-to-sequence model to predict the next pitch period; The sum of the first probability and the second probability is 1, and the first probability gradually decreases as the time step increases.

4. The method according to claim 1, characterized in that, The lost frame includes at least one frame; the feature parameters of the previous audio frame include the stored feature parameters and / or the predicted feature parameters.

5. The method according to claim 1, characterized in that, In response to detecting the loss of the audio frame, the method further includes: Based on the pitch period sequence of the preceding audio frame and the pitch period sequence of the following audio frame of the last lost frame, a pitch period correction sequence of the last lost frame is determined. Based on the pitch period correction sequence and the predicted pitch period sequence of the last lost frame, the pitch period target sequence of the last lost frame is determined; wherein the pitch period target sequence belongs to the feature parameter.

6. The method according to claim 5, characterized in that, The step of determining the pitch period correction sequence of the last lost frame based on the pitch period sequence of the preceding audio frame and the pitch period sequence of the following audio frame in the lost frames includes: Based on the pitch period sequence of the preceding audio frame and the pitch period sequence of the following audio frame in the last lost frame, a spline interpolation model of the subframe in the audio frame is constructed. The frame number of the lost frame is input into the spline interpolation model to determine the pitch period of the subframe in the last lost frame; Based on the pitch period of the subframe in the last lost frame, determine the pitch period correction sequence of the last lost frame.

7. The method according to claim 1, characterized in that, In cases where the lost frames comprise multiple consecutive frames, after determining the audio signal of the lost frames, the method further includes: The audio signals of the identified lost frames are attenuated according to the order of the lost frames.

8. An audio compensation device, characterized in that, include: The feature processing module is used to receive audio signals and extract and store feature parameters of audio frames in the audio signals; wherein, the feature parameters include spectral features and pitch period sequence; the pitch period sequence is formed by the pitch periods of the subframes in the audio frame in sequence; A feature prediction module is configured to, in response to detecting the loss of the audio frame, predict the feature parameters of the lost frame based on the feature parameters of the preceding audio frame of the lost frame; A signal determination module is used to determine the audio signal of the lost frame based on the characteristic parameters of the lost frame.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the audio compensation method as described in any one of claims 1-7.

10. A storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to perform the audio compensation method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the audio compensation method as described in any one of claims 1-7.

Images