Method and apparatus for processing audio experiencing packet loss, medium, and device

Abstract

Embodiments of the present disclosure provide a method and apparatus for processing audio experiencing packet loss, a medium, and a device. A specific embodiment of the method comprises: for a first frame of audio experiencing packet loss, acquiring a pre-stored historical audio frame which is before the first frame and is continuous with the first frame; on the basis of the historical audio frame, performing prediction on the first frame by means of a pre-trained neural network model, obtaining first predicted audio data of the first frame and second predicted audio data corresponding to a target segment of a second frame in the audio, the target segment being connected in time to the first frame; and, on the basis of the second predicted audio data and audio data of the second frame comprised by the audio, reconstructing the audio data of the second frame.

Description

Method, device, medium and equipment for processing lost audio packets

[0001] The present application claims priority to the Chinese patent application No. 202410452103.5, filed on April 15, 2024, entitled “Method, device, medium and equipment for processing lost audio packets”, the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] Embodiments of the present disclosure relate to the technical field of audio processing, and in particular to a method, device, medium and equipment for processing lost audio packets. BACKGROUND

[0003] With the development of Internet technology and communication technology, audio streaming has become a very important part of network communication. In practice, due to various factors such as network congestion, bandwidth limitation, hardware failure, etc., audio data packets may be lost during transmission, which will have a serious impact on the quality of voice communication and cause a decline in user experience. Therefore, it is urgent to solve the problem of repairing and recovering lost audio packets. SUMMARY

[0004] Embodiments of the present disclosure describe a method and device for processing lost audio packets, which can predict first predicted audio data corresponding to a first frame of lost packets in the audio based on historical audio frames before the first frame, and predict second predicted audio data corresponding to a target segment of a second frame, wherein the target segment is continuous in time with the first frame. Then, reconstruct the audio data of the second frame based on the second predicted audio data and the audio data of the second frame included in the audio. In this way, the continuity and smoothness between the predicted first predicted audio data corresponding to the first frame and the adjacent frames can be ensured to obtain better audio compensation effect.

[0005] According to a first aspect, a method for processing lost audio packets is provided, comprising: for a first frame of lost packets in the audio, obtaining a historical audio frame stored in advance before the first frame and continuous with the first frame; based on the historical audio frame, predicting the first frame by a pre-trained neural network model to obtain first predicted audio data of the first frame and second predicted audio data corresponding to a target segment of a second frame in the audio, the target segment being continuous in time with the first frame; and reconstructing the audio data of the second frame based on the second predicted audio data and the audio data of the second frame included in the audio.

[0006] In an embodiment, the obtaining, for the first frame of the packet loss in the audio, the historical audio frame stored in advance before the first frame and continuous to the first frame includes: determining to use the neural network model to perform the packet loss compensation on the first frame of the packet loss in a case where a current condition meets a preset condition, where the preset condition includes at least one of the following: a previous frame of the first frame is a voiced frame, and a number of audio streams currently decoded by an audio decoder does not exceed a preset threshold; and the obtaining, for the first frame of the packet loss in the audio, the historical audio frame stored in advance before the first frame and continuous to the first frame. In this way, the neural network model can be used to perform the packet loss compensation on the frame of the packet loss in a case where the preset condition is met.

[0007] In an embodiment, the reconstructing, based on the second predicted audio data and the audio data of the second frame included in the audio, the audio data of the second frame includes: adding the second predicted audio data and a target segment of the audio data of the second frame included in the audio after a windowing operation, to obtain the audio data corresponding to the reconstructed second frame. In this way, the reconstructed second frame and the first frame can be more continuous and smooth.

[0008] In an embodiment, the neural network model is a time domain model, and the time domain model is obtained by training in the following manner: inputting a first sample historical frame corresponding to a first sample frame in a first sample audio into a to-be-trained time domain model, performing prediction on the first sample frame by the to-be-trained time domain model to obtain first sample predicted audio data for the first sample frame and first partial predicted audio data corresponding to a target segment of a next frame of the first sample frame; reconstructing partial audio data corresponding to the target segment of the next frame of the first sample frame based on the first partial predicted audio data and the partial audio data corresponding to the target segment of the next frame of the first sample frame in the first sample audio, to obtain first partial reconstructed audio data; taking the audio data of the first sample frame in the first sample audio and the partial audio data corresponding to the target segment of the next frame of the first sample frame as first labels, and taking the first sample predicted audio data and the first partial reconstructed audio data as first prediction data; and adjusting model parameters of the to-be-trained time domain model based on a difference loss of the first labels and the first prediction data. In this way, the time domain model can be trained, so that the packet loss compensation can be performed in the time domain.

[0009] In an embodiment, the neural network model is a frequency domain model; and the frequency domain model is trained by: performing short-time Fourier transform (STFT) on a second sample history frame corresponding to a second sample frame in a second sample audio to obtain a history STFT coefficient corresponding to the second sample history frame; inputting the history STFT coefficient into a to-be-trained frequency domain model to obtain a first predicted STFT coefficient, wherein the first predicted STFT coefficient includes a second predicted STFT coefficient corresponding to the second sample frame and a partial predicted STFT coefficient corresponding to partial audio data corresponding to a target segment of a next frame of the second sample frame; performing inverse short-time Fourier transform (iSTFT) based on the first predicted STFT coefficient to obtain second sample predicted audio data of the second sample frame and second partial predicted audio data corresponding to the target segment of the next frame of the second sample frame; reconstructing the partial audio data corresponding to the target segment of the next frame of the second sample frame based on the second partial predicted audio data and the partial audio data corresponding to the target segment of the next frame of the second sample frame in the second sample audio to obtain second partial reconstructed audio data; taking the audio data of the second sample frame and the audio data corresponding to the target segment of the next frame of the second sample frame in the second sample audio as a second label and taking the second sample predicted audio data and the second partial reconstructed audio data as second predicted data; and adjusting model parameters of the to-be-trained frequency domain model based on a difference loss of the second label and the second predicted data. Thus, the frequency domain model can be trained, so that the packet loss compensation can be performed in the frequency domain.

[0010] In an embodiment, the method is performed by an audio decoder, and the obtaining, for the first frame of the lost packet in the audio, the pre-stored history audio frame before the first frame and continuous to the first frame includes: the audio decoder, when playing the first frame of the audio, obtains, for the first frame of the lost packet in the audio, the pre-stored history audio frame before the first frame and continuous to the first frame after determining that the first frame is the frame of the lost packet; and the method further includes: the audio decoder plays the first frame based on the first predicted audio data and plays the second frame based on the reconstructed audio data of the second frame. Thus, the first frame and the second frame played can be smoother.

[0011] In an embodiment, the audio decoder includes a preset packet loss compensation algorithm; and the method further includes: in a case where it is determined not to use the neural network model, performing packet loss compensation on the first frame by using the packet loss compensation algorithm included in the audio decoder. Thus, the packet loss compensation can be implemented without using the neural network model.

[0012] According to a second aspect, a processing apparatus for audio with packet loss is provided, comprising: an obtaining unit configured to, for a first frame of packet loss in audio, obtain a historical audio frame stored in advance before the first frame and continuous to the first frame; a predicting unit configured to, based on the historical audio frame, predict the first frame by a pre-trained neural network model to obtain first predicted audio data of the first frame and second predicted audio data corresponding to a target segment of a second frame in the audio, the target segment being connected to the first frame in time; and a reconstructing unit configured to reconstruct audio data of the second frame based on the second predicted audio data and audio data of the second frame included in the audio.

[0013] According to a third aspect, a computer program product is provided, comprising a computer program, the computer program being executed by a processor to implement the method according to any one of the first aspect.

[0014] According to a fourth aspect, a computer readable storage medium is provided, having stored thereon a computer program, the computer program being executed in a computer to cause the computer to perform the method according to any one of the first aspect.

[0015] According to a fifth aspect, an electronic device is provided, comprising a memory and a processor, the memory having stored therein executable code, the processor executing the executable code to implement the method according to any one of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0016] FIG. 1 shows a schematic diagram of an application scenario to which embodiments of the present disclosure can be applied;

[0017] FIG. 2 shows a flowchart of a method for processing audio with packet loss according to an embodiment;

[0018] FIG. 3 shows a schematic diagram of an example of using a neural network model in combination with a packet loss compensation algorithm in an audio decoder;

[0019] FIG. 4 shows a schematic diagram of an example of training a time domain model;

[0020] FIG. 5 shows a schematic diagram of an example of a prediction result output by a frequency domain model to be trained;

[0021] FIG. 6 shows a schematic diagram of an example of reconstructing a second frame;

[0022] FIG. 7 shows a schematic block diagram of a processing apparatus for audio with packet loss according to an embodiment;

[0023] FIG. 8 shows a schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the use range, the use scenario, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0025] For example, in response to receiving the active request of the user, the prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will need to obtain and use the personal information of the user. Thus, the user can autonomously select whether to provide the personal information to the software or hardware such as the electronic device, the application program, the server or the storage medium, etc. that performs the operation of the technical solutions of the present disclosure according to the prompt information.

[0026] As an optional but non-limiting implementation manner, in response to receiving the active request of the user, the manner of sending the prompt information to the user may, for example, be the manner of a pop-up window, and the prompt information may be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide the personal information to the electronic device.

[0027] It can be understood that the above notification and obtaining of the authorization of the user are only illustrative, and do not limit the implementation manners of the present disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation manners of the present disclosure.

[0028] The technical solutions provided by the present disclosure will be described in further detail below in combination with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings. It should be noted that, in the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0029] As mentioned before, it is an urgent problem to recover the audio packet loss. At present, in the real-time communication scene, the PLC (Packet Loss Concealment) technology in the receiving end audio decoder. For example, the PLC algorithm in OPUS Silk can usually recover higher audio quality in the 40ms (millisecond) short packet loss scene. But in the longer packet loss scene (for example, 60-100ms continuous packet loss), the energy attenuation of the recovered audio is significant, and the quality is significantly reduced. With the development of AI (Artificial Intelligence) technology, in some ways, AI model can be used to realize the related research of PLC. At present, the AI PLC model mostly uses a continuous state model, and the model state also needs to be updated in the non-packet loss scene to ensure the quality of the output recovered audio when packet loss occurs. However, in actual application, the calling probability of the PLC algorithm in the audio decoder is relatively low, the continuous state model will introduce a large amount of additional computational complexity, and some PLC models will introduce a certain algorithm delay (usually about 20ms), which is not suitable for the current audio link, and it is difficult to apply.

[0030] Therefore, the embodiment of the present disclosure provides a processing method for audio packet loss, which can perform audio prediction based on historical audio frames of the audio packet loss frames, so as to ensure the continuity and smoothness between the predicted audio data corresponding to the predicted audio frames and the front and rear frames, and obtain better audio compensation effect. FIG. 1 shows a schematic diagram of an application scenario to which the embodiment of the present disclosure can be applied. As shown in FIG. 1, in the application scenario shown in FIG. 1, taking the sampling rate of the audio as 16kHz (kilohertz) and the frame length as 10ms as an example, each frame can include audio data of 160 sampling points. Assuming that the xth frame in the audio S has packet loss, for the xth frame, the historical audio frames before the xth frame and continuous with the xth frame can be obtained from the buffer. In this example, the historical audio frames corresponding to the xth frame include x-6, x-5, …, x-1 frames, that is, 6 frames before the xth frame. Then, based on the historical audio frames, the xth frame is predicted through a pre-trained neural network model to obtain first predicted audio data corresponding to the xth frame and second predicted audio data corresponding to a target segment of the x+1th frame in the audio. Here, the target segment is connected with the xth frame in time. In this example, the first predicted audio data includes 160 sampling point data, and the second predicted audio data includes 40 sampling points. Based on the first predicted audio data, the audio of the xth frame can be played, and the data stored in the buffer can also be updated. That is, the data cached in the buffer is updated to x-5, x-4, …, x frames. Then, based on the second predicted audio data and the audio data of the x+1th frame included in the audio S, the audio data of the x+1th frame is reconstructed. In this way, the continuity and smoothness of the xth frame and the x+1th frame can be ensured.

[0031] According to the method and the device for processing the audio with packet loss provided by the embodiments of the present disclosure, for the first frame with packet loss in the audio, a historical audio frame before the first frame and continuous with the first frame is obtained. Then, based on the historical audio frame, the first frame is predicted by a pre-trained neural network model to obtain first predicted audio data of the first frame and second predicted audio data corresponding to a target segment of a second frame in the audio. The second frame is the next frame of the first frame. Finally, based on the second predicted audio data and audio data of the second frame included in the audio, the audio data of the second frame is reconstructed. Since the first predicted audio data and the second predicted audio data are obtained based on the historical audio frame, the continuity and smoothness between the first frame and the previous frame can be ensured, and the audio data of the second frame is reconstructed based on the second predicted audio data, which can ensure the continuity and smoothness between the first frame and the next frame. Therefore, the embodiments of the present disclosure can ensure the continuity and smoothness between the first predicted audio data corresponding to the predicted first frame and the previous and next frames, so as to obtain better audio compensation effect.

[0032] Referring to FIG. 2, FIG. 2 shows a flowchart of a method for processing audio with packet loss according to an embodiment. The method can be performed by an audio decoder. As shown in FIG. 2, the method for processing audio with packet loss can include the following steps 201 to 203.

[0033] Step 201, for the first frame with packet loss in the audio, a historical audio frame before the first frame and continuous with the first frame is obtained. In the present embodiment, the existing (all or part) audio frames can be pre-stored. For example, the existing part of the audio frames can be pre-stored in a buffer. In this way, for the first frame with packet loss in the audio, the historical audio frame before the first frame and continuous with the first frame can be obtained from the buffer. Here, the first frame can refer to the frame with packet loss in the audio.

[0034] In some implementations, a preset packet loss compensation algorithm can be included in the audio decoder. For example, taking the Silk decoder in OPUS as an example, a PLC algorithm can be included in the Silk decoder. Here, a neural network model for predicting part of the audio data of the frame with packet loss and the next frame thereof can be pre-trained and used in combination with the packet loss compensation algorithm in the audio decoder. At this time, the above step 201 can include the following steps S1 and S2.

[0035] Step S1, in a case where the current condition meets a preset condition, it is determined to use the neural network model to perform packet loss compensation on the first frame of the packet loss audio. In the present implementation, the condition can be preset according to actual needs to determine whether to use the neural network model to perform packet loss compensation on the first frame of the packet loss audio. As an example, the preset condition can include but is not limited to: the previous frame of the first frame is a voiced frame, the number of audio streams currently decoded by the audio decoder is not more than a preset threshold, and the like.

[0036] In practice, when the packet loss frame is a unvoiced frame or a silence frame, the packet loss compensation algorithm included in the audio decoder can well recover the packet loss frame. Therefore, the neural network model does not need to be called, and the voiced frame needs to be called. Generally, the previous frame of the voiced frame is also a voiced frame with a high probability, so when the previous frame of the packet loss frame is a voiced frame, the neural network model can be used to perform packet loss compensation on the packet loss frame.

[0037] In addition, in a case where the audio decoder needs to decode multiple audio streams at the same time, if the neural network model is used to perform packet loss compensation on the packet loss frame, the computational complexity will be high. Therefore, in a case where the number of audio streams currently decoded by the audio decoder is not more than a preset threshold, the neural network model can be used to perform packet loss compensation on the packet loss frame.

[0038] Step S2, for the first frame of the packet loss audio, a pre-stored historical audio frame between and continuous with the first frame is obtained. In the present implementation, in a case where it is determined to use the neural network model to perform packet loss compensation on the first frame of the packet loss audio, a pre-stored historical audio frame before and continuous with the first frame is obtained. Through the present implementation, in a case where the current condition meets the preset condition, the historical audio frame corresponding to the first frame can be obtained to call the neural network model for packet loss compensation.

[0039] In other implementations, the audio decoder can include a preset packet loss compensation algorithm, and the above-mentioned method for processing the packet loss audio further includes: in a case where it is determined not to use the neural network model, using the packet loss compensation algorithm included in the audio decoder to perform packet loss compensation on the first frame.

[0040] As shown in FIG. 3, FIG. 3 shows a schematic diagram of one example of using the neural network model in combination with the packet loss compensation algorithm in the audio decoder. In the example shown in FIG. 3, the history audio frames used in predicting the audio data by the neural network model can be buffered in a buffer. Based on this, the process shown in FIG. 3 can include the following steps 301-306, specifically: step 301, judging whether the packet loss occurs according to the currently received audio data. Step 302, if the packet loss does not occur, updating the data required by the packet loss compensation algorithm in the audio decoder according to the received audio data. Updating the data in the buffer according to the received audio data. Then, outputting the audio data. Step 303, if the packet loss occurs, using the packet loss compensation algorithm in the audio decoder to perform the packet loss compensation, obtaining a first packet loss compensation result. Step 304, judging whether to use the neural network model. Step 305, if the neural network model is not used, updating the data in the buffer based on the first packet loss compensation result obtained in step 303. Then, outputting the audio data containing the first packet loss compensation result. Step 306, if the neural network model is used, using the neural network model to perform the packet loss compensation, obtaining a second packet loss compensation result, and using the second packet loss compensation result to overwrite the first packet loss compensation result. In addition, some data of the decoder can also be updated according to the second packet loss compensation result. For example, updating the state of the decoder, updating the data required by the packet loss compensation algorithm, updating the data in the buffer, etc. Then, outputting the audio data containing the second packet loss compensation result. In this way, the neural network model can be used in combination with the packet loss compensation algorithm in the audio decoder, and the efficiency and quality of the packet loss compensation can be improved.

[0041] At step 202, based on the historical audio frames, the first frame is predicted by a pre-trained neural network model to obtain first predicted audio data of the first frame and second predicted audio data corresponding to a target segment of a second frame in the audio. In this embodiment, a neural network model for predicting the partial audio data of the lost packet frame and the next frame thereof can be pre-trained, and the neural network model can be a neural network model of various network structures. For example, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), etc. The neural network model can predict the first frame based on the historical audio frames to obtain the first predicted audio data of the first frame and the second predicted audio data corresponding to the target segment of the second frame. Here, the second frame can refer to the next frame of the first frame, and the target segment of the second frame can refer to the previous segment of the second frame, so the target segment is connected in time with the first frame. That is, the neural network model can predict the audio data corresponding to the lost packet frame, and also predict the audio data of the previous segment of the next frame of the lost packet frame, which is used for smoothing with the next frame to avoid discontinuity between the predicted frame and the next frame.

[0042] Here, the number of sampling points included in the target segment can be set according to actual needs. For example, taking the sampling rate of the audio as 16 kHz (kilohertz) and the frame length as 10 ms as an example, 160 sampling points can be included in each frame. At this time, the target segment can be set to include 40 sampling points. Taking the sampling rate of the audio as 16 kHz (kilohertz) and the frame length as 20 ms as an example, 320 sampling points can be included in each frame, and at this time, the target segment can be set to include 80 sampling points.

[0043] In some implementations, the neural network model described above can be a time domain model, which can be trained through the following steps 1 to 4. Specifically, step 1, input the first sample historical frame corresponding to the first sample frame in the first sample audio into the to-be-trained time domain model, and predict the first sample frame by the to-be-trained time domain model to obtain the first sample predicted audio data for the first sample frame, and the first part of the predicted audio data corresponding to the target segment of the next frame of the first sample frame. Step 2, based on the first part of the predicted audio data and the part of the audio data corresponding to the target segment of the next frame of the first sample frame in the first sample audio, reconstruct the part of the audio data corresponding to the target segment of the next frame of the first sample frame to obtain the first part of the reconstructed audio data. Step 3, take the audio data corresponding to the target segment of the next frame of the first sample frame in the first sample audio as the first label, and take the first sample predicted audio data and the first part of the reconstructed audio data as the first prediction data. Step 4, adjust the model parameters of the to-be-trained time domain model based on the difference loss of the first label and the first prediction data. For example, adjust the model parameters of the to-be-trained time domain model to minimize the difference loss of the first label and the first prediction data, so as to obtain a time domain model that meets the prediction requirements.

[0044] As shown in FIG. 4, FIG. 4 shows a schematic diagram of an example of training a time domain model. In the example shown in FIG. 4, the first sample audio can be any segment of audio, and the first sample audio can include audio frames x-6, x-5, …, x, x+1, … x+5, etc. In this example, the x frame in the first sample audio is taken as the first sample frame, and the first sample historical frame corresponding to the x frame can include a preset number of frames before the x frame. In this example, the first sample historical frame includes x-6, x-5, … x-1 frames. After inputting the first sample historical frame corresponding to the x frame into the to-be-trained time domain model 401, the to-be-trained time domain model 401 predicts the x frame to obtain the first sample predicted audio data for the x frame, and the first part of the predicted audio data corresponding to the previous segment (i.e., a preset number of sampling points) of the next frame (i.e., x+1 frame) of the x frame.

[0045] Then, based on the first part of the predicted audio data and the corresponding part of the audio data of the previous segment of the x+1th frame in the first sample audio, the corresponding audio data of the previous segment of the x+1th frame is reconstructed to obtain the first part of the reconstructed audio data. At this time, the audio data of the xth frame and the audio data of the previous segment of the x+1th frame in the first sample audio can be taken as the first label, and the first sample predicted audio data output by the to-be-trained time domain model 401 for the xth frame and the first part of the reconstructed audio data obtained by reconstruction can be taken as the first prediction data. Finally, a preset loss function can be used to calculate the difference loss between the first label and the first prediction data. With the goal of minimizing the difference loss, the model parameters of the to-be-trained time domain model 401 are adjusted to obtain the time domain model. As an example, the difference loss can be directly calculated in the time domain. As another example, the difference loss can also be calculated in the frequency domain. For example, in order to consider the inter-frame continuity, padding operation (for example, padding using the original values of the preset frames before and after) can be performed on the first prediction data and the first label before short-time Fourier transform (STFT). Then, the short-time Fourier transform is performed to obtain the STFT coefficients corresponding to the first prediction data and the first label, respectively, and then the difference loss between the two STFT coefficients is calculated. It is found through research and analysis that the discontinuity difference between the predicted xth frame and the x-1th frame is obvious in the STFT domain. Reducing this difference in training can ensure the accuracy of the predicted audio while ensuring the continuity between the xth frame and the x-1th frame.

[0046] In some implementations, the neural network model can also be a frequency domain model, which can be trained by the following steps 1 to 6. Specifically, step 1, performing short-time Fourier transform (STFT) on the second sample historical frame corresponding to the second sample frame in the second sample audio to obtain historical STFT coefficients corresponding to the second sample historical frame. Step 2, inputting the historical STFT coefficients into the frequency domain model to be trained to obtain first predicted STFT coefficients. Here, the first predicted STFT coefficients can include second predicted STFT coefficients corresponding to the second sample frame, and partial predicted STFT coefficients corresponding to partial audio data corresponding to the target segment of the next frame of the second sample frame. Step 3, performing Inverse Short-Time Fourier Transform (ISTFT) based on the first predicted STFT coefficients to obtain second sample predicted audio data of the second sample frame, and second partial predicted audio data corresponding to the target segment of the next frame of the second sample frame. Step 4, reconstructing the partial audio data corresponding to the target segment of the next frame of the second sample frame based on the second partial predicted audio data and the partial audio data corresponding to the target segment of the next frame of the second sample frame in the second sample audio to obtain second partial reconstructed audio data. Step 5, taking the audio data of the second sample frame and the audio data corresponding to the target segment of the next frame of the second sample frame in the second sample audio as the second label, and taking the second sample predicted audio data and the second partial reconstructed audio data as the second predicted data. Step 6, adjusting the model parameters of the frequency domain model to be trained based on the difference loss of the second label and the second predicted data.

[0047] As shown in FIG. 5, FIG. 5 shows a schematic diagram of an example of a prediction result output by the frequency domain model to be trained. In the example shown in FIG. 5, the second sample audio can be any segment of audio. In the example shown in FIG. 5, the second sample historical frame corresponding to the second sample frame can include 1024 sample points of audio data, and the frequency domain model to be trained can output a prediction result of 384 sample points of audio data corresponding to the packet loss position. It can be understood that the 384 sample points can include predicted audio data of the second sample frame and predicted audio data of the target segment of the next frame of the second sample frame.

[0048] In this example, the second sample history frame can be first subjected to short-time Fourier transform to obtain the history STFT coefficients corresponding to the second sample history frame. It can be understood that, in the short-time Fourier transform STFT, in order to process the signal boundary problem, ensure that each window can be subjected to Fourier transform completely, and reduce the spectral leakage caused by the discontinuity of the window boundary, padding operation can be performed. For example, the padding value can be 0, or can be a certain specific value. In this example, the parameters of the short-time Fourier transform STFT can be set as follows: the number of fft (fast Fourier transform) points n_fft = 512, the window length window = n_fft, the window shift hop_size = n_fft / 4, and the padding = n_fft / 2. That is, in this example, the 256 sampling points before the second sample history frame and the 256 sampling points after the second sample history frame can be padded. After the short-time Fourier transform STFT, the history STFT coefficients s0-s8 can be obtained. In the short-time Fourier transform STFT, s0, s1, s7, and s8 are related to the padding.

[0049] Then, the history STFT coefficients s0-s8 can be input into the to-be-trained frequency domain model, and the first predicted STFT coefficients s7-s11 can be output by the to-be-trained frequency domain model. In the inverse short-time Fourier transform iSTFT, s7 and s8 in the first predicted STFT coefficients are used to replace s7 and s8 in the history STFT coefficients. The first predicted STFT coefficients are combined with the history STFT coefficients, and s0-s11 are obtained. Then, the combination result is subjected to the inverse short-time Fourier transform iSTFT, and the audio data of 1024 sampling points included in the second sample history frame and the audio data of 384 sampling points corresponding to the packet loss position are obtained. The audio data of 384 sampling points corresponding to the packet loss position can be used as the output predicted result.

[0050] It can be understood that the number of sampling points of the sample history frame, the number of sampling points of the predicted result, and the like in the example shown in FIG. 5 are only illustrative, and are not limited.

[0051] At step 203, the audio data of the second frame is reconstructed based on the second predicted audio data and the audio data of the second frame included in the audio. In this embodiment, since only the data of the historical audio frame corresponding to the first frame is used when predicting the audio data of the first frame, and the audio data of the next frame (i.e., the second frame) of the first frame is not used, the disclosed embodiment does not need to wait until the audio data of the second frame is received before predicting the audio data of the first frame. Therefore, the disclosed embodiment has no algorithmic latency. However, there may be discontinuity between the predicted audio data of the first frame and the audio data of the next frame. In order to alleviate the discontinuity with the next frame, the disclosed embodiment predicts the audio data corresponding to the target segment of the next frame of the first frame at the same time as predicting the audio data of the first frame. In this way, the audio data of the second frame can be reconstructed based on the second predicted audio data predicted by the neural network model and the audio data of the second frame included in the audio.

[0052] It can be understood that the processing method for audio packet loss of the disclosed embodiment can regard all packet loss conditions (e.g., single packet loss, continuous packet loss, etc.) as single packet loss tasks, and the states of the continuous packet loss scenario neural network model are independent of each other. For example, in a continuous packet loss scenario, each single packet loss in the continuous packet loss can be regarded as a separate packet loss task to call the neural network model for processing, and there is no dependency relationship between multiple calls of the model.

[0053] In some implementations, the above step 203 can be specifically performed as follows: the second predicted audio data is added to the target segment of the audio data of the second frame included in the audio after a windowing operation, to obtain the reconstructed audio data corresponding to the second frame.

[0054] In this implementation, the second predicted audio data corresponding to the target segment of the second frame predicted and the target segment of the audio data of the second frame included in the audio can be added after being subjected to a windowing operation, respectively, to obtain the reconstructed audio data corresponding to the second frame. In this way, the discontinuity between the first frame and the second frame can be alleviated. Here, the windowing operation can multiply a signal with a specific window function, so as to reduce spectral leakage and correct the discontinuity of the signal. Various window functions can be used in the windowing operation, such as the Hanning window, the Hamming window, etc.

[0055] As shown in FIG. 6, FIG. 6 shows a schematic diagram of an example of reconstructing the second frame. In the example shown in FIG. 6, the sampling rate of the audio is 16 kHz (kilohertz), the frame length is 10 ms, 160 sampling points can be included in each frame, and the target section includes 40 sampling points. In FIG. 6, the x frame is the first frame, and the x+1 frame is the second frame. That is, when the neural network model is predicting, the audio data of the 160 sampling points of the x frame is predicted, and the audio data of the first 40 sampling points of the x+1 frame (i.e., the next frame of the x frame) is also predicted. In this example, the audio data of the x+1 frame can be obtained. For example, the audio data of the x+1 frame is received from the sending end, or the audio data of the x+1 frame is predicted and obtained through the disclosed embodiments. At this time, the audio data of the first 40 sampling points of the x+1 frame predicted by the neural network model and the audio data of the first 40 sampling points of the x+1 frame obtained later can be added after being windowed, respectively. That is, the overlapping parts of the x frame and the x+1 frame are windowed and added, and the audio data of the reconstructed x+1 frame is obtained.

[0056] In some implementations, the method for processing the lost packet of the audio described above can be performed by an audio decoder. Based on this, the step 201 described above can specifically include the following content: when playing to the first frame of the audio, after determining that the first frame is a lost packet frame, for the lost packet first frame in the audio, the audio decoder obtains the pre-stored historical audio frame before the first frame and continuous with the first frame. Based on this, the step 203 described above can specifically include the following content: the audio decoder plays the first frame based on the first predicted audio data, and plays the second frame based on the reconstructed audio data of the second frame. In this way, the audio after the loss compensation can be played.

[0057] In the above process, in the above embodiment of the present disclosure, for the first lost packet frame in the audio, the pre-stored historical audio frame before the first frame and continuous with the first frame is obtained. Then, based on the historical audio frame, the first frame is predicted through the pre-trained neural network model to obtain the first predicted audio data of the first frame and the second predicted audio data corresponding to the target section of the second frame in the audio. The second frame is the next frame of the first frame. Finally, based on the second predicted audio data and the audio data of the second frame included in the audio, the audio data of the second frame is reconstructed. Since the first predicted audio data and the second predicted audio data are obtained based on the historical audio frame, the continuity and smoothness between the first frame and the previous frame can be ensured. And based on the second predicted audio data to reconstruct the audio data of the second frame, the continuity and smoothness between the first frame and the next frame can be ensured. Therefore, the embodiments of the present disclosure can ensure the continuity and smoothness between the first predicted audio data corresponding to the predicted first frame and the previous and next frames, so as to obtain a better audio compensation effect.

[0058] According to another aspect, embodiments provide a device for processing packet-loss audio. The device for processing packet-loss audio can be deployed in an audio decoder.

[0059] FIG. 7 shows a schematic block diagram of a device for processing packet-loss audio according to an embodiment. The device shown in FIG. 7 is configured to perform the method shown in FIG. 2. As shown in FIG. 7, the device for processing packet-loss audio 700 includes an obtaining unit 701 configured to, for a first frame of packet-loss in an audio, obtain a historical audio frame stored in advance before the first frame and continuous to the first frame. The device for processing packet-loss audio 700 further includes a predicting unit 702 configured to, based on the historical audio frame, predict the first frame by a pre-trained neural network model to obtain first predicted audio data of the first frame and second predicted audio data corresponding to a target segment of a second frame in the audio, the target segment being connected to the first frame in time. The device for processing packet-loss audio 700 further includes a reconstructing unit 703 configured to reconstruct audio data of the second frame based on the second predicted audio data and audio data of the second frame included in the audio.

[0060] In some optional implementations of the present embodiments, the obtaining unit 701 is further configured to, in a case where a current condition meets a preset condition, determine to perform packet-loss compensation on the first frame of packet-loss by the neural network model. The preset condition includes at least one of the following: a previous frame of the first frame is a voiced frame, and a number of audio streams currently decoded by the audio decoder does not exceed a preset threshold. For the first frame of packet-loss in the audio, the historical audio frame stored in advance before the first frame and continuous to the first frame is obtained.

[0061] In some optional implementations of the present embodiments, the reconstructing unit 703 is further configured to add the second predicted audio data and a target segment of the audio data of the second frame included in the audio after a windowing operation to obtain audio data corresponding to the reconstructed second frame.

[0062] In some optional implementations of the present embodiment, the neural network model is a time domain model, and the time domain model is trained in the following manner: inputting first sample historical frames corresponding to a first sample frame in first sample audio into a to-be-trained time domain model, predicting the first sample frame by the to-be-trained time domain model to obtain first sample predicted audio data for the first sample frame and first partial predicted audio data corresponding to a target segment of a next frame of the first sample frame. Based on the first partial predicted audio data and partial audio data corresponding to the target segment of the next frame of the first sample frame in the first sample audio, reconstructing the partial audio data corresponding to the target segment of the next frame of the first sample frame to obtain first partial reconstructed audio data. Taking audio data of the first sample frame and the partial audio data corresponding to the target segment of the next frame of the first sample frame in the first sample audio as first labels, and taking the first sample predicted audio data and the first partial reconstructed audio data as first prediction data. Adjusting model parameters of the to-be-trained time domain model based on a difference loss of the first labels and the first prediction data.

[0063] In some optional implementations of the present embodiment, the neural network model is a frequency domain model, and the frequency domain model is trained in the following manner: performing short-time Fourier transform (STFT) on second sample historical frames corresponding to a second sample frame in second sample audio to obtain historical STFT coefficients corresponding to the second sample historical frames; inputting the historical STFT coefficients into a to-be-trained frequency domain model to obtain first predicted STFT coefficients. The first predicted STFT coefficients include second predicted STFT coefficients corresponding to the second sample frame and partial predicted STFT coefficients corresponding to partial audio data of a target segment of a next frame of the second sample frame; performing inverse short-time Fourier transform (iSTFT) based on the first predicted STFT coefficients to obtain second sample predicted audio data of the second sample frame and second partial predicted audio data corresponding to the target segment of the next frame of the second sample frame; based on the second partial predicted audio data and partial audio data corresponding to the target segment of the next frame of the second sample frame in the second sample audio, reconstructing the partial audio data corresponding to the target segment of the next frame of the second sample frame to obtain second partial reconstructed audio data; taking audio data of the second sample frame and audio data corresponding to the target segment of the next frame of the second sample frame in the second sample audio as second labels, and taking the second sample predicted audio data and the second partial reconstructed audio data as second prediction data; and adjusting model parameters of the to-be-trained frequency domain model based on a difference loss of the second labels and the second prediction data.

[0064] In some optional implementation forms of the embodiment, the apparatus 700 is deployed in an audio decoder, the obtaining unit 701 is further configured to, when playing the first frame of the audio, obtain a historical audio frame stored in advance before the first frame and continuous to the first frame, after determining that the first frame is a lost frame in the audio; and the apparatus 700 further comprises a playing unit (not shown in the figure), configured to play the first frame based on the first predicted audio data and play the second frame based on the reconstructed audio data of the second frame.

[0065] In some optional implementation forms of the embodiment, the audio decoder comprises a preset lost packet compensation algorithm; and the apparatus 700 further comprises a lost packet compensation unit (not shown in the figure), configured to, in the case of determining not to use the neural network model, compensate the first frame for lost packets using the lost packet compensation algorithm comprised in the audio decoder.

[0066] The apparatus embodiment corresponds to the method embodiment, and the specific description can be referred to the description of the method embodiment, which will not be repeated here. The apparatus embodiment is based on the corresponding method embodiment and has the same technical effect as the corresponding method embodiment. The specific description can be referred to the corresponding method embodiment.

[0067] According to another aspect of the embodiment, a computer readable storage medium is also provided, which stores a computer program. When the computer program is executed in a computer, the computer is caused to execute the method described in FIG. 2.

[0068] According to another aspect of the embodiment, an electronic device is also provided, which comprises a memory and a processor. The memory stores an executable code. When the processor executes the executable code, the method described in FIG. 2 is implemented.

[0069] The above describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order in which they are recited, and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily have to be performed in the specific order described or illustrated, nor in consecutive order, unless specifically stated or required. In some embodiments, multitasking and parallel processing can be advantageous.

[0070] Reference is made to FIG. 8, which shows a structural schematic diagram of an electronic device 800 suitable for implementing the embodiments of the present application. The electronic device shown in FIG. 8 is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0071] As shown in FIG. 8, the electronic device 800 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or loaded into a random access memory (RAM) 803 from a storage device 808. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0072] Generally, the following devices can be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 can allow the electronic device 800 to communicate with other devices wirelessly or via wires to exchange data. Although FIG. 8 shows the electronic device 800 with various devices, it should be understood that all of the illustrated devices are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed. Each block shown in FIG. 8 can represent one device or, as needed, multiple devices.

[0073] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-described functions defined in the methods of embodiments of the present application are performed.

[0074] Embodiments of the present disclosure also provide a computer readable storage medium, having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the method provided by the present disclosure.

[0075] It should be noted that the computer readable medium in the embodiments of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the embodiments of the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (Radio Frequency) or the like, or any suitable combination of the above.

[0076] The computer readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device. The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to: for a first frame of a packet loss in audio, obtain a historical audio frame stored in advance before the first frame and continuous with the first frame; based on the historical audio frame, predict the first frame by a pre-trained neural network model to obtain first predicted audio data of the first frame and second predicted audio data corresponding to a target segment of a second frame in the audio, the target segment being connected in time with the first frame; and based on the second predicted audio data and audio data of the second frame included in the audio, reconstruct the audio data of the second frame.

[0077] Computer program code for carrying out operations of embodiments of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or electronic device. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0078] The various embodiments in the present disclosure are described in progressive manner, and the same or similar parts among the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, the storage medium and computing device embodiments are described simply because they are substantially similar to the method embodiments, and the relevant parts can be referred to the description of the method embodiments.

[0079] Those skilled in the art should be aware that, in the above one or more examples, the functions described in the embodiments of the present disclosure can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium.

[0080] The above detailed description has further explained the purpose, technical solutions and advantages of the embodiments of the present disclosure. It should be understood that the above description is only a specific implementation of the embodiments of the present disclosure, and is not intended to limit the protection scope of the present disclosure. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for processing audio with packet loss, comprising: obtaining a historical audio frame stored in advance before and continuous with a first frame with packet loss in the audio; based on the historical audio frame, predicting the first frame by a pre-trained neural network model to obtain first predicted audio data of the first frame and second predicted audio data corresponding to a target segment of a second frame in the audio, the target segment being connected with the first frame in time; based on the second predicted audio data and audio data of the second frame included in the audio, reconstructing the audio data of the second frame.

2. The method of claim 1, wherein the obtaining a historical audio frame stored in advance before and continuous with a first frame with packet loss in the audio comprises: if a current condition meets a preset condition, performing packet loss compensation on the first frame using the neural network model, wherein the preset condition comprises at least one of the following: a previous frame of the first frame is a voiced frame, and a number of audio streams currently decoded by an audio decoder does not exceed a preset threshold; and obtaining a historical audio frame stored in advance before and continuous with the first frame.

3. The method of claim 2, wherein a preset packet loss compensation algorithm is included in the audio decoder; and the method further comprises: if the current condition does not meet the preset condition, performing packet loss compensation on the first frame using the preset packet loss compensation algorithm included in the audio decoder.

4. The method of claim 1, wherein the reconstructing the audio data of the second frame based on the second predicted audio data and the audio data of the second frame included in the audio comprises: adding the second predicted audio data and a target segment of the audio data of the second frame included in the audio after windowing operation to obtain audio data corresponding to the reconstructed second frame.

5. The method of claim 1, wherein the neural network model is a time domain model, and the time domain model is trained by the following way: inputting a first sample historical frame corresponding to a first sample frame in a first sample audio into a to-be-trained time domain model, predicting the first sample frame by the to-be-trained time domain model to obtain first sample predicted audio data for the first sample frame and first partial predicted audio data corresponding to a target segment of a next frame of the first sample frame; based on the first partial predicted audio data and partial audio data corresponding to the target segment of the next frame of the first sample frame in the first sample audio, reconstructing the partial audio data corresponding to the target segment of the next frame of the first sample frame to obtain first partial reconstructed audio data; taking the audio data of the first sample frame in the first sample audio and the partial audio data corresponding to the target segment of the next frame of the first sample frame as first labels, and taking the first sample predicted audio data and the first partial reconstructed audio data as first predicted data; and ​ adjust model parameters of the to-be-trained time domain model based on a difference loss of the first label and the first prediction data.

6. The method of claim 1, wherein the neural network model is a frequency domain model; and the frequency domain model is trained by the following ways: performing short-time Fourier transform (STFT) on a second sample history frame corresponding to a second sample frame in a second sample audio to obtain a history STFT coefficient corresponding to the second sample history frame; inputting the history STFT coefficient into a to-be-trained frequency domain model to obtain first prediction STFT coefficients, wherein the first prediction STFT coefficients include a second prediction STFT coefficient corresponding to the second sample frame and a partial prediction STFT coefficient corresponding to partial audio data corresponding to a target segment of a next frame of the second sample frame; performing inverse short-time Fourier transform (iSTFT) based on the first prediction STFT coefficients to obtain second sample prediction audio data of the second sample frame and second partial prediction audio data corresponding to the target segment of the next frame of the second sample frame; reconstructing the partial audio data corresponding to the target segment of the next frame of the second sample frame based on the second partial prediction audio data and the partial audio data corresponding to the target segment of the next frame of the second sample frame in the second sample audio to obtain second partial reconstruction audio data; taking the audio data of the second sample frame and the audio data corresponding to the target segment of the next frame of the second sample frame in the second sample audio as a second label and taking the second sample prediction audio data and the second partial reconstruction audio data as second prediction data; and adjusting model parameters of the to-be-trained frequency domain model based on a difference loss of the second label and the second prediction data.

7. The method of any one of claims 1 to 6, the method being performed by an audio decoder, and the obtaining, for a first frame of audio that is lost, of a history audio frame that is stored in advance and that is continuous with the first frame, comprises: the audio decoder, when playing to the first frame of the audio, after determining that the first frame is a frame that is lost, obtaining, for a first frame of audio that is lost, a history audio frame that is stored in advance and that is continuous with the first frame; and the method further comprises that the audio decoder plays the first frame based on the first prediction audio data and plays the second frame based on the reconstructed audio data of the second frame.

8. An apparatus for processing lost audio, comprising: an obtaining unit configured to, for a first frame of audio that is lost, obtain a history audio frame that is stored in advance and that is continuous with the first frame; a prediction unit configured to, based on the history audio frame, predict, by a pre-trained neural network model, the first frame to obtain first prediction audio data of the first frame and second prediction audio data corresponding to a target segment of a second frame of the audio, the target segment being connected in time with the first frame; and and The reconstruction unit is configured to reconstruct the audio data of the second frame based on the second predicted audio data and the audio data of the second frame included in the audio data. 9.A computer readable storage medium having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-7. 10.An electronic device comprising a memory and a processor, the memory having stored therein executable code, the processor implementing the method of any one of claims 1-7 when executing the executable code.

Images