Audio frame supplementing method and device and storage medium

Abstract

The invention discloses an audio frame supplementing method and device and a storage medium, and relates to the technical field of audio analysis, and the method comprises the steps: determining a corresponding pitch period and a sampling number according to a previous audio frame and a next audio frame of a lost audio frame, and then carrying out the sampling of the lost audio frame according to the pitch period, the sampling number and a target number; determining a target number and a target length of reconstructed audio blocks, then determining a first coefficient and a second coefficient of the previous audio frame and the next audio frame based on a discrete cosine transform algorithm, and determining the target number of reconstructed audio blocks according to the first coefficient and the second coefficient, and finally, intercepting a target audio block with the target length from the reconstructed audio block, and inserting the target audio block into the audio frame. According to the method, the audio frame missing in the middle is reconstructed through the previous audio frame and the next audio frame of the lost audio frame and is inserted into the audio frame, so that the cost for processing the packet loss of the audio data is reduced.

Description

Technical Field

[0001] The present application relates to the field of audio analysis technology, and in particular to an audio frame interpolation method, device and storage medium. Background Art

[0002] At present, during the transmission of audio data, audio data packets are queued at the inlet side and in the intermediate switching nodes. After a certain period of time, the queued audio data packets are either abandoned by the intermediate nodes or treated as lost by the receiving end. In order to ensure the complete transmission of audio data, a data request is usually sent to the transmission party to transmit the lost audio data packets to the receiving end. However, the above method requires additional network bandwidth, which makes the related scheme more costly when dealing with audio data packet loss.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The present application provides an audio frame interpolation method, device and storage medium, aiming to solve the problem of high cost in related solutions when processing audio data packet loss.

[0005] To achieve the above object, the present application provides an audio frame interpolation method, which comprises the following steps:

[0006] Determine the corresponding pitch period and sampling quantity according to the previous audio frame and the next audio frame of the lost audio frame;

[0007] Determining a target number and a target length of reconstructed audio blocks according to the pitch period, the sampling number and the target number, wherein the target number is the number of frames of the lost audio frames;

[0008] Determine a first coefficient and a second coefficient of the previous audio frame and the next audio frame based on a discrete cosine transform algorithm;

[0009] Determining the target number of the reconstructed audio blocks according to the first coefficient and the second coefficient;

[0010] A target audio block of the target length is cut out from the reconstructed audio block and inserted into the audio data.

[0011] In one embodiment, before the step of determining the corresponding pitch period and the number of samples according to the previous audio frame and the next audio frame of the lost audio frame, the step further includes:

[0012] If audio frame loss is detected, determining the lost audio frame and the corresponding target number;

[0013] According to the lost audio frame, the corresponding previous audio frame and the next audio frame are determined.

[0014] In one embodiment, before the step of determining the corresponding pitch period and the number of samples according to the previous audio frame and the next audio frame of the lost audio frame, the step further includes:

[0015] Determining target coefficients after discrete cosine transformation according to the previous audio frame;

[0016] Windowing the target coefficient according to a Hamming window function to obtain a windowed target function;

[0017] Calculate the root mean square energy and zero crossing rate of the previous audio frame based on the objective function;

[0018] Determining, according to the root mean square energy and the zero-crossing rate, that the previous audio frame is speech;

[0019] Execute the step of determining the corresponding pitch period and sampling quantity according to the previous audio frame and the next audio frame of the lost audio frame.

[0020] In one embodiment, the step of determining that the previous audio frame is speech according to the root mean square energy and the zero crossing rate comprises:

[0021] Determining a ratio of the root mean square energy to the zero crossing rate according to the root mean square energy and the zero crossing rate;

[0022] When the ratio is greater than a preset threshold, the previous audio frame is determined to be speech.

[0023] In one embodiment, the step of determining the corresponding pitch period and the number of samples according to the previous audio frame and the next audio frame of the lost audio frame comprises:

[0024] Decoding the previous audio frame and the next audio frame to obtain audio samples;

[0025] Determining the number of the audio samples, and determining the number as the sampling number;

[0026] The pitch period is determined according to the previous audio frame and the next audio frame.

[0027] In one embodiment, the step of determining the pitch period according to the previous audio frame and the next audio frame comprises:

[0028] Get the peak detection function;

[0029] Solving the peak detection function according to the previous audio frame and the next audio frame to obtain a target value;

[0030] Based on the target value, the pitch period is determined.

[0031] In one embodiment, before the step of cutting out the target audio block of the target length from the reconstructed audio block and inserting it into the audio data, the method further includes:

[0032] Acquire an adjacent audio block in the reconstructed audio block, wherein the adjacent audio block consists of a first audio block and a second audio block, and is used to represent two adjacent audio blocks;

[0033] Determining, according to the first audio block and the second audio block, whether the adjacent audio blocks match;

[0034] When the adjacent audio blocks match, the adjacent audio block is determined as the target audio block.

[0035] In one embodiment, the step of determining whether the adjacent audio blocks match according to the first audio block and the second audio block includes:

[0036] Acquire first data at a first position in the first audio block, and acquire second data at a second position in the second audio block;

[0037] When the first data matches the second data, it is determined that the adjacent audio blocks match.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides an audio frame interpolation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the audio frame interpolation method as described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the audio frame interpolation method as described above are implemented.

[0040] The present application provides an audio frame interpolation method, an audio frame interpolation device, and a storage medium. According to the previous audio frame and the next audio frame of the lost audio frame, the corresponding pitch period and the number of samples are determined, and then the target number and the target length of the reconstructed audio blocks are determined according to the pitch period, the number of samples, and the target number, wherein the target number is the number of frames of the lost audio frame, and then based on the discrete cosine transform algorithm, the first coefficient and the second coefficient of the previous audio frame and the next audio frame are determined, and according to the first coefficient and the second coefficient, the target number of the reconstructed audio blocks is determined, and finally the target audio block of the target length is intercepted from the reconstructed audio block and inserted into the audio frame. The present application reconstructs the missing audio frame in the middle through the previous audio frame and the next audio frame of the lost audio frame, and inserts it into the audio frame, thereby reducing the cost of processing audio data packet loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 This is a flowchart of a first embodiment of the audio frame interpolation method of the present application;

[0044] Figure 2 This is a flow chart of a second embodiment of the audio frame interpolation method of the present application;

[0045] Figure 3 This is a flow chart of a third embodiment of the audio frame interpolation method of the present application;

[0046] Figure 4 This is a schematic diagram of the architecture of the hardware operating environment of the audio frame interpolation method device involved in the embodiment of the present application.

[0047] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0050] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0051] The main solution of the present application is: determine the corresponding pitch period and sampling number according to the previous audio frame and the next audio frame of the lost audio frame; determine the target number and target length of the reconstructed audio blocks according to the pitch period, the sampling number and the target number, wherein the target number is the number of frames of the lost audio frame; determine the first coefficient and the second coefficient of the previous audio frame and the next audio frame based on the discrete cosine transform algorithm; determine the target number of the reconstructed audio blocks according to the first coefficient and the second coefficient; and cut the target audio block of the target length from the reconstructed audio block and insert it into the audio data.

[0052] At present, during the transmission of audio data, audio data packets are queued at the inlet side and in the intermediate switching nodes. After a certain period of time, the queued audio data packets are either abandoned by the intermediate nodes or treated as lost by the receiving end. In order to ensure the complete transmission of audio data, a data request is usually sent to the transmission party to transmit the lost audio data packets to the receiving end. However, the above method requires additional network bandwidth, which makes the related scheme more costly when dealing with audio data packet loss.

[0053] The present application determines the corresponding pitch period and number of samples based on the previous audio frame and the next audio frame of the lost audio frame, and then determines the target number and target length of the reconstructed audio blocks based on the pitch period, the number of samples and the target number, wherein the target number is the number of frames of the lost audio frame, and then determines the first coefficient and the second coefficient of the previous audio frame and the next audio frame based on the discrete cosine transform algorithm, and determines the target number of the reconstructed audio blocks based on the first coefficient and the second coefficient, and finally intercepts the target audio block of the target length from the reconstructed audio block and inserts it into the audio frame. The present application reconstructs the missing audio frame in the middle through the previous audio frame and the next audio frame of the lost audio frame, and inserts it into the audio frame, thereby reducing the cost of processing audio data packet loss.

[0054] It should be noted that the execution subject of this embodiment may be an audio frame interpolation system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an audio frame interpolation device capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the audio frame interpolation system as an example to illustrate this embodiment and the following embodiments.

[0055] Embodiment 1

[0056] Please refer to Figure 1 In a first embodiment, the audio frame interpolation method comprises the following steps:

[0057] Step S10: Determine the corresponding pitch period and sampling quantity according to the previous audio frame and the next audio frame of the lost audio frame.

[0058] In this embodiment, an audio interpolation system performs processing actions. During the transmission of audio data, the audio interpolation system detects the transmission process in real time. When it is detected that audio data is lost, the corresponding entire audio frame is obtained. The previous audio frame and the next audio frame of the lost audio frame are obtained in the entire audio frame. The pitch period is the reciprocal of the pitch frequency. It should be noted that pitch is an attribute of sound, which is used to represent the height of the sound or the height of the tone, and is mainly determined by the frequency of the vibration of the object. The corresponding pitch period and the number of samples are determined by the previous audio frame and the next audio frame.

[0059] Specifically, the audio signal features are extracted from the previous audio frame and the next audio frame. First, the audio signal is preprocessed, such as denoising, framing, etc. Then, the pitch period extraction algorithm is selected, including the time domain method, the frequency domain method and the hybrid method. Among them, the time domain method includes the autocorrelation method, the average amplitude difference method, etc. The time domain method directly determines the pitch period from the speech waveform. The frequency domain method includes the cepstrum method. The frequency domain method transforms the speech signal into the frequency domain to calculate the pitch period. The hybrid method is a method that combines the time domain and the frequency domain. First, the signal channel model parameters are extracted, and then the signal is filtered to obtain the sound source sequence. Finally, the time domain method is used to obtain the pitch period.

[0060] After determining the pitch period extraction algorithm, the audio signals of the previous audio frame and the next audio frame are processed according to the selected pitch detection method to calculate their pitch periods. In the process of determining the number of samples, the sampling rate of the audio signal needs to be determined first. The sampling rate defines the number of samples extracted from the continuous signal and composed of discrete signals per second. According to the format of the audio signal (such as AAC, MP3, etc.) and the sampling rate, the number of samples per frame of the audio signal can be determined. For example, for the AAC format, a frame usually contains 1024 sampling points; for the MP3 format, a frame contains 1152 sampling points. Since the duration of the lost audio frame is unknown, it can be estimated by the duration of the previous audio frame and the next audio frame and the interval between them. Assuming that the audio signal is continuous and uniform, the number of samples of the lost audio frame can be estimated based on this assumption. If the duration of the lost audio frame is known, the number of samples of the lost audio frame can be directly calculated based on the sampling rate.

[0061] Optionally, in this embodiment, the step of determining the corresponding pitch period and the number of samples according to the previous audio frame and the next audio frame of the lost audio frame includes:

[0062] Decode the previous audio frame and the next audio frame to obtain audio samples; determine the number of the audio samples and determine the number as the sampling number; determine the pitch period according to the previous audio frame and the next audio frame.

[0063] Specifically, first prepare the corresponding decoder according to the format of the audio frame, such as AAC, MP3, etc. You can use a library such as FFmpeg, which provides decoding functions for multiple audio formats. Then assign a context to the decoder and set the necessary parameters, such as the number of channels, sampling rate, sampling bits, etc., which can be obtained from the header information of the audio frame. Then read the raw data of the previous audio frame and the next audio frame and parse them using the decoder. The raw data can be packaged into AVPacket and then sent to the decoder for processing. After the decoder processes the AVPacket, it will generate decoded audio samples in the form of AVFrame. These audio samples can be received by calling functions such as avcodec_receive_frame.

[0064] After receiving the AVFrame, the audio samples stored in the AVFrame are obtained. The AVFrame contains multiple fields, among which the linesize field indicates the number of bytes occupied by the audio samples of each channel. According to the linesize field of the AVFrame and the number of bits of the audio samples, such as 16 bits, 32 bits, etc., the number of samples of each channel can be calculated. Then, according to the number of channels, the total number of samples can be calculated. The calculation formula is: total number of samples = linesize / number of sample bits / number of channels. Linesize can represent the number of bytes occupied by all the samples of a single channel. In addition, according to the format and sampling rate of the audio frame, it can be verified whether the calculated number of samples is reasonable. For example, for the AAC format, a frame usually contains 1024 sampling points; for the MP3 format, a frame may contain 1152 sampling points. If the calculated number of samples is significantly different from these values, it is necessary to recheck the decoding process or parameter settings. Finally, after calculating the number of samples, the pitch period is determined according to the previous audio frame and the next audio frame.

[0065] Further, the step of determining the pitch period according to the previous audio frame and the next audio frame comprises:

[0066] Acquire a peak detection function; solve the peak detection function according to the previous audio frame and the next audio frame to obtain a target value; and determine the pitch period based on the target value.

[0067] Specifically, the previous audio frame and the next audio frame are preprocessed, such as denoising, filtering and normalization, to improve the accuracy of peak detection. A weight function is selected, which assigns different weights according to the delay value. Assume that the processed previous audio frame is represented by y[n] (1≤n≤m), where m is the number of samples. Based on the above audio frame, the peak detection function is solved, where the peak detection function is a peak detection function based on the weighted autocorrelation principle, and the specific formula is:

[0068]

[0069] Then there is The target value is T(p). When T(p) reaches its maximum value, the value of the independent variable p is the pitch period of frame y.

[0070] Optionally, in this embodiment, before step S10, the following steps are further included:

[0071] If audio frame loss is detected, the lost audio frame and the corresponding target number are determined; and according to the lost audio frame, the corresponding previous audio frame and the next audio frame are determined.

[0072] Specifically, first, it is necessary to obtain audio data from the audio source. And pre-process the obtained audio data, including decoding and denoising. Divide the audio data into frames, and detect the integrity of each frame. Specifically, it is possible to detect whether the frame is lost by comparing the frame sequence number, check frame or other frame integrity identifier. After detecting the frame loss, mark the lost frame, and record its frame number and loss time. Count the number of lost frames, that is, the target number, which can be achieved by traversing the marked lost frames. Then locate the position of the lost frame in the audio data according to the frame number of the lost frame. Before the lost frame, find the nearest complete frame, that is, the previous audio frame, and after the lost frame, find the nearest complete frame, that is, the next audio frame.

[0073] Step S20: determining a target number and a target length of reconstructed audio blocks according to the pitch period, the sampling number and the target number, wherein the target number is the number of frames of the lost audio frames.

[0074] In this embodiment, based on step S10, it can be known that the pitch period is the larger period of the pitch period in samples of frame A and frame B, which is used as the pitch period. The high pitch period is then determined as the target length of the reconstructed audio block, and the number of target blocks is calculated. The target number refers to the number of lost audio frames, and the lost audio frames are the audio parts that need to be reconstructed.

[0075] Specifically, based on the pitch period and the sampling rate, it is possible to calculate how many sampling points each audio block should contain. The pitch period provides the basic rhythm of the audio signal, while the sampling rate determines how many sample points can be collected per second. Considering the continuity and smoothness of the audio signal, the size of the audio block should contain at least two basic periods to ensure that the characteristics of the speech can be displayed. Based on the target number (the number of frames of the lost audio frames) and the size of each audio block, the number of audio blocks that need to be reconstructed can be calculated.

[0076] Optionally, the larger of the pitch periods in samples of frame A and frame B is taken. Then the number of blocks that need to be reconstructed for the lost frame l is:

[0077] The length of the reconstructed block is

[0078] Step S30: Determine the first coefficient and the second coefficient of the previous audio frame and the next audio frame based on a discrete cosine transform algorithm.

[0079] In this embodiment, the discrete cosine transform (DCT) algorithm is a transform related to the Fourier transform, which is similar to the discrete Fourier transform (DFT), but only uses real numbers for calculation. The discrete cosine transform has the property of orthogonal transformation, and the basis vectors of its transformation matrix can describe the relevant characteristics of human speech signals. Through the discrete cosine transform algorithm, the first coefficient and the second coefficient are determined in the calculation of the previous audio frame and the next audio frame, wherein the first coefficient and the second coefficient correspond to the calculation of the previous audio frame and the next audio frame respectively.

[0080] Specifically, in the discrete cosine transform process, the audio file is first read, and the original signal data of the previous audio frame and the next audio frame are extracted. According to the actual application requirements, the appropriate DCT type is selected. The DCT transform is applied to the signal data of the previous audio frame and the next audio frame respectively. Specifically, it can be achieved by calling the DCT transform function (such as the dct function in MATLAB). The DCT transform converts the audio signal from the time domain to the frequency domain to obtain a series of transform coefficients. The required first coefficient and second coefficient are extracted from the DCT transform result.

[0081] Optionally, assume that the previous audio frame is A and the next audio frame is B. The DCT coefficient of the data in the last pitch period of frame A after discrete cosine transformation (DCT) is A[k] (1≤k≤m), and the DCT coefficient of the data in the first pitch period of frame B after DCT is B[k] (1≤k≤m), where m is the number of samples.

[0082] Step S40: Determine the target number of the reconstructed audio blocks according to the first coefficient and the second coefficient.

[0083] In this embodiment, after obtaining the first coefficient and the second coefficient corresponding to the previous audio frame and the next audio frame based on the discrete cosine transform algorithm, the reconstructed audio block is obtained by inverse discrete cosine transform according to a preset formula.

[0084] Specifically, it is assumed that the first coefficient is A[k] (1≤k≤m) and the second coefficient is B[k] (1≤k≤m). Then the i-th reconstructed audio block It is based on the following formula:

[0085]

[0086] That is, it is obtained by inverse discrete cosine transform, where A[k] is the first coefficient and B[k] is the second coefficient. i represents the i-th reconstructed audio block, Nb is the target number of reconstructed audio blocks, and m is the number of samples.

[0087] Step S50: intercepting a target audio block of the target length from the reconstructed audio block, and inserting the target audio block into the audio data.

[0088] In this embodiment, after obtaining the target number of reconstructed audio blocks, the target length of the lost audio frame is obtained. According to the target length, the target audio block of the target length is intercepted in the reconstructed audio block. The target audio block is inserted into the audio data, which is the data transmitted during the transmission process, and is the audio data obtained after parsing the transmission data. It should be noted that the audio data here is the audio data of the packet lost during the transmission process.

[0089] As an optional implementation method of intercepting an audio block of a target length, the length of the target audio block is first determined, that is, the length of the intercepted audio block is determined. The starting position and the ending position of the target audio block in the reconstructed audio block are determined. Then, it is ensured that the reconstructed audio block has been generated according to the previous steps and stored in an appropriate format and position. In the reconstructed audio block, the starting point of the target audio block is located according to the preset starting position. Starting from the starting point, the audio segment is intercepted according to the target length to form the target audio block.

[0090] The intercepted target audio block is then saved to an appropriate storage location to facilitate subsequent acquisition. In the original audio frame or the processed audio frame, the insertion position of the target audio block is determined. The intercepted target audio block is inserted into the determined insertion position. During the insertion process, attention should be paid to the continuity and smoothness of the audio signal to avoid abrupt transition effects. If there is an obvious transition trace between the insertion position and the original audio frame, cross-fade, interpolation or other audio processing techniques can be used to smooth the transition area and improve the overall quality of the audio.

[0091] In the technical solution provided in this embodiment, the corresponding pitch period and sampling number are first determined according to the previous audio frame and the next audio frame of the lost audio frame, and then the target number and target length of the reconstructed audio block are determined according to the pitch period, the sampling number and the target number, wherein the target number is the number of frames of the lost audio frame, and then the first coefficient and the second coefficient of the previous audio frame and the next audio frame are determined based on the discrete cosine transform algorithm, and the target number of the reconstructed audio blocks is determined according to the first coefficient and the second coefficient, and finally the target audio block of the target length is intercepted from the reconstructed audio block and inserted into the audio frame. The solution of this embodiment reconstructs the missing audio frame in the middle through the previous audio frame and the next audio frame of the lost audio frame, and inserts it into the audio frame, thereby reducing the cost of processing audio data packet loss.

[0092] In addition, the related scheme calculates the lost audio frames through interpolation methods such as spectrum interpolation. In the actual processing process, it is difficult to align the phase, and waveform matching will introduce additional delays, and the processing effect of the untoned signal is poor. The related scheme constructs the audio frame through the insertion and replacement method, which will have an obvious sense of interruption and the audio frame filling effect is poor. Based on this, the scheme in the first embodiment of the present application can also improve the effect of audio frame filling.

[0093] Embodiment 2

[0094] Please refer to Figure 2 In the second embodiment, before the step of determining the corresponding pitch period and the number of samples according to the previous audio frame and the next audio frame of the lost audio frame, it also includes:

[0095] Step S60: Determine the target coefficient after discrete cosine transformation according to the previous audio frame.

[0096] Step S70: windowing the target coefficient according to the Hamming window function to obtain a windowed target function.

[0097] In this embodiment, the previous audio frame can be used as judgment data and as basic data for subsequent processing. The next audio frame can also be used as basic data for subsequent processing, or any audio frame in the audio data excluding the lost audio frame. The previous audio frame is processed by a discrete cosine transform algorithm to obtain a transformed target coefficient. Then, a preset Hamming window function is obtained, and the target coefficient is windowed based on the Hamming window function to obtain a windowed target function.

[0098] Specifically, the Hamming window is a window function of finite width, which has non-zero values ​​within a specified interval and is zero outside the interval. Windowing refers to multiplying the target coefficient with the Hamming window function to obtain the target function.

[0099] Step S80: Calculating the root mean square energy and zero-crossing rate of the previous audio frame based on the objective function.

[0100] Step S90: Determine whether the previous audio frame is speech based on the RMS energy and the zero-crossing rate.

[0101] Step S100: Execute the step of determining the corresponding pitch period and sampling quantity according to the previous audio frame and the next audio frame of the lost audio frame.

[0102] In this embodiment, after the target function is calculated by the Hamming window function and the previous audio frame, the root mean square energy and the zero-crossing rate of the previous audio frame are calculated according to the target function. Both the root mean square energy and the zero-crossing rate have corresponding formulas, and specific values ​​can be calculated in combination with the target function. The root mean square energy is used to describe the energy or intensity level of a signal over a period of time. It is obtained by calculating the average of the square values ​​of the signal and then taking the square root. The zero-crossing rate refers to the ratio of the sign change of a signal, that is, the ratio of the number of times the signal changes from a positive number to a negative number or from a negative number to a positive number to the total length of the signal.

[0103] Specifically, the zero-crossing rate is calculated by first dividing the signal into multiple short-time windows, each of which is called a frame. This is to divide the signal into smaller parts for processing, so as to capture the local characteristics of the signal. Then, for each frame of the signal, determine whether the signs of adjacent sampling points are opposite. If the signs are opposite, it means that the signal has passed through the zero point at this point, that is, there is a zero-crossing point. Finally, the number of zero-crossing points in each frame is divided by the frame length (that is, the number of sampling points) to obtain the zero-crossing rate of the frame.

[0104] For example, assuming that the basic data to be processed is the previous audio frame y[n] (1≤n≤m), its DCT coefficient is Y[n] (1≤n≤m), and the Hamming window function is:

[0105]

[0106] The windowed DCT coefficient is Yw[n] (1≤n≤m; Yw[n]=Y[n]×w[n]). The root mean square energy and zero crossing rate of the audio frame can be calculated from the windowed DCT coefficient. The calculation formula of the root mean square energy is:

[0107]

[0108] The formula for calculating the zero-crossing rate is:

[0109]

[0110] Finally, after the root mean square energy and the zero-crossing rate are calculated, it is determined whether the previous audio frame is speech according to the root mean square energy and the zero-crossing rate. When the previous audio frame is speech, the step of determining the corresponding pitch period and the number of samples according to the previous audio frame and the next audio frame of the lost audio frame is performed.

[0111] Optionally, in this embodiment, the step of determining that the previous audio frame is speech according to the root mean square energy and the zero crossing rate includes:

[0112] According to the root mean square energy and the zero-crossing rate, a ratio of the root mean square energy to the zero-crossing rate is determined; when the ratio is greater than a preset threshold, the previous audio frame is determined to be speech.

[0113] Specifically, after the root mean square energy and the zero-crossing rate are calculated, the root mean square energy and the zero-crossing rate are divided to obtain a ratio. Then, a preset threshold is obtained, wherein the preset threshold can be set according to specific needs, and the ratio is compared with the preset threshold. If the ratio is greater than the preset threshold, it is determined that the previous audio frame is voice, that is, the corresponding audio frame is voice, otherwise it is not voice, that is, silence.

[0114] It should be noted that when the audio frame is silent, the step of determining the corresponding pitch period and the number of samples according to the previous audio frame and the next audio frame of the lost audio frame is not performed in the above steps. It is necessary to reacquire another audio frame and use the above method to determine whether it is voice. If the audio frame is voice, the above steps are performed again.

[0115] In the technical solution provided in this embodiment, the target coefficient after discrete cosine transformation is determined according to the previous audio frame, and then the target coefficient is windowed according to the Hamming window function to obtain the windowed target function, and then the root mean square energy and zero crossing rate of the previous audio frame are calculated based on the target function, and according to the root mean square energy and the zero crossing rate, the previous audio frame is determined to be speech, and finally the step of determining the corresponding pitch period and sampling number according to the previous audio frame and the next audio frame of the lost audio frame is performed. This embodiment improves the accuracy of audio frame supplementation by judging whether the previous audio frame is speech and performing subsequent steps when the audio frame is speech.

[0116] Embodiment 3

[0117] Please refer to Figure 3 In the third embodiment, before the step of cutting out the target audio block of the target length from the reconstructed audio block and inserting it into the audio data, the method further includes:

[0118] Step S110: Acquire adjacent audio blocks in the reconstructed audio block, wherein the adjacent audio blocks are composed of a first audio block and a second audio block, and are used to represent two adjacent audio blocks.

[0119] Step S120: determining whether the adjacent audio blocks match according to the first audio block and the second audio block.

[0120] Step S130: When the adjacent audio blocks match, determine the adjacent audio block as the target audio block.

[0121] In this embodiment, before the target audio block of the target length is intercepted in the reconstructed audio block, adjacent audio blocks in the reconstructed audio block are first obtained, wherein the adjacent audio blocks are composed of a first audio block and a second audio block, and are used to represent two adjacent audio blocks. According to the matching degree of the first audio block and the second audio block, it is determined whether the adjacent audio blocks match. And when the adjacent audio blocks match, the adjacent audio block is determined as the target audio block.

[0122] Specifically, a loop iterator is used to traverse the reconstructed audio block list. For each audio block, according to its position in the list, its adjacent audio blocks are found, usually the previous and next audio blocks. These two adjacent audio blocks are respectively used as the first audio block and the second audio block. Then, according to actual needs, the conditions for matching adjacent audio blocks are defined. Specifically, they include the similarity of audio data, the continuity of timestamps, the matching of audio features, etc. According to the defined matching conditions, the matching degree between the first audio block and the second audio block is calculated. Specifically, it can be achieved through steps such as comparison of audio data, feature extraction and similarity calculation. The calculated matching degree is compared with a preset threshold. If the matching degree is greater than or equal to the threshold, the adjacent audio blocks are considered to match; otherwise, they are considered to be mismatched. For the matched adjacent audio blocks, they are added to Blocki as target audio blocks.

[0123] Optionally, in this embodiment, the step of determining whether the adjacent audio blocks match according to the first audio block and the second audio block includes:

[0124] First data at a first position is obtained in the first audio block, and second data at a second position is obtained in the second audio block; when the first data matches the second data, it is determined that the adjacent audio blocks match.

[0125] Specifically, the first data at the first position in the first audio block and the second data at the second position in the second audio block can be obtained, and the first data and the second data can be compared. When the two match, it is determined that the adjacent audio blocks match. It should be noted that the first position here can be the head of the first audio block, and the second position can be the tail of the second audio block. It depends on the positions of the first audio block and the second audio block in the reconstructed audio block. When the position of the first audio block is after the second audio block, the position relationship is as described above, otherwise the position relationship is opposite.

[0126] During the comparison process, in order to ensure continuity of the inserted audio data, the first data and the second data should be identical.

[0127] In the technical solution provided in this embodiment, adjacent audio blocks in the reconstructed audio block are obtained, the adjacent audio blocks are composed of a first audio block and a second audio block, and are used to represent two adjacent audio blocks. Then, according to the first audio block and the second audio block, it is determined whether the adjacent audio blocks match. Finally, when the adjacent audio blocks match, the adjacent audio block is determined as the target audio block. By determining whether the adjacent audio blocks match, the accuracy of the intercepted audio blocks is improved, and the accuracy of reconstructing the lost audio frame is further improved.

[0128] Since the system introduced in the embodiment of the present application is a system used to implement the method of the embodiment of the present application, based on the method introduced in the embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems used in the method of the embodiment of the present application belong to the scope of protection of this application.

[0129] The present application provides an audio frame interpolation method device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the audio frame interpolation method in the above-mentioned embodiment 1.

[0130] Reference below Figure 4 , which shows a schematic diagram of the structure of an audio frame interpolation method device suitable for implementing the embodiment of the present application. The audio frame interpolation method device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The audio frame interpolation method and device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0131] like Figure 4As shown, the audio frame interpolation method device may include a processing device 1001 (e.g., a core processor, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the audio frame interpolation method device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the audio frame interpolation method device to communicate wirelessly or wired with other devices to exchange data. Although the audio frame interpolation method device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0132] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0133] The audio frame interpolation method device provided by the present application adopts the audio frame interpolation method in the above embodiment, which can solve the technical problem that the related solution has high cost when processing audio data packet loss. Compared with the prior art, the beneficial effects of the audio frame interpolation method device provided by the present application are the same as the beneficial effects of the audio frame interpolation method provided by the above embodiment, and the other technical features in the audio frame interpolation method device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0134] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0135] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0136] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the audio frame interpolation method in the above-mentioned embodiment.

[0137] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0138] The computer-readable storage medium may be included in the audio frame interpolation method device; or may exist independently without being assembled into the audio frame interpolation method device.

[0139] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the audio frame interpolation method device, the audio frame interpolation method device: determines the corresponding pitch period and sampling number according to the previous audio frame and the next audio frame of the lost audio frame; determines the target number and target length of the reconstructed audio blocks according to the pitch period, the sampling number and the target number, wherein the target number is the number of frames of the lost audio frame; determines the first coefficient and the second coefficient of the previous audio frame and the next audio frame based on the discrete cosine transform algorithm; determines the target number of the reconstructed audio blocks according to the first coefficient and the second coefficient; intercepts the target audio block of the target length in the reconstructed audio block, and inserts it into the audio data.

[0140] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0143] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned audio frame interpolation method, and can solve the technical problem that the related solutions have high costs when processing audio data packet loss. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the audio frame interpolation method provided by the above-mentioned embodiment, and will not be repeated here.

[0144] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned audio frame interpolation method when executed by a processor.

[0145] The computer program product provided by the present application can solve the technical problem that the related solutions have high costs when processing audio data packet loss. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the audio frame supplementation method provided by the above embodiment, which will not be repeated here.

[0146] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. An audio frame interpolation method, characterized in that: The audio frame interpolation method comprises the following steps: Determine the corresponding pitch period and sampling quantity according to the previous audio frame and the next audio frame of the lost audio frame; Determining a target number and a target length of reconstructed audio blocks according to the pitch period, the sampling number and the target number, wherein the target number is the number of frames of the lost audio frames; Determine a first coefficient and a second coefficient of the previous audio frame and the next audio frame based on a discrete cosine transform algorithm; Determining the target number of the reconstructed audio blocks according to the first coefficient and the second coefficient; A target audio block of the target length is cut out from the reconstructed audio block and inserted into the audio data.

2. The method according to claim 1, characterized in that Before the step of determining the corresponding pitch period and the number of samples according to the previous audio frame and the next audio frame of the lost audio frame, the method further includes: If audio frame loss is detected, determining the lost audio frame and the corresponding target number; According to the lost audio frame, the corresponding previous audio frame and the next audio frame are determined.

3. The method according to claim 1, characterized in that Before the step of determining the corresponding pitch period and the number of samples according to the previous audio frame and the next audio frame of the lost audio frame, the method further includes: Determining target coefficients after discrete cosine transformation according to the previous audio frame; Windowing the target coefficient according to a Hamming window function to obtain a windowed target function; Calculate the root mean square energy and zero crossing rate of the previous audio frame based on the objective function; Determining, according to the root mean square energy and the zero-crossing rate, that the previous audio frame is speech; Execute the step of determining the corresponding pitch period and sampling quantity according to the previous audio frame and the next audio frame of the lost audio frame.

4. The method according to claim 3, characterized in that The step of determining that the previous audio frame is speech according to the root mean square energy and the zero crossing rate comprises: Determining a ratio of the root mean square energy to the zero crossing rate according to the root mean square energy and the zero crossing rate; When the ratio is greater than a preset threshold, the previous audio frame is determined to be speech.

5. The method according to claim 1, characterized in that The step of determining the corresponding pitch period and the number of samples according to the previous audio frame and the next audio frame of the lost audio frame comprises: Decoding the previous audio frame and the next audio frame to obtain audio samples; Determining the number of the audio samples, and determining the number as the sampling number; The pitch period is determined according to the previous audio frame and the next audio frame.

6. The method according to claim 5, characterized in that The step of determining the pitch period according to the previous audio frame and the next audio frame comprises: Get the peak detection function; Solving the peak detection function according to the previous audio frame and the next audio frame to obtain a target value; Based on the target value, the pitch period is determined.

7. The method according to claim 1, characterized in that Before the step of intercepting the target audio block of the target length from the reconstructed audio block and inserting it into the audio data, the method further includes: Acquire an adjacent audio block in the reconstructed audio block, wherein the adjacent audio block consists of a first audio block and a second audio block, and is used to represent two adjacent audio blocks; Determining, according to the first audio block and the second audio block, whether the adjacent audio blocks match; When the adjacent audio blocks match, the adjacent audio block is determined as the target audio block.

8. The method according to claim 7, characterized in that The step of determining whether the adjacent audio blocks match according to the first audio block and the second audio block comprises: Acquire first data at a first position in the first audio block, and acquire second data at a second position in the second audio block; When the first data matches the second data, it is determined that the adjacent audio blocks match.

9. An audio frame interpolation device, characterized in that: The audio frame interpolation device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the audio frame interpolation method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the audio frame interpolation method according to any one of claims 1 to 8 are implemented.

Images