Audio processing method, apparatus and electronic device

Abstract

The application relates to an audio processing method, device and electronic equipment. The method comprises: performing audio repair by using an audio repair strategy based on an audio type of a target audio frame; wherein the target audio frame is determined according to frame types of a current audio frame and a previous audio frame; the frame types comprise normal audio frames, packet loss audio frames and repaired audio frames; the audio types comprise human voice clear tone, human voice turbidity and music tone; the audio repair strategy is used to generate repair frame data according to the audio type of the target audio frame; and the repair frame data is used to replace audio frames or data fusion. The application can improve the continuity of repaired audio, thereby improving the auditory experience of a user.

Description

Technical Field

[0001] This application relates to the field of audio data processing technology, and in particular to an audio processing method, apparatus and electronic device. Background Technology

[0002] During network transmission, audio is susceptible to factors such as network latency, bandwidth fluctuations, and electromagnetic interference, inevitably resulting in packet loss. Packet loss can cause audio stuttering and disconnections at the receiving end, severely impacting the listening experience. Therefore, audio packet loss repair technology is needed to replenish lost audio data, thereby achieving "seamless" packet loss handling.

[0003] However, traditional audio packet loss repair solutions directly replace lost data with a mute signal, which can easily cause noticeable stuttering and disconnection, disrupting audio continuity and affecting the listening experience. Summary of the Invention

[0004] Therefore, it is necessary to provide an audio processing method, apparatus, device, and program product that can improve the continuity of repaired audio in response to the above-mentioned technical problems.

[0005] Firstly, this application provides an audio processing method, the method comprising:

[0006] Based on the audio type of the target audio frame, audio restoration strategies are used for audio restoration.

[0007] The target audio frame is determined based on the frame type of the current audio frame and the previous audio frame; the frame type includes normal audio frames, lost audio frames, and repaired audio frames; the audio type includes unvoiced human voices, voiced human voices, and musical sounds; the audio repair strategy is used to generate repair frame data based on the audio type of the target audio frame; the repair frame data is used to replace audio frames or for data fusion.

[0008] In one embodiment, audio repair is performed using an audio repair strategy based on the audio type of the target audio frame, including:

[0009] If the current audio frame is a lost audio frame and the previous audio frame is a normal audio frame, then the current audio frame is taken as the target audio frame, and the audio type of the target audio frame is determined.

[0010] Execute an audio repair strategy that matches the audio type of the target audio frame to obtain repaired frame data;

[0011] Replace the current audio frame with the repaired frame data.

[0012] In one embodiment, determining the audio type of the target audio frame includes:

[0013] Based on the high-frequency and full-frequency signals of historical cached audio data, a signal classification strategy is used to determine the audio type corresponding to the target audio frame.

[0014] Among them, the historical cached audio data is the audio data closest to the target audio frame with a preset cache duration.

[0015] In one embodiment, the signal classification strategy includes:

[0016] Obtain the signal energy ratio between the high-frequency signal and the full-frequency signal;

[0017] Based on the comparison between the signal energy ratio and the signal classification threshold, it is determined whether the audio type of the target audio frame is voiced human voice.

[0018] In one embodiment, the signal classification strategy further includes:

[0019] If it is determined that the audio type of the target audio frame is not voiced human voice, then obtain the zero-crossing rate and inter-frame correlation of each frame corresponding to the historical cached audio data.

[0020] If there are two consecutive frames in the historical cached audio data where the inter-frame correlation is greater than the preset correlation threshold, the audio type of the target audio frame is determined based on the zero-crossing rate.

[0021] If no two consecutive frames in the historical cached audio data have an inter-frame correlation greater than the preset correlation threshold, or if the inter-frame correlation of each frame in the historical cached audio data is less than or equal to the preset correlation threshold, then the audio type of the target audio frame is determined to be human voice unfiltered audio.

[0022] In one embodiment, the audio restoration strategy includes:

[0023] If the target audio frame is a human voice unvoiced audio, then the unvoiced audio restoration strategy is executed.

[0024] If the target audio frame has a voiced human voice, then the voiced audio restoration strategy is executed.

[0025] If the target audio frame has a musical tone as its audio type, then the musical tone repair strategy will be executed.

[0026] In one embodiment, the unvoiced sound restoration strategy includes:

[0027] Obtain the first AR coefficient corresponding to the first audio data; wherein, the first audio data represents the audio data of the first preset duration that is closest to the target audio frame;

[0028] The first excitation source is obtained by inverse filtering the audio frame preceding the target audio frame using the first AR coefficient.

[0029] Using the last audio data in the audio frame preceding the target audio frame as the initial value of the filter, the first excitation source is filtered using the first AR coefficient to obtain the first synthesized signal.

[0030] The first synthesized signal is attenuated using a fast attenuation factor to obtain repaired frame data.

[0031] In one embodiment, the voiced sound restoration strategy includes:

[0032] Retrieve associated audio frame data from historical cached audio data where the inter-frame correlation is greater than a preset correlation threshold;

[0033] If the peak position in the associated audio frame data is within the multiple error range, then obtain the waveform copy data of the target audio frame, as well as the pitch direction trend and pitch precision direction index corresponding to the associated audio frame data;

[0034] If the pitch direction trend is consistent with the pitch precision direction index, then the pitch precision direction index is used as the pitch direction of the repair frame.

[0035] Based on the pitch direction of the repaired frame, the second AR coefficient corresponding to the second audio data is used to filter the second excitation source corresponding to the waveform copy data to obtain the second synthesized signal; wherein, the second audio data represents the audio data of the second preset duration closest to the target audio frame;

[0036] The second synthesized signal is attenuated using a fast attenuation factor to obtain repaired frame data.

[0037] In one embodiment, the voiced sound restoration strategy further includes:

[0038] If the pitch direction trend is inconsistent with the pitch precise direction index, the waveform copy data is interpolated using a preset interpolation factor to obtain interpolated waveform data.

[0039] The third excitation source corresponding to the interpolated waveform data is filtered using the second AR coefficient to obtain the third synthesized signal;

[0040] The third synthesized signal is attenuated using a fast attenuation factor to obtain repaired frame data.

[0041] In one embodiment, the voiced sound restoration strategy further includes:

[0042] If the peak position in the associated audio frame data is outside the multiple error range, the audio type of the target audio frame will be switched to human voice unvoiced, and the unvoiced repair strategy will be executed.

[0043] In one embodiment, the music restoration strategy includes:

[0044] Obtain the first spectrum corresponding to the third audio data and the second spectrum corresponding to the fourth audio data; wherein, the third audio data represents the audio data of the third preset duration that is closest to the target audio frame; the fourth audio data represents the audio data of the fourth preset duration that is closest to the target audio frame and whose inter-frame correlation is greater than a preset correlation judgment threshold;

[0045] Based on the preset sub-band division strategy, the mean amplitude and variance of the first spectrum are obtained;

[0046] A third spectrum that conforms to the mean and variance of amplitude is randomly generated;

[0047] The third spectrum and the second spectrum are weighted and fused based on a preset weight ratio to obtain the fourth spectrum;

[0048] Repair frame data is generated using the fourth spectrum and the slow decay factor.

[0049] In one embodiment, audio repair is performed using an audio repair strategy based on the audio type of the target audio frame, and further includes:

[0050] If the current audio frame is a lost audio frame and the previous audio frame is a repaired audio frame, then set the audio type of the current audio frame to be the same as the audio type of the previous audio frame.

[0051] The current audio frame is used as the target audio frame. An audio repair strategy matching the audio type of the target audio frame is executed to obtain repaired frame data.

[0052] Replace the target audio frame with the repaired frame data.

[0053] In one embodiment, audio repair is performed using an audio repair strategy based on the audio type of the target audio frame, and further includes:

[0054] If the current audio frame is a normal audio frame and the previous audio frame is a repaired audio frame, then obtain the audio volume of the previous audio frame and use the previous audio frame as the target audio frame.

[0055] If the audio volume has not decayed to zero, an audio repair strategy matching the audio type of the target audio frame is executed to obtain repaired frame data, and the repaired frame data is then fused with the current audio frame for output.

[0056] If the audio volume has decayed to zero, the target audio frame will be merged with the current audio frame for output.

[0057] Secondly, this application provides an audio processing apparatus, the apparatus comprising:

[0058] The audio restoration unit performs audio restoration based on the audio type of the target audio frame using an audio restoration strategy.

[0059] The target audio frame is determined based on the frame type of the current audio frame and the previous audio frame; the audio type includes unvoiced human voice, voiced human voice, and musical voice; the audio restoration strategy is used to generate restoration frame data based on the audio type of the target audio frame; the restoration frame data is used to replace audio frames or perform data fusion.

[0060] Thirdly, this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any one of the first aspects.

[0061] The aforementioned audio processing method, apparatus, and electronic device can determine the target audio frame based on the frame type of the current audio frame and the previous audio frame. Then, based on the audio type of the target audio frame, an audio repair strategy is used to generate repair frame data according to the audio type of the target audio frame, so as to achieve audio repair through the repair frame data. In this way, the continuity of the repaired audio can be improved, thereby significantly improving the user's auditory experience in the case of packet loss. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is an application environment diagram of an audio processing method in one embodiment;

[0064] Figure 2 This is a flowchart illustrating an audio processing method in one embodiment;

[0065] Figure 3 This is a schematic diagram of the first process for audio repair in one embodiment;

[0066] Figure 4 This is a first schematic diagram of historical cached audio data in one embodiment;

[0067] Figure 5 This is a schematic diagram of the first process of a signal classification strategy in one embodiment;

[0068] Figure 6 This is a schematic diagram of the second process of a signal classification strategy in one embodiment;

[0069] Figure 7 This is a second schematic diagram of historical cached audio data in one embodiment;

[0070] Figure 8 This is a flowchart illustrating an audio restoration strategy in one embodiment;

[0071] Figure 9 This is a flowchart illustrating the unclear sound restoration strategy in one embodiment;

[0072] Figure 10 This is a schematic diagram illustrating the execution process of the clear sound restoration strategy in one embodiment;

[0073] Figure 11 This is a schematic diagram of the first process of a voiced sound restoration strategy in one embodiment;

[0074] Figure 12 This is a schematic diagram illustrating the execution process of a voiced sound restoration strategy in one embodiment;

[0075] Figure 13 This is a schematic diagram of the second process of a voiced sound restoration strategy in one embodiment;

[0076] Figure 14 This is a flowchart illustrating a musical tone restoration strategy in one embodiment;

[0077] Figure 15 This is a schematic diagram illustrating the execution process of a music sound restoration strategy in one embodiment;

[0078] Figure 16 This is a schematic diagram of the second process for audio repair in one embodiment;

[0079] Figure 17 This is a schematic diagram of the third process for audio repair in one embodiment;

[0080] Figure 18 This is a third schematic diagram of historical cached audio data in one embodiment;

[0081] Figure 19 This is a structural block diagram of an audio processing device in one embodiment;

[0082] Figure 20 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0084] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0085] The audio processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on the cloud or other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0086] In one exemplary embodiment, such as Figure 2 As shown, an audio processing method is provided, which is applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps S202. Wherein:

[0087] Step S202: Based on the audio type of the target audio frame, perform audio repair using an audio repair strategy.

[0088] The target audio frame is determined based on the frame type of the current audio frame and the previous audio frame; the frame type includes normal audio frames, lost audio frames, and repaired audio frames; the audio type includes unvoiced human voices, voiced human voices, and musical sounds; the audio repair strategy is used to generate repair frame data based on the audio type of the target audio frame; the repair frame data is used to replace audio frames or for data fusion.

[0089] For example, the receiving end of the terminal can receive the audio data collected by the sensor in real time, and then perform frame processing according to a preset specification (e.g., 10ms / frame), while caching the most recent valid audio frames of a preset duration (e.g., 50ms) in a set storage area.

[0090] In practical applications, the terminal can determine whether packet loss has occurred through methods such as data packet sequence number verification and timeout detection. If the terminal detects packet loss, it immediately initiates the audio repair process and records information such as the location and duration of packet loss. It should be noted that the methods for implementing packet loss detection described above are not limited to the implementation methods mentioned in the above embodiments, and this application does not specifically limit the method for implementing packet loss detection.

[0091] Specifically, the terminal can use audio repair strategies to repair audio data that has experienced packet loss, based on the audio type of the target audio frame.

[0092] Understandably, in phonetics and acoustics, voiceless human voices in the audio type are a type of consonant pronunciation, the core characteristic of which is that the vocal cords do not vibrate during pronunciation; voiced human voices in the audio type are also a type of consonant pronunciation, the core characteristic of which is that the vocal cords vibrate during pronunciation; while musical sounds in the audio type refer generally to signals that are not purely human voices, such as the non-human voice parts in an audio clip containing background music. Furthermore, due to the different pronunciation mechanisms, voiceless and voiced human voices determine whether the sound signals are periodic signals.

[0093] In some examples, the audio type corresponding to an audio frame can be determined based on a signal classification strategy. In some possible implementations, the signal classification strategy can distinguish signal types based on spectral distribution, zero-crossing rate statistical distribution, and waveform correlation, thereby determining the audio type corresponding to the audio frame. It should be noted that in practical applications, other signal classification methods can also be used to determine the audio type corresponding to an audio frame, and this application does not specifically limit the method of determining the audio type.

[0094] For example, the target audio frame can be determined based on the frame types of the current audio frame and the previous audio frame. For instance, if the current audio frame is a lost audio frame and the previous audio frame is a normal audio frame, then the current audio frame is used as the target audio frame. Alternatively, if the current audio frame is a lost audio frame and the previous audio frame is a repaired audio frame, then the audio type of the current audio frame is set to the same as the audio type of the previous audio frame, and the current audio frame is used as the target audio frame. Or, if the current audio frame is a normal audio frame and the previous audio frame is a repaired audio frame, then the previous audio frame is used as the target audio frame.

[0095] In practical applications, audio restoration strategies can generate corresponding restoration frame data based on the audio type of the target audio frame, thereby repairing lost audio packets using this data. For example, audio restoration strategies may include unvoiced tone restoration strategies, voiced tone restoration strategies, and musical tone restoration strategies.

[0096] For example, the clear audio restoration strategy may include: obtaining the first AR coefficient corresponding to the first audio data; performing inverse filtering on the previous audio frame using the first AR coefficient to obtain the first excitation source; using the last audio data in the previous audio frame as the initial value of the filter, and performing filtering processing on the first excitation source using the first AR coefficient to obtain the first synthesized signal; and performing attenuation processing on the first synthesized signal using a fast attenuation factor to obtain the restored frame data; wherein, the first audio data represents the audio data of the first preset duration closest to the target audio frame.

[0097] For example, the voiced sound restoration strategy may include: acquiring associated audio frame data from historical cached audio data where the inter-frame correlation is greater than a preset correlation threshold; if the peak position in the associated audio frame data is within the multiple error range, acquiring waveform copy data of the target audio frame, as well as the pitch direction trend and pitch precision direction index corresponding to the associated audio frame data; if the pitch direction trend is consistent with the pitch precision direction index, using the pitch precision direction index as the pitch direction of the restoration frame; according to the pitch direction of the restoration frame, using the second AR coefficient corresponding to the second audio data to filter the second excitation source corresponding to the waveform copy data to obtain a second synthesized signal; using a fast attenuation factor to attenuate the second synthesized signal to obtain the restoration frame data; wherein, the second audio data represents the audio data closest to the target audio frame with a second preset duration.

[0098] For example, the music restoration strategy may include: obtaining a first spectrum corresponding to the third audio data and a second spectrum corresponding to the fourth audio data; obtaining the mean amplitude and variance of the first spectrum according to a preset sub-band division strategy; randomly generating a third spectrum that conforms to the mean amplitude and variance; weighting and fusing the third spectrum and the second spectrum based on a preset weight ratio to obtain a fourth spectrum; and generating restoration frame data using the fourth spectrum and a slow attenuation factor; wherein the third audio data represents the audio data of the third preset duration that is closest to the target audio frame; and the fourth audio data represents the audio data of the fourth preset duration that is closest to the target audio frame and whose inter-frame correlation is greater than a preset correlation judgment threshold.

[0099] The aforementioned audio processing method determines the target audio frame based on the frame types of the current and previous audio frames. Then, based on the audio type of the target audio frame, an audio restoration strategy is used to generate restoration frame data, thereby achieving audio restoration through the restoration frame data. Compared to traditional audio restoration schemes with low continuity (such as silence filling and waveform duplication), this application uses differentiated processing to ensure that the restoration of human voice audio frames does not feel mechanically repetitive, while the restoration of musical audio frames can conform to the natural dynamic changes of the audio. This allows the user's subjective listening experience to remain close to the original audio even in the presence of packet loss, thus significantly improving the user's listening experience.

[0100] In one embodiment, such as Figure 3 As shown, based on the audio type of the target audio frame, an audio repair strategy is used to perform audio repair, including the following steps S302 to S304. Wherein:

[0101] Step S302: If the current audio frame is a lost audio frame and the previous audio frame is a normal audio frame, then the current audio frame is taken as the target audio frame, and the audio type of the target audio frame is determined.

[0102] Step S304: Execute the audio repair strategy that matches the audio type of the target audio frame to obtain repaired frame data.

[0103] Step S302: Replace the current audio frame with the repaired frame data.

[0104] For example, such as Figure 4 As shown, Figure 4 The terminal caches the most recent exemplary historical audio data in the designated storage area with a preset duration of 50ms. At this time, the current audio frame is a lost audio frame and the previous audio frame is a normal audio frame.

[0105] For example, when the current audio frame is used as the target audio frame, if the current audio frame is determined to be a human voiceless tone, the voiceless tone restoration strategy in the audio restoration strategy can be executed; if the current audio frame is determined to be a human voiced tone, the voiced tone restoration strategy in the audio restoration strategy can be executed; if the current audio frame is determined to be a musical tone, the musical tone restoration strategy in the audio restoration strategy can be executed.

[0106] Specifically, when the current audio frame is a lost audio frame and the previous audio frame is a normal audio frame, the terminal can use the current audio frame as the target audio frame, determine the audio type of the target audio frame, and then execute an appropriate audio repair strategy based on the audio type of the target audio frame to obtain repaired frame data. This application, by implementing differentiated audio repair strategies, can effectively improve the continuity of the repaired audio, thereby significantly improving the user's auditory experience in the event of packet loss.

[0107] In one embodiment, determining the audio type of the target audio frame includes the following steps: based on the high-frequency signal and full-frequency signal of historical cached audio data, using a signal classification strategy to determine the audio type corresponding to the target audio frame.

[0108] Among them, the historical cached audio data can be the audio data closest to the target audio frame with a preset cache duration.

[0109] For example, the preset buffer duration can be set to 50ms, and the duration of each valid audio frame in the historical buffered audio data can be set to 10ms. It should be noted that the buffer duration and frame division method of the historical buffered audio data can be adaptively adjusted according to actual application scenarios, device conditions, and other factors. This application embodiment does not impose specific limitations on the buffer duration and frame division method of the historical buffered audio data.

[0110] In some possible implementations, signal classification strategies can distinguish signal types based on spectral distribution, zero-crossing rate statistical distribution, and waveform correlation, thereby determining the audio type corresponding to the audio frame.

[0111] Specifically, the terminal can determine the audio type corresponding to the target audio frame by using a signal classification strategy based on the high-frequency and full-frequency signals of historical cached audio data.

[0112] In one embodiment, such as Figure 5 As shown, the signal classification strategy includes the following steps S402 to S404. Wherein:

[0113] Step S402: Obtain the signal energy ratio between the high-frequency signal and the full-frequency signal.

[0114] Step S404: Based on the comparison result between the signal energy ratio and the signal classification judgment threshold, determine whether the audio type of the target audio frame is voiced human voice.

[0115] For example, based on historical buffered audio data, the high-frequency signal (above 1kHz) and the full-frequency signal corresponding to the historical buffered audio data can be separated using an IIR (Infinite Impulse Response) high-pass filter. Optionally, the cutoff frequency of the IIR high-pass filter can be 1kHz.

[0116] In some examples, the signal energy ratio (energyHigh / energyTotal) between the high-frequency signal and the full-frequency signal is compared with a signal classification threshold (set to T1). If the signal energy ratio is less than or equal to the signal classification threshold T1, the audio type of the target audio frame is determined to be voiced human voice; if the signal energy ratio is greater than the signal classification threshold T1, further classification of unvoiced and musical tones is required. Optionally, the signal classification threshold T1 can be set to 0.6.

[0117] Specifically, the terminal can obtain the signal energy ratio between the high-frequency signal and the full-frequency signal. If the signal energy ratio is less than or equal to the signal classification threshold, the target audio frame is determined to be a voiced human voice.

[0118] In one embodiment, such as Figure 6 As shown, the signal classification strategy further includes the following steps S502 to S506. Wherein:

[0119] Step S502: If it is determined that the audio type of the target audio frame is not voiced human voice, then the zero-crossing rate and inter-frame correlation of each frame corresponding to the historical cached audio data are obtained.

[0120] Step S504: If there are two consecutive frames in the historical cached audio data where the inter-frame correlation is greater than the preset correlation judgment threshold, then the audio type of the target audio frame is determined based on the zero-crossing rate.

[0121] Step S506: If no two consecutive frames in the historical cached audio data have an inter-frame correlation greater than the preset correlation determination threshold, or if the inter-frame correlation of each frame in the historical cached audio data is less than or equal to the preset correlation determination threshold, then the audio type of the target audio frame is determined to be human voice unfiltered audio.

[0122] The zero-crossing rate refers to the number of times the audio signal crosses zero within the statistical window. For example, with each audio frame lasting 10ms, the statistical window size could be set to 50ms. Figure 7 As shown. It should be noted that during audio repair, the audio frame at the packet loss location (i.e., the lost audio frame) is considered unusable. In this case, the historical cached audio data refers to the audio data before the packet loss location for a preset cache duration.

[0123] For example, inter-frame correlation can refer to the coherence coefficient between consecutive frames of a valid audio frame; wherein, the coherence coefficient between consecutive frames can be the ratio between the self-power spectrum and the cross-power spectrum of consecutive frames of a valid audio frame. Optionally, the preset correlation determination threshold T2 can be set to 0.7.

[0124] In some examples, the audio type of the target audio frame is determined based on the zero-crossing rate of the historical cached audio data. This may include: if the zero-crossing rate is lower than a preset zero-crossing rate threshold, the target audio frame can be identified as a musical tone; if the zero-crossing rate is higher than the preset zero-crossing rate threshold, the target audio frame can be identified as a human voice.

[0125] Specifically, if the terminal determines that the audio type of the target audio frame is not voiced human voice, it can obtain the zero-crossing rate corresponding to the historical cached audio data and the inter-frame correlation of each valid audio frame in the historical cached audio data. If there are two consecutive frames in the historical cached audio data where the inter-frame correlation is greater than the preset correlation judgment threshold (i.e., T2), the audio type of the target audio frame is determined based on the zero-crossing rate of the historical cached audio data. If there are no two consecutive frames in the historical cached audio data where the inter-frame correlation is greater than the preset correlation judgment threshold, or if the inter-frame correlation of each valid audio frame in the historical cached audio data is less than or equal to the preset correlation judgment threshold, the audio type of the target audio frame can be determined to be unvoiced human voice. It can be understood that the above classification mechanism of this application, based on the three-level classification method of "high-frequency energy proportion + zero-crossing rate + correlation", can accurately distinguish between unvoiced human voice, voiced human voice, and musical sounds, thereby laying the foundation for subsequent differentiated repair.

[0126] In one embodiment, such as Figure 8 As shown, the audio restoration strategy may include the following steps S602 to S606. Wherein:

[0127] Step S602: If the audio type of the target audio frame is human voice unvoiced, then execute the unvoiced sound repair strategy.

[0128] Step S604: If the audio type of the target audio frame is voiced human voice, then execute the voiced sound repair strategy.

[0129] Step S606: If the audio type of the target audio frame is musical tone, then execute the musical tone repair strategy.

[0130] For example, audio restoration strategies may include unvoiced tone restoration strategies, voiced tone restoration strategies, and musical tone restoration strategies.

[0131] Specifically, when the terminal determines that the audio type of the target audio frame is unvoiced human voice, it executes an unvoiced sound restoration strategy; when it determines that the audio type of the target audio frame is voiced human voice, it executes a voiced sound restoration strategy; and when it determines that the audio type of the target audio frame is musical sound, it executes a musical sound restoration strategy. By executing differentiated audio restoration strategies, this application can effectively improve the continuity of the restored audio, thereby significantly improving the user's auditory experience in the event of packet loss.

[0132] In one embodiment, such as Figure 9As shown, the clear sound restoration strategy includes the following steps S702 to S706. Wherein:

[0133] Step S702: Obtain the first AR coefficient corresponding to the first audio data; wherein, the first audio data represents the audio data of the first preset duration that is closest to the target audio frame.

[0134] Step S704: Perform inverse filtering on the previous audio frame of the target audio frame using the first AR coefficient to obtain the first excitation source.

[0135] Step S706: Using the last audio data in the audio frame preceding the target audio frame as the initial value of the filter, the first excitation source is filtered using the first AR coefficient to obtain the first synthesized signal.

[0136] Step S708: Attenuate the first synthesized signal using a fast attenuation factor to obtain repaired frame data.

[0137] The final audio data can refer to the data of a preset number of actual audio sampling points closest to the target audio frame. Optionally, the preset number can be the AR filter order corresponding to the first AR coefficient. Optionally, the first preset duration can be set to 40ms. Optionally, the AR (AutoRegressive) coefficients can be estimated using the Yule-Waler equation method.

[0138] For example, the fast decay factor can be set as a linear coefficient that causes the audio signal to decay to -90dB over 80ms.

[0139] Specifically, with Figure 10 Taking the illustrated schematic diagram of an exemplary unvoiced audio frame restoration process as an example, the terminal can extract the corresponding first AR coefficient from the first audio data. Then, it uses this first AR coefficient to perform inverse filtering on the preceding audio frame of the target audio frame to obtain a first excitation source. Next, the last audio data in the preceding audio frame is used as the initial value of the AR filter. Based on this initial value, the first excitation source is filtered using the first AR coefficient to obtain a first synthesized signal. Finally, a fast attenuation factor is used to attenuate the first synthesized signal, ultimately obtaining restored frame data with unvoiced human voice. It is understood that by using the first AR coefficient to filter the first excitation source in this embodiment, the repetitive characteristics of the audio signal can be effectively destroyed, thereby adding natural variations to reduce the unnaturalness when introducing restored frame data, and thus significantly improving the user's auditory experience.

[0140] In one embodiment, such as Figure 11 As shown, the voiced sound restoration strategy includes the following steps S802 to S810. Wherein:

[0141] Step S802: Obtain associated audio frame data from historical cached audio data where the inter-frame correlation is greater than a preset correlation judgment threshold.

[0142] Step S804: If the peak position in the associated audio frame data is within the multiple error range, then obtain the waveform copy data of the target audio frame, as well as the pitch direction trend and pitch precision direction index corresponding to the associated audio frame data.

[0143] Step S806: If the pitch direction trend is consistent with the pitch precision direction index, then the pitch precision direction index is used as the pitch direction of the repair frame.

[0144] Step S808: Based on the pitch direction of the repair frame, the second excitation source corresponding to the waveform copy data is filtered using the second AR coefficient corresponding to the second audio data to obtain the second synthesized signal; wherein, the second audio data represents the audio data of the second preset duration closest to the target audio frame.

[0145] Step S810: Attenuate the second synthesized signal using a fast attenuation factor to obtain repaired frame data.

[0146] For example, taking a peak statistical sliding window with 400 sampling points as an example, the peak may appear at positions such as "100", "201", and "300". It is understandable that since the fundamental frequency of human voice fluctuates, the peak position often does not fall on an integer multiple of the fixed fundamental frequency. Therefore, this embodiment sets a multiple error range so that within this range, the peak position is considered to be periodic. For example, regarding the peak positions "100", "201", and "300" mentioned above, although the peak at position "201" exhibits a fluctuation of 1 amplitude, the peak position can still be considered periodic.

[0147] For example, the magnitude of the multiple error range can be adaptively set according to the jitter of the fundamental frequency of the human voice. Optionally, the error threshold corresponding to the multiple error range can be set to ±5% of the fundamental frequency period. The preset correlation judgment threshold T2 can be set to 0.7, the second preset duration can be set to 20ms, and the fast decay factor can be set to a linear coefficient that causes the audio signal to decay to -90dB after 80ms.

[0148] In some examples, the pitch direction trend and precise pitch direction index corresponding to the associated audio frame data can be obtained in the following ways: ① Equal weight confirmation: Calculate the difference between adjacent peak positions (i.e., p(j+1)-p(j), where p(j) is the peak position of the j-th peak). If the difference is greater than 0 (i.e., rising), record 1; if the difference is less than 0 (i.e., falling), record -1. Sum the above difference results to obtain the pitch direction trend. ② Numerical weighted confirmation: Take the median of the difference between adjacent peak positions, median(p(j+1)-p(j)), as the precise pitch direction index. It can be understood that if the pitch direction trend calculated above is consistent with the calculated precise pitch direction index, then the precise pitch direction index can be used as the pitch direction of the repair frame.

[0149] Specifically, with Figure 12 Taking an exemplary diagram of a voiced audio frame repair process as an example, the terminal obtains associated audio frame data from historical cached audio data where the inter-frame correlation is greater than a preset correlation judgment threshold (i.e., T2). If the peak position in the associated audio frame data is within the multiple error range, the terminal obtains the waveform copy data of the target audio frame, as well as the pitch direction trend and pitch precision direction index corresponding to the associated audio frame data. If the pitch direction trend is consistent with the pitch precision direction index, the terminal can use the pitch precision direction index as the pitch direction of the repair frame. Then, based on the pitch direction of the repair frame, the terminal uses the second AR coefficient extracted from the second audio data to filter the second excitation source corresponding to the waveform copy data to obtain the second synthesized signal. Then, the terminal uses a fast attenuation factor to attenuate the second synthesized signal to obtain the repair frame data.

[0150] In one embodiment, such as Figure 13 As shown, the voiced sound restoration strategy further includes the following steps S902 to S906. Wherein:

[0151] Step S902: If the pitch direction trend is inconsistent with the pitch precise direction index, then the waveform copy data is interpolated using a preset interpolation factor to obtain interpolated waveform data.

[0152] Step S904: Filter the third excitation source corresponding to the interpolated waveform data using the second AR coefficient to obtain the third synthesized signal.

[0153] Step S906: Attenuate the third synthesized signal using a fast attenuation factor to obtain repaired frame data.

[0154] For example, the preset interpolation factor can be set to inc_factor=1. In some examples, if the pitch direction trend is inconsistent with the precise pitch direction index, the preset interpolation factor inc_factor is used to perform waveform interpolation copying to obtain interpolated waveform data, thereby maintaining the original pitch direction of the audio data.

[0155] Specifically, with Figure 12 Taking the illustrated schematic diagram of an exemplary voiced audio frame repair process as an example, when the pitch direction trend and the pitch precision direction index are inconsistent, the terminal can use a preset interpolation factor (e.g., inc_factor=1) to interpolate the waveform copy data to obtain interpolated waveform data. Then, the second AR coefficient extracted from the second audio data is used to filter the third excitation source corresponding to the interpolated waveform data to further obtain the third synthesized signal. Finally, the third synthesized signal is attenuated using a fast attenuation factor to obtain the repaired frame data.

[0156] In one embodiment, the voiced sound restoration strategy further includes the following steps: if the peak position in the associated audio frame data is outside the multiple error range, the audio type of the target audio frame is switched to unvoiced human voice, and the unvoiced sound restoration strategy is executed.

[0157] It is understandable that if the peak position in the associated audio frame data is outside the multiple error range, it indicates that the periodicity of the associated audio frame data is relatively weak. In this case, it is more suitable to use the clear sound restoration strategy for audio restoration.

[0158] Specifically, with Figure 12 Taking the illustrated schematic diagram of an exemplary voiced audio frame repair process as an example, when the peak position in the associated audio frame data is outside the multiple error range, the terminal can convert the audio class of the target audio frame to unvoiced human voice and then use the unvoiced repair strategy for audio repair.

[0159] In one embodiment, such as Figure 14 As shown, the music sound restoration strategy includes the following steps S1002 to S1010. Wherein:

[0160] Step S1002: Obtain the first spectrum corresponding to the third audio data and the second spectrum corresponding to the fourth audio data; wherein, the third audio data represents the audio data of the third preset duration that is closest to the target audio frame; the fourth audio data represents the audio data of the fourth preset duration that is closest to the target audio frame and whose inter-frame correlation is greater than a preset correlation judgment threshold.

[0161] Step S1004: According to the preset sub-band division strategy, obtain the mean amplitude and variance of the first spectrum;

[0162] Step S1006: Randomly generate a third spectrum that conforms to the mean and variance of the amplitude.

[0163] Step S1008: The third spectrum and the second spectrum are weighted and fused based on a preset weight ratio to obtain the fourth spectrum.

[0164] Step S1010: Using the fourth spectrum and the slow attenuation factor, repair frame data is generated.

[0165] For example, waveform signals with temporal correlation (i.e., inter-frame correlation greater than a preset correlation threshold T2) can be found from historical cache data, and then the fourth audio data can be obtained by waveform copying.

[0166] It is understandable that by performing Fast Fourier Transform (FFT) processing on the third and fourth audio data respectively, the first and second spectra can be obtained respectively.

[0167] In some examples, the preset subband division strategy may include dividing the first spectrum into a preset number of subbands for statistical analysis of the mean and variance of the amplitude corresponding to the first spectrum. Optionally, the preset number of subbands may be set to 32, the third preset duration may be set to 50ms, the fourth preset duration may be set to 10ms, and the preset weight ratio may be set to 7:3.

[0168] For example, based on the amplitude mean corresponding to the first spectrum and variance It can randomly generate amplitude averages whose mean conforms to the first spectrum mentioned above. And the variance is within the variance of the first spectrum mentioned above. The third spectrum within the range.

[0169] In some examples, generating repaired frame data using a fourth spectrum and a slow attenuation factor can include: attenuating the fourth spectrum using the slow attenuation factor to obtain the repaired frame spectral distribution; and using an inverse fast Fourier transform (IFFT) to convert the repaired frame spectral distribution into time-domain repaired frame data. Optionally, the slow attenuation factor can be set to keep the audio signal at 1 (i.e., no attenuation) for 34ms, and then attenuate the audio signal to -90dB after 500ms following 34ms.

[0170] It should be noted that, due to the strong randomness of unvoiced human voice signals and their often short duration, repairing excessively long audio frames will not conform to the natural distribution and may result in unnatural sounds. Therefore, this application employs a rapid attenuation factor with exponential decay throughout the unvoiced sound restoration strategy. Musical tones and voiced human voices, on the other hand, are highly correlated signals with longer durations and greater predictability. Therefore, this application employs a slow attenuation factor with a relatively slower decay rate in the voiced sound restoration strategy and the musical sound restoration strategy. It can be understood that, through the design of the aforementioned attenuation factors, this application enables the restored frame data to more closely resemble the real audio data, thereby making the restored audio data more natural and significantly improving the user's listening experience.

[0171] Specifically, with Figure 15 Taking the illustrated schematic diagram of an exemplary musical audio frame repair process as an example, the terminal obtains the first spectrum corresponding to the third audio data and the second spectrum corresponding to the fourth audio data. Then, according to the preset sub-band division strategy, it obtains the amplitude mean and variance corresponding to the first spectrum. Based on the above amplitude mean and variance, it randomly generates a third spectrum that conforms to the amplitude mean and variance. Then, it weights and fuses the third spectrum and the second spectrum according to a preset weight ratio (e.g., 7:3) to obtain the fourth spectrum. Finally, it uses the fourth spectrum and a slow attenuation factor to generate repair frame data.

[0172] In some possible implementations, if the packet loss duration exceeds a preset packet loss duration (e.g., 100ms), the average amplitude corresponding to the first spectrum can be updated every preset statistical duration (e.g., 50ms). and variance To adapt to real-time changes in musical tones.

[0173] In one embodiment, such as Figure 16 As shown, based on the audio type of the target audio frame, audio repair is performed using an audio repair strategy, and further includes the following steps S1102 to S1106. Wherein:

[0174] Step S1102: If the current audio frame is a lost audio frame and the previous audio frame is a repaired audio frame, then set the audio type of the current audio frame to be the same as the audio type of the previous audio frame.

[0175] Step S1104: Take the current audio frame as the target audio frame, execute the audio repair strategy that matches the audio type of the target audio frame, and obtain the repaired frame data.

[0176] Step S1106: Replace the target audio frame with the repaired frame data.

[0177] It is understandable that if the current audio frame is a lost audio frame and the previous audio frame is a repaired audio frame, it means that the previous audio frame has already completed the audio frame repair through one of the branches of the audio repair strategy (i.e., unvoiced sound repair strategy, voiced sound repair strategy, musical sound repair strategy). However, if the current audio frame is still a lost audio frame, the audio repair strategy corresponding to the audio type of the previous audio frame can be directly used to repair the audio frame.

[0178] It should be noted that, due to the application of attenuation factors (i.e., the fast attenuation factor and the slow attenuation factor mentioned above) in the audio restoration strategy, when the packet loss time is long, after the audio restoration strategy is executed cyclically, the continuously generated restoration frame data will show continuous energy decay until the volume of the restoration frame data is zero.

[0179] Specifically, when the current audio frame is a lost audio frame and the previous audio frame is a repaired audio frame, the terminal sets the audio type of the current audio frame to be the same as the audio type of the previous audio frame, and uses the current audio frame as the target audio frame. Then, it executes an audio repair strategy that matches the audio type of the target audio frame to obtain repaired frame data, and then uses the repaired frame data to replace the target audio frame.

[0180] In one embodiment, such as Figure 17 As shown, based on the audio type of the target audio frame, audio repair is performed using an audio repair strategy, and further includes the following steps S1202 to S1206. Wherein:

[0181] Step S1202: If the current audio frame is a normal audio frame and the previous audio frame is a repaired audio frame, then obtain the audio volume of the previous audio frame and use the previous audio frame as the target audio frame.

[0182] In step S1204, if the audio volume has not decayed to zero, an audio repair strategy matching the audio type of the target audio frame is executed to obtain repair frame data, and the repair frame data is fused with the current audio frame for output.

[0183] Step S1206: If the audio volume has decayed to zero, the target audio frame is merged with the current audio frame and output.

[0184] Understandably, with Figure 18 Taking an exemplary audio signal data example, due to the presence of an attenuation factor in the audio restoration strategy, the volume corresponding to each restored audio frame will gradually decrease according to the audio restoration strategy. Figure 18 The volume of each repaired audio frame has the following relationship: V1 > V2 > V3.

[0185] For example, the repaired frame data and the target audio frame can be fused by linear weighting (e.g., y=1-x, where y is the weight corresponding to the repaired frame data and x is the weight corresponding to the target audio frame) to achieve a fused output.

[0186] Specifically, when the current audio frame is a normal audio frame and the previous audio frame is a repaired audio frame, the terminal obtains the audio volume of the previous audio frame and uses it as the target audio frame. Then, if the audio volume of the previous audio frame has not decayed to zero, an audio repair strategy matching the audio type of the target audio frame is executed to obtain repaired frame data, and the repaired frame data is fused with the current audio frame for output. If the audio volume of the previous audio frame has decayed to zero, the target audio frame is fused with the current audio frame for output. Through this method, this application can eliminate abrupt changes in the boundaries of the repaired audio data, thereby ensuring the smoothness of the audio restoration process and significantly improving the continuity of the repaired audio and the user's auditory experience.

[0187] It is understood that, compared with traditional audio restoration solutions, this application has at least the following beneficial technical effects: First, by classifying and differentiating audio types, this application makes the restoration of lost human voice audio data free of mechanical repetition, while the restoration of lost musical audio data conforms to natural dynamic changes, thus making the user's subjective listening experience close to real audio; Second, the audio classification logic of this application does not require the use of complex models and can meet the low latency requirements of real-time transmission scenarios in various devices; Furthermore, the audio processing method of this application is compatible with human voice (unvoiced and voiced), musical, and mixed audio scenarios, and can be effectively applied to various application scenarios such as voice calls, online live streaming, and music streaming; Furthermore, the audio processing method of this application can effectively solve the stuttering and disconnection problems in traditional audio restoration technologies through attenuation factor adjustment, inter-frame fusion, and transition frame design; In addition, this application can maintain stable restoration effects under different packet loss rates (1%-30%), thus enabling the audio processing method of this application to effectively adapt to packet loss audio processing in complex network environments.

[0188] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0189] Based on the same inventive concept, this application also provides an audio processing apparatus for implementing the audio processing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more audio processing apparatus embodiments provided below can be found in the limitations of the audio processing method described above, and will not be repeated here.

[0190] In one exemplary embodiment, such as Figure 18 As shown, this application provides an audio processing apparatus 1900, including: an audio repair unit 1902, used to perform audio repair based on the audio type of a target audio frame using an audio repair strategy; wherein, the target audio frame is determined according to the frame type of the current audio frame and the previous audio frame; the audio type includes unvoiced human voice, voiced human voice, and musical sound; the audio repair strategy is used to generate repair frame data according to the audio type of the target audio frame; the repair frame data is used to replace audio frames or perform data fusion.

[0191] In one embodiment, the audio repair unit 1902 is further configured to: if the current audio frame is a lost audio frame and the previous audio frame is a normal audio frame, then take the current audio frame as the target audio frame and determine the audio type of the target audio frame; execute an audio repair strategy that matches the audio type of the target audio frame to obtain repair frame data; and replace the current audio frame with the repair frame data.

[0192] In one embodiment, the audio repair unit 1902 is further configured to determine the audio type corresponding to the target audio frame based on the high-frequency signal and full-frequency signal of the historical cached audio data using a signal classification strategy; wherein, the historical cached audio data is the audio data closest to the target audio frame with a preset cache duration.

[0193] In one embodiment, the signal classification strategy includes: obtaining the signal energy ratio between the high-frequency signal and the full-frequency signal; and determining whether the audio type of the target audio frame is voiced human voice based on the comparison result between the signal energy ratio and the signal classification judgment threshold.

[0194] In one embodiment, the signal classification strategy further includes: if it is determined that the audio type of the target audio frame is not voiced human voice, then the zero-crossing rate and the inter-frame correlation of each frame corresponding to the historical cached audio data are obtained; if there are two consecutive frames in the historical cached audio data where the inter-frame correlation is greater than a preset correlation determination threshold, then the audio type of the target audio frame is determined based on the zero-crossing rate; if there are no two consecutive frames in the historical cached audio data where the inter-frame correlation is greater than the preset correlation determination threshold, or if the inter-frame correlation of each frame in the historical cached audio data is less than or equal to the preset correlation determination threshold, then the audio type of the target audio frame is determined to be unvoiced human voice.

[0195] In one embodiment, the audio restoration strategy includes: if the audio type of the target audio frame is unvoiced human voice, then an unvoiced restoration strategy is executed; if the audio type of the target audio frame is voiced human voice, then a voiced restoration strategy is executed; if the audio type of the target audio frame is musical sound, then a musical sound restoration strategy is executed.

[0196] In one embodiment, the voice clearing restoration strategy includes: obtaining a first AR coefficient corresponding to a first audio data; wherein the first audio data represents audio data of a first preset duration that is closest to the target audio frame; performing inverse filtering on the audio frame preceding the target audio frame using the first AR coefficient to obtain a first excitation source; using the last audio data in the audio frame preceding the target audio frame as the initial value of the filter, and performing filtering processing on the first excitation source using the first AR coefficient to obtain a first synthesized signal; and performing attenuation processing on the first synthesized signal using a fast attenuation factor to obtain restored frame data.

[0197] In one embodiment, the voiced sound restoration strategy includes: acquiring associated audio frame data from historical cached audio data where the inter-frame correlation is greater than a preset correlation threshold; if the peak position in the associated audio frame data is within a multiple error range, acquiring waveform copy data of the target audio frame, as well as the pitch direction trend and pitch precision direction index corresponding to the associated audio frame data; if the pitch direction trend is consistent with the pitch precision direction index, using the pitch precision direction index as the pitch direction of the restoration frame; based on the pitch direction of the restoration frame, using the second AR coefficient corresponding to the second audio data to filter the second excitation source corresponding to the waveform copy data to obtain a second synthesized signal; wherein, the second audio data represents the audio data closest to the target audio frame and of a second preset duration; and using a fast attenuation factor to attenuate the second synthesized signal to obtain the restoration frame data.

[0198] In one embodiment, the voiced sound restoration strategy further includes: if the pitch direction trend is inconsistent with the pitch precision direction index, then using a preset interpolation factor to interpolate the waveform copy data to obtain interpolated waveform data; using a second AR coefficient to filter the third excitation source corresponding to the interpolated waveform data to obtain a third synthesized signal; and using a fast attenuation factor to attenuate the third synthesized signal to obtain restored frame data.

[0199] In one embodiment, the voiced sound restoration strategy further includes: if the peak position in the associated audio frame data is outside the multiple error range, then the audio type of the target audio frame is switched to unvoiced human voice, and the unvoiced sound restoration strategy is executed.

[0200] In one embodiment, the music restoration strategy includes: acquiring a first spectrum corresponding to third audio data and a second spectrum corresponding to fourth audio data; wherein the third audio data represents audio data of a third preset duration that is closest to the target audio frame; and the fourth audio data represents audio data of a fourth preset duration that is closest to the target audio frame and whose inter-frame correlation is greater than a preset correlation threshold; obtaining the mean amplitude and variance of the first spectrum according to a preset sub-band division strategy; randomly generating a third spectrum that conforms to the mean amplitude and variance; weighting and fusing the third spectrum and the second spectrum based on a preset weight ratio to obtain a fourth spectrum; and generating restoration frame data using the fourth spectrum and a slow attenuation factor.

[0201] In one embodiment, the audio repair unit 1902 is further configured to: if the current audio frame is a lost audio frame and the previous audio frame is a repaired audio frame, set the audio type of the current audio frame to be the same as the audio type of the previous audio frame; use the current audio frame as the target audio frame, execute an audio repair strategy that matches the audio type of the target audio frame to obtain repaired frame data; and replace the target audio frame with the repaired frame data.

[0202] In one embodiment, the audio repair unit 1902 is further configured to: if the current audio frame is a normal audio frame and the previous audio frame is a repaired audio frame, obtain the audio volume of the previous audio frame and use the previous audio frame as the target audio frame; if the audio volume has not decayed to zero, execute an audio repair strategy that matches the audio type of the target audio frame to obtain repaired frame data, and fuse the repaired frame data with the current audio frame for output; if the audio volume has decayed to zero, fuse the target audio frame with the current audio frame for output.

[0203] Each module in the aforementioned audio processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0204] In one exemplary embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 20 As shown, this electronic device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface and input device are connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an audio processing method.

[0205] Those skilled in the art will understand that Figure 20 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0206] In one embodiment, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0207] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0208] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0209] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0210] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0211] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An audio processing method, characterized in that, The method includes: Based on the audio type of the target audio frame, audio restoration strategies are used for audio restoration. The target audio frame is determined based on the frame type of the current audio frame and the previous audio frame; the frame type includes normal audio frames, lost audio frames, and repaired audio frames; the audio type includes unvoiced human voice, voiced human voice, and musical sounds; the audio repair strategy is used to generate repair frame data based on the audio type of the target audio frame; the repair frame data is used to replace audio frames or perform data fusion.

2. The method according to claim 2, characterized in that, The audio repair strategy based on the audio type of the target audio frame includes: If the current audio frame is a lost audio frame and the previous audio frame is a normal audio frame, then the current audio frame is taken as the target audio frame, and the audio type of the target audio frame is determined. Execute the audio repair strategy that matches the audio type of the target audio frame to obtain the repaired frame data; The current audio frame is replaced with the repaired frame data.

3. The method according to claim 2, characterized in that, Determining the audio type of the target audio frame includes: Based on the high-frequency and full-frequency signals of historical cached audio data, the audio type corresponding to the target audio frame is determined using a signal classification strategy. The historical cached audio data refers to the audio data closest to the target audio frame with a preset cache duration.

4. The method according to claim 3, characterized in that, The signal classification strategy includes: Obtain the signal energy ratio between the high-frequency signal and the full-frequency signal; Based on the comparison between the signal energy ratio and the signal classification threshold, it is determined whether the audio type of the target audio frame is the voiced human voice.

5. The method according to claim 4, characterized in that, The signal classification strategy also includes: If it is determined that the audio type of the target audio frame is not the voiced human voice, then the zero-crossing rate and inter-frame correlation of each frame corresponding to the historical cached audio data are obtained. If there are two consecutive frames in the historical cached audio data where the inter-frame correlation is greater than a preset correlation threshold, then the audio type of the target audio frame is determined based on the zero-crossing rate. If no two consecutive frames in the historical cached audio data have an inter-frame correlation greater than the preset correlation threshold, or if the inter-frame correlation of each frame in the historical cached audio data is less than or equal to the preset correlation threshold, then the audio type of the target audio frame is determined to be the unvoiced human voice.

6. The method according to claim 1, characterized in that, The audio restoration strategy includes: If the audio type of the target audio frame is the human voice unvoiced, then the unvoiced restoration strategy is executed. If the audio type of the target audio frame is the voiced human voice, then the voiced voice repair strategy is executed; If the audio type of the target audio frame is musical tone, then the musical tone repair strategy is executed.

7. The method according to claim 6, characterized in that, The unrecorded sound restoration strategy includes: Obtain the first AR coefficient corresponding to the first audio data; the first audio data represents the audio data of the first preset duration closest to the target audio frame. The first excitation source is obtained by inverse filtering the previous audio frame of the target audio frame using the first AR coefficient. Using the last audio data in the audio frame preceding the target audio frame as the initial value of the filter, the first excitation source is filtered using the first AR coefficient to obtain the first synthesized signal. The first synthesized signal is attenuated using a fast attenuation factor to obtain the repaired frame data.

8. The method according to claim 6, characterized in that, The voiced sound restoration strategy includes: Retrieve associated audio frame data from historical cached audio data where the inter-frame correlation is greater than a preset correlation threshold; If the peak position in the associated audio frame data is within the multiple error range, then obtain the waveform copy data of the target audio frame, as well as the pitch direction trend and pitch precision direction index corresponding to the associated audio frame data. If the tone direction trend is consistent with the tone precision direction index, then the tone precision direction index is used as the tone direction of the repair frame; Based on the pitch direction of the repaired frame, the second AR coefficient corresponding to the second audio data is used to filter the second excitation source corresponding to the waveform copy data to obtain the second synthesized signal; wherein, the second audio data represents the audio data of the second preset duration closest to the target audio frame; The second synthesized signal is attenuated using a fast attenuation factor to obtain the repaired frame data.

9. The method according to claim 8, characterized in that, The voiced sound restoration strategy also includes: If the tone direction trend is inconsistent with the tone precise direction index, then the waveform copy data is interpolated using a preset interpolation factor to obtain interpolated waveform data. The third excitation source corresponding to the interpolated waveform data is filtered using the second AR coefficient to obtain the third synthesized signal; The third synthesized signal is attenuated using the fast attenuation factor to obtain the repaired frame data.

10. The method according to claim 9, characterized in that, The voiced sound restoration strategy also includes: If the peak position in the associated audio frame data is outside the multiple error range, the audio type of the target audio frame is switched to the human voice unvoiced, and the unvoiced repair strategy is executed.

11. The method according to claim 6, characterized in that, The musical tone restoration strategy includes: Obtain the first spectrum corresponding to the third audio data and the second spectrum corresponding to the fourth audio data; wherein, the third audio data represents the audio data of the third preset duration that is closest to the target audio frame; the fourth audio data represents the audio data of the fourth preset duration that is closest to the target audio frame and has an inter-frame correlation greater than a preset correlation determination threshold; According to the preset sub-band division strategy, the mean amplitude and variance of the first spectrum are obtained; A third spectrum is randomly generated that conforms to the mean amplitude and the variance. The third spectrum and the second spectrum are weighted and fused based on a preset weight ratio to obtain the fourth spectrum; The repair frame data is generated using the fourth spectrum and the slow attenuation factor.

12. The method according to any one of claims 1 to 11, characterized in that, The audio repair based on the audio type of the target audio frame, using an audio repair strategy, also includes: If the current audio frame is a lost audio frame and the previous audio frame is a repaired audio frame, then the audio type of the current audio frame is set to be the same as the audio type of the previous audio frame. The current audio frame is used as the target audio frame, and the audio repair strategy that matches the audio type of the target audio frame is executed to obtain repaired frame data; The target audio frame is replaced using the repaired frame data.

13. The method according to any one of claims 1 to 11, characterized in that, The audio repair based on the audio type of the target audio frame, using an audio repair strategy, also includes: If the current audio frame is a normal audio frame and the previous audio frame is a repaired audio frame, then obtain the audio volume of the previous audio frame and use the previous audio frame as the target audio frame. If the audio volume does not decay to zero, the audio repair strategy matching the audio type of the target audio frame is executed to obtain repaired frame data, and the repaired frame data is fused with the current audio frame for output. If the audio volume has decayed to zero, the target audio frame and the current audio frame are merged and output.

14. An audio processing apparatus, characterized in that, The device includes: The audio restoration unit performs audio restoration based on the audio type of the target audio frame using an audio restoration strategy. The target audio frame is determined based on the frame type of the current audio frame and the previous audio frame; the audio type includes unvoiced human voice, voiced human voice, and musical voice; the audio repair strategy is used to generate repair frame data based on the audio type of the target audio frame; the repair frame data is used to replace audio frames or perform data fusion.

15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 13.

Images