Audio processing method, audio processing device, electronic device and storage medium

Abstract

The present disclosure provides an audio processing method, an audio processing device, an electronic device and a storage medium. The audio processing method may include: obtaining current network status information of a voice transmission network; determining the encoding method and number of redundant frames to be encoded when encoding a current audio frame based on the current network status information; encoding the current audio frame using a multi-level vector quantization encoding method, and encoding the number of redundant frames based on the determined encoding method. The present disclosure can use a coding method suitable for the current network status to reduce the code rate of coding and decoding redundant frames, thereby reducing the bandwidth burden of the transmission network.

Description

Technical Field

[0001] The present disclosure relates to the field of audio technology, and in particular to an audio processing method, an audio processing device, an electronic device and a storage medium for voice transmission in a weak network environment. Background Art

[0002] In the audio real-time communication scenario, the audio data transmission capacity is heavily dependent on the network bandwidth. A weak network environment will limit the transmission of audio data, directly leading to packet loss during the transmission process, which will lead to poor communication between the two parties in real-time communication or serious impact on the sound quality. Therefore, methods to combat weak networks in audio transmission have always been the focus of the industry. Summary of the invention

[0003] The present disclosure provides an audio processing method, an audio processing device, an electronic device and a storage medium to at least solve the problem of packet loss in voice transmission under a weak network environment. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, an audio processing method is provided. The method may include: obtaining current network status information of a voice transmission network; determining a coding method and a number of redundant frames to be encoded when encoding a current audio frame based on the current network status information; encoding the current audio frame using a multi-level vector quantization coding method, and encoding the number of redundant frames based on the determined coding method.

[0005] Optionally, the encoding method and number of redundant frames to be encoded when encoding the current audio frame are determined based on the current network status information, including: when the packet loss rate in the current network status information is less than a preset threshold, the number is determined to be zero; when the packet loss rate is greater than or equal to the preset threshold, the number is determined based on a preset range within which the packet loss rate is located, and the encoding method for each redundant frame is determined.

[0006] Optionally, the number is determined based on a preset range of the packet loss rate, including: when the packet loss rate is in a first preset range, the number is determined as a preset number; when the packet loss rate is in a second preset range, the number is determined as twice the preset number, wherein the minimum value of the second preset range is greater than the maximum value of the first preset range; when the packet loss rate is in a third preset range, the number is determined as N times the preset number, wherein the minimum value of the third preset range is greater than the maximum value of the second preset range, and N is an integer greater than 2.

[0007] Optionally, when the number of redundant frames to be encoded is not zero, encoding the number of redundant frames based on a determined encoding method includes: encoding the number of redundant frames using a multi-level vector quantization encoding method.

[0008] Optionally, when the number of redundant frames to be encoded is not zero, the number of redundant frames is encoded based on a determined encoding method, including: encoding at least part of the number of redundant frames using a similarity encoding method or a differential vector quantization encoding method, wherein the similarity encoding method is used to determine similar audio frames similar to the redundant frames and encode a flag indicating the similar audio frames, and the differential vector quantization encoding method is used to encode differential features between the redundant frames and the current audio frame.

[0009] Optionally, encoding the number of redundant frames based on a determined encoding method includes: grouping the number of redundant frames; for each group of redundant frames, performing the following operations: for a first redundant frame, determining a similar audio frame similar to the first redundant frame based on an audio frame adjacent to the first redundant frame, and encoding a flag indicating the similar audio frame; for a second redundant frame, encoding a differential feature between the second redundant frame and the current audio frame; for a third redundant frame, determining a similar audio frame similar to the third redundant frame based on an audio frame adjacent to the third redundant frame, and encoding a flag indicating the similar audio frame; for a fourth redundant frame, encoding the fourth redundant frame using a multi-level vector quantization encoding method.

[0010] Optionally, determining a similar audio frame that is similar to the first redundant frame based on an audio frame adjacent to the first redundant frame includes: constructing a reference audio frame based on the current audio frame and the second redundant frame; calculating feature similarities between the first redundant frame and the reference audio frame, the current audio frame and the second redundant frame respectively; and selecting the similar audio frame from the reference audio frame, the current audio frame and the second redundant frame based on the feature similarities.

[0011] According to a second aspect of an embodiment of the present disclosure, an audio processing method is provided. The method may include: obtaining current network status information of a voice transmission network; determining a decoding method and a number of redundant packets to be decoded when decoding a current audio packet based on the current network status information; decoding the current audio packet using a multi-level vector quantization decoding method, and decoding the number of redundant packets based on the determined decoding method.

[0012] Optionally, when the number of redundant packets to be decoded is not zero, the number of redundant packets is decoded based on a determined decoding method, including: using a multi-level vector quantization decoding method to decode the number of redundant packets to obtain characteristic information of redundant frames corresponding to the number of redundant packets.

[0013] Optionally, when the number of redundant packets to be decoded is not zero, the number of redundant packets is decoded based on a determined decoding method, including: using a similarity decoding method or a differential vector quantization decoding method to decode at least part of the number of redundant packets, wherein the similarity decoding method is used to decode a flag indicating a similar audio frame corresponding to the redundant packet and obtain feature information of the similar audio frame based on the flag, and the differential vector quantization decoding method is used to decode a differential feature and obtain feature information of the audio frame corresponding to the redundant packet based on the differential feature, wherein the differential feature is a differential feature between the audio frame corresponding to the redundant packet and the current audio frame.

[0014] Optionally, decoding the number of redundant packets based on a determined decoding method includes: grouping the number of redundant packets; for each group of redundant packets, performing the following operations: for a first redundant packet, obtaining a flag indicating a similar audio frame corresponding to the first redundant packet by decoding the first redundant packet, and obtaining characteristic information of the similar audio frame based on the flag as characteristic information of the audio frame corresponding to the first redundant packet; for a second redundant packet, obtaining a differential feature between the audio frame corresponding to the second redundant packet and the current audio frame by decoding the second redundant packet, and obtaining characteristic information of the audio frame corresponding to the second redundant packet based on the differential feature; for a third redundant packet, obtaining a flag indicating a similar audio frame corresponding to the third redundant packet by decoding the third redundant packet, and obtaining characteristic information of the similar audio frame based on the flag as characteristic information of the audio frame corresponding to the third redundant packet; for a fourth redundant packet, decoding the fourth redundant packet using a multi-level vector quantization decoding method to obtain characteristic information of the audio frame corresponding to the fourth redundant packet.

[0015] Optionally, the method may also include: decoding the current audio packet and the number of redundant packets to obtain feature information corresponding to the current audio packet and feature information corresponding to the redundant packet; restoring feature information of an audio frame lost before the current audio frame based on the feature information corresponding to the redundant packet; after restoring the feature information of the lost audio frame using the redundant packet, detecting whether there are still lost audio frames; if it is detected that there are still lost audio frames, using a neural network-based packet loss concealment model to obtain feature information of the lost audio frame.

[0016] Optionally, using a neural network-based packet loss concealment model to obtain feature information of the audio frame may include: using the packet loss concealment model to process feature information of at least one audio frame before the audio frame and features of at least one audio frame after the audio frame to obtain feature information of the audio frame.

[0017] The packet loss concealment model of the method can be trained based on the following manner: acquiring feature information of continuous audio frames and determining a target audio frame among the continuous audio frames; predicting feature information of the target audio frame using the packet loss concealment model based on feature information of a first number of audio frames before the target audio frame and feature information of a second number of audio frames after the target audio frame, wherein the first number and the second number are positive integers greater than or equal to 1; and updating parameters of the packet loss concealment model based on the predicted feature information of the target audio frame and the acquired feature information of the target audio frame.

[0018] According to a third aspect of an embodiment of the present disclosure, an audio processing device is provided. The device may include: a network status monitoring module configured to obtain current network status information of a voice transmission network; a determination module configured to determine, based on the current network status information, an encoding method and a number of redundant frames to be encoded when encoding a current audio frame; and an encoding module configured to encode the current audio frame using a multi-level vector quantization encoding method, and encode the number of redundant frames based on the determined encoding method.

[0019] Optionally, the determination module is configured to execute: when the packet loss rate in the current network status information is less than a preset threshold, the number is determined to be zero; when the packet loss rate is greater than or equal to the preset threshold, the number is determined based on a preset range of the packet loss rate, and the encoding method of each redundant frame is determined.

[0020] Optionally, the determination module is configured to execute: when the packet loss rate is within a first preset range, determining the number as a preset number; when the packet loss rate is within a second preset range, determining the number as twice the preset number, wherein the minimum value of the second preset range is greater than the maximum value of the first preset range; when the packet loss rate is within a third preset range, determining the number as N times the preset number, wherein the minimum value of the third preset range is greater than the maximum value of the second preset range, wherein N is an integer greater than 2.

[0021] Optionally, when the number of redundant frames to be encoded is not zero, the determination module is configured to execute: determining the encoding method of the number of redundant frames as a multi-level vector quantization encoding method.

[0022] Optionally, when the number of redundant frames to be encoded is not zero, the determination module is configured to perform: determining the encoding method of at least some of the redundant frames among the number of redundant frames as a similarity encoding method or a differential vector quantization encoding method, wherein the similarity encoding method is used to determine similar audio frames similar to the redundant frames and encode a flag indicating the similar audio frames, and the differential vector quantization encoding method is used to encode differential features between the redundant frames and the current audio frame.

[0023] Optionally, the determination module is configured to perform grouping processing on the number of redundant frames; for each group of redundant frames, perform the following determination operations: for the first redundant frame, determine a similar audio frame similar to the first redundant frame based on an audio frame adjacent to the first redundant frame, and determine to encode a flag indicating the similar audio frame; for the second redundant frame, determine to encode a differential feature between the second redundant frame and the current audio frame; for the third redundant frame, determine a similar audio frame similar to the third redundant frame based on an audio frame adjacent to the third redundant frame, and determine to encode a flag indicating the similar audio frame; for the fourth redundant frame, determine to encode the fourth redundant frame using a multi-level vector quantization encoding method.

[0024] Optionally, the determination module is configured to perform: constructing a reference audio frame based on the current audio frame and the second redundant frame; calculating feature similarities between the first redundant frame and the reference audio frame, the current audio frame and the second redundant frame respectively; and selecting the similar audio frame from the reference audio frame, the current audio frame and the second redundant frame based on the feature similarities.

[0025] According to a fourth aspect of an embodiment of the present disclosure, an audio processing device is provided. The device may include: a network status monitoring module configured to execute acquisition of current network status information of a voice transmission network; a determination module configured to execute, based on the current network status information, a decoding method and a number of redundant packets to be decoded when decoding a current audio packet; a decoding module configured to execute decoding of the current audio packet using a multi-level vector quantization decoding method, and to decode the number of redundant packets based on the determined decoding method.

[0026] Optionally, when the number of redundant packets to be decoded is not zero, the decoding module is configured to execute: decoding the number of redundant packets using a multi-level vector quantization decoding method to obtain characteristic information of redundant frames corresponding to the number of redundant packets.

[0027] Optionally, when the number of redundant packets to be decoded is not zero, the decoding module is configured to perform: decoding at least part of the redundant packets among the number of redundant packets using a similarity decoding method or a differential vector quantization decoding method, wherein the similarity decoding method is used to decode a flag indicating a similar audio frame corresponding to the redundant packet and obtain feature information of the similar audio frame based on the flag, and the differential vector quantization decoding method is used to decode a differential feature and obtain feature information of the audio frame corresponding to the redundant packet based on the differential feature, wherein the differential feature is a differential feature between the audio frame corresponding to the redundant packet and the current audio frame.

[0028] Optionally, the decoding module is configured to perform grouping processing on the number of redundant packets; for each group of redundant packets, the following operations are performed: for the first redundant packet, a flag indicating a similar audio frame corresponding to the first redundant packet is obtained by decoding the first redundant packet, and characteristic information of the similar audio frame is obtained based on the flag as characteristic information of the audio frame corresponding to the first redundant packet; for the second redundant packet, a differential feature between the audio frame corresponding to the second redundant packet and the current audio frame is obtained by decoding the second redundant packet, and characteristic information of the audio frame corresponding to the second redundant packet is obtained based on the differential feature; for the third redundant packet, a flag indicating a similar audio frame corresponding to the third redundant packet is obtained by decoding the third redundant packet, and characteristic information of the similar audio frame is obtained based on the flag as characteristic information of the audio frame corresponding to the third redundant packet; for the fourth redundant packet, the fourth redundant packet is decoded using a multi-level vector quantization decoding method to obtain characteristic information of the audio frame corresponding to the fourth redundant packet.

[0029] Optionally, the device may also include a packet loss concealment module configured to perform: restoring characteristic information of an audio frame lost before the current audio frame based on characteristic information corresponding to the number of redundant packets; after restoring the characteristic information of the lost audio frame using the redundant packets, detecting whether there are still lost audio frames; if it is detected that there are still lost audio frames, using a neural network-based packet loss concealment model to obtain the characteristic information of the lost audio frame.

[0030] Optionally, the packet loss concealment module may be configured to perform: using the packet loss concealment model, processing feature information of at least one audio frame before the audio frame and features of at least one audio frame after the audio frame to obtain feature information of the audio frame.

[0031] The packet loss concealment model is trained based on the following method: acquiring feature information of continuous audio frames and determining a target audio frame among the continuous audio frames; predicting feature information of the target audio frame using the packet loss concealment model based on feature information of a first number of audio frames before the target audio frame and feature information of a second number of audio frames after the target audio frame, wherein the first number and the second number are positive integers greater than or equal to 1; and updating parameters of the packet loss concealment model based on the predicted feature information of the target audio frame and the acquired feature information of the target audio frame.

[0032] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, which may include: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the audio processing method as described above.

[0033] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, which, when executed by at least one processor, prompts the at least one processor to execute the audio processing method as described above.

[0034] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, wherein instructions in the computer program product are executed by at least one processor in an electronic device to perform the audio processing method as described above.

[0035] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0036] The present disclosure considers the current network status to determine the number of redundant frames to be encoded and decoded and the corresponding method, uses a coding method suitable for the current network status to reduce the bit rate of encoding and decoding redundant frames, and combines the use of a multi-level vector quantization coding method to reduce the bit rate of encoding and decoding current frames, thereby reducing the bandwidth burden of the transmission network.

[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0039] Figure 1 is a diagram of an application environment for real-time communication according to an embodiment of the present disclosure;

[0040] Figure 2 is a flowchart of an audio processing method for speech coding according to an embodiment of the present disclosure;

[0041] Figure 3 is a schematic diagram of a method for encoding redundant frames according to an embodiment of the present disclosure;

[0042] Figure 4 is a flowchart of an audio processing method for speech decoding according to an embodiment of the present disclosure;

[0043] Figure 5 is a schematic diagram of a packet loss concealment model according to an embodiment of the present disclosure;

[0044] Figure 6 is a flowchart for audio processing according to an embodiment of the present disclosure;

[0045] Figure 7 is a structural schematic diagram of an audio processing device according to an embodiment of the present disclosure;

[0046] Figure 8 is a block diagram of an audio processing device for speech coding according to an embodiment of the present disclosure;

[0047] Fig. 9 is a block diagram of an audio processing device for speech decoding according to an embodiment of the present disclosure;

[0048] Fig.10 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0050] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the embodiments of the present disclosure as defined by the claims and their equivalents. Various specific details are included to assist in understanding, but these details are considered to be exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and structures are omitted for clarity and brevity.

[0051] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it should be clear to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and is not intended to limit the purpose of the present disclosure as defined by the claims and their equivalents.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0053] The forward error correction (FEC) method based on the traditional codec method can improve the audio transmission quality in a weak network environment by adding redundant packets and packet loss concealment technology. However, this method requires a higher bit rate. For example, to achieve high-quality audio transmission, it is generally necessary to rely on an audio bit rate of about 24kbps. The cost of adding redundant packets is relatively high, and adding too many redundant packets will also increase the network bandwidth requirements, forming a certain burden. On the other hand, this traditional method of packet loss concealment is generally difficult to ensure good sound quality when the packet loss rate is high, and it is difficult to ensure a smooth communication experience.

[0054] The present invention aims to propose a method for weak network resistance of voice transmission based on deep learning encoding and decoding, which improves the audio transmission quality in a weak network environment through a low bit rate redundant packet strategy and a deep learning packet loss concealment method, and ensures a good audio real-time communication experience.

[0055] Hereinafter, according to various embodiments of the present disclosure, the method and apparatus of the present disclosure will be described in detail with reference to the accompanying drawings.

[0056] Figure 1 is a diagram of an application environment for real-time communication according to an embodiment of the present disclosure.

[0057] Reference Figure 1 , the application environment 100 includes a terminal 110 and a terminal 120 .

[0058] Terminal 110 and terminal 120 may be terminals where users are located. For example, two users use terminals 110 and 120 for real-time communication, respectively. Terminal 110 and terminal 120 may be at least one of a smart phone, a tablet computer, a portable computer, and a desktop computer. Terminal 110 may be installed with a target application, such as an application for voice communication. Although only two terminals are shown in this embodiment for illustration, those skilled in the art will appreciate that the number of terminals may be two or more. The disclosed embodiments do not impose any restrictions on the number of terminals and device types.

[0059] Terminal 110 and terminal 120 can communicate via a wireless network, so that users of terminal 110 and terminal 120 can communicate in real time. For example, the network may include a local area network (LAN), a wide area network (WAN), a wireless link, an intranet, the Internet, or a combination thereof.

[0060] Terminal 110 may encode the voice input by the user of terminal 110, and then send the encoded voice data to terminal 120, and terminal 120 may decode the received encoded data, and then restore the decoded data to a voice signal. In addition, terminal 120 may encode the voice input by the user of terminal 120, and then send the encoded voice data to terminal 110, and terminal 110 may decode the received encoded data, and then restore the decoded data to a voice signal. Terminal 110 and / or terminal 120 may simultaneously implement encoding and decoding of voice.

[0061] According to the present disclosure, when the terminal 110 serves as an encoding end / transmitting end and the terminal 120 serves as a decoding end / receiving end, the terminal 110 may determine the number of redundant frames to be encoded together with the current frame and the corresponding encoding method according to the network state of the communication, and then send the encoded voice data to the terminal 120. The terminal 120 may determine the number of redundant packets to be decoded and the decoding method according to the network state, and then decode the current voice packet and the redundant packet using the corresponding decoding method, and further restore the lost frame features using a packet loss concealment method based on deep learning, thereby improving the audio transmission quality in a weak network environment and ensuring good audio real-time communication.

[0062] The present disclosure can select the appropriate number of redundant packets and encoding and decoding methods according to the network status, so that in the case of weak network, good voice intelligibility and good voice quality can be guaranteed, thereby improving user experience. Figures 2 to 5 Describes in more detail how to encode and decode speech in different network conditions.

[0063] Figure 2 is a flowchart of an audio processing method for speech coding according to an embodiment of the present disclosure. Figure 2 The audio processing method can be implemented in any electronic device with an audio encoding function. The electronic device can be at least one of the following, for example, a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader (e-bookreader), a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), a Moving Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) player, a camera, and a wearable device, etc.

[0064] Reference Figure 2 In step S101, the current network status information of the voice transmission network is obtained. For example, the bandwidth, packet loss rate and other information of the transmission network from the sending end to the receiving end can be monitored in real time. Taking the detection of the packet loss rate as an example, the detection is generally performed based on the packet loss sequence counted by the receiving end. Each sent audio packet has a continuously increasing sequence number. The receiving end can determine which audio packets are lost based on the sequence number of the received audio packet, thereby calculating the packet loss rate and feeding it back to the sending end. The above examples are only exemplary, and the present disclosure is not limited thereto.

[0065] In step S102, an encoding method and number of redundant frames to be encoded when encoding a current audio frame are determined based on current network status information.

[0066] As an example, when the packet loss rate in the current network status information is less than a preset threshold, it can be considered that there is no need to superimpose redundant packets at present, and at this time, the number of redundant frames to be encoded can be determined to be zero. When the packet loss rate is greater than or equal to the preset threshold, it can be considered that the current network is not good, and it is necessary to superimpose redundant packets to recover the packet loss caused by the weak network environment. At this time, the number of redundant frames to be encoded can be determined based on the preset range of the packet loss rate of the current network, and the encoding method of each redundant frame can be determined. Here, the preset threshold can be set differently according to the actual situation. For example, the preset threshold can be set to 5%.

[0067] When the packet loss rate of the current network is within a first preset range, the number of redundant frames may be determined as a preset number. When the packet loss rate is within a second preset range, the number of redundant frames may be determined as twice the preset number, wherein the minimum value of the second preset range is greater than the maximum value of the first preset range. When the packet loss rate is within a third preset range, the number of redundant frames may be determined as N times the preset number, wherein the minimum value of the third preset range is greater than the maximum value of the second preset range, wherein N is an integer greater than 2. Here, the preset number and the preset range may be set differently according to actual conditions, for example, the preset number may be set to 4 or 6. The endpoints of each preset range may be determined by a sound quality curve such as audio packet loss, and the specific values ​​may be adjusted according to the characteristics of the codec grid.

[0068] For example, when the network status is good, such as when the packet loss rate is less than 5%, there is no need to superimpose and send redundant frame information, and only the encoded current audio packet can be sent. When the packet loss rate is greater than or equal to 5% and less than 10%, a preset number of redundant frame information can be superimposed and sent, such as superimposing and sending 4 redundant frame information. When the packet loss rate is greater than or equal to 10% and less than 20%, redundant frame information twice the preset number can be superimposed and sent, such as superimposing 8 redundant frame information, so that when the continuous packet loss reaches 8, the receiving end can still recover all the voice feature data. If a higher packet loss rate occurs, such as greater than or equal to 20%, more redundant frame information can be added as needed, such as three times or more redundant packets of the preset number. The redundant frame information can be increased according to the corresponding multiples according to the size of the preset range. For example, when the preset number is 4, the number of redundant frames increased is a multiple of 4, that is, every 4 redundant frames are a group. The above examples are only exemplary, and the present disclosure is not limited thereto.

[0069] According to an embodiment of the present disclosure, several encoding methods may be pre-stored, and when it is determined that redundant packets need to be sent in an overlapping manner, encoding methods may be assigned to each redundant frame to obtain a redundant packet. A multi-level vector quantization encoding method may be used to encode the current audio frame and the determined redundant frame. Alternatively, a multi-level vector quantization encoding method may be used to encode the current audio frame, and a similarity encoding method or a differential vector quantization encoding method may be used to encode at least part of the redundant frames in the determined redundant frame. Here, the similarity encoding method may be used to determine similar audio frames that are similar to the redundant frame and to encode a flag indicating the similar audio frame. A differential vector quantization encoding method may be used to encode differential features between a redundant frame and the current audio frame.

[0070] When determining similar audio frames, a reference audio frame may be constructed based on a previous audio frame and a subsequent audio frame of the current redundant frame, and feature similarities between the current redundant frame and the reference audio frame, the previous audio frame, and the subsequent audio frame are calculated, and similar audio frames are selected from the reference audio frame, the previous audio frame, and the subsequent audio frame based on the feature similarities. The number of constructed reference frames may be one, two, or more.

[0071] In the case where it is determined that a redundant packet needs to be sent in superposition with the current audio frame, a coding method may be allocated to each redundant frame so that a coding operation is performed using the corresponding coding method when encoding the redundant frame.

[0072] The redundant frames to be encoded may be grouped and then the encoding operation may be performed on each group of redundant frames. For example, four redundant frames may be grouped and then the encoding operation may be performed on each group of four redundant frames.

[0073] For example, in the case where the number of redundant frames to be encoded is 4M, for every 4 redundant frames among the 4M redundant frames, a similarity encoding method may be assigned to the first redundant frame, that is, a similar audio frame similar to the first redundant frame is determined based on an audio frame adjacent to the first redundant frame, and a flag indicating the similar audio frame is encoded. A differential vector quantization encoding method may be assigned to the second redundant frame, that is, a differential feature between the second redundant frame and the current audio frame is encoded. A similarity encoding method may be assigned to the third redundant frame, that is, a similar audio frame similar to the third redundant frame is determined based on an audio frame adjacent to the third redundant frame, and a flag indicating the similar audio frame is encoded. A multi-level vector quantization encoding method may be assigned to the fourth redundant frame, that is, the fourth redundant frame is encoded using the multi-level vector quantization encoding method.

[0074] As another example, for each group of 5 redundant frames, the first redundant frame can be encoded using the similarity coding method, the second redundant frame can be encoded using the similarity coding method, the third redundant frame can be encoded using the differential vector quantization coding method, the fourth redundant frame can be encoded using the similarity coding method, and the fifth redundant frame can be encoded using the similarity coding method.

[0075] However, the encoding method of each redundant frame is merely exemplary, and the present disclosure is not limited thereto. Each redundant frame to be encoded may be assigned any one of a similarity encoding method, a differential vector quantization encoding method, and a multi-level vector quantization encoding method.

[0076] The low bit rate redundant packet strategy proposed in the present disclosure reduces the bit rate of redundant packets that need to be added by using a self-developed similarity coding method and a differential vector quantization coding method, thereby reducing the burden on network bandwidth caused by adding redundant packets.

[0077] In step S103, the current audio frame is encoded using a multi-level vector quantization encoding method, and the determined redundant frame is encoded based on the determined encoding method.

[0078] The input speech signal can be divided into frames, and for each audio frame, features of the audio frame can be extracted, and the extracted audio features can be encoded using a multi-level vector quantization encoding method. The multi-level vector quantization encoding method can refer to encoding a signal by cascading multiple vector quantization codes, and there is a residual between the signal after vector quantization encoding and the real signal. After the multiple vector quantization codes are cascaded, each level of vector quantization encoding can encode the residual between the signal after the previous several levels of vector quantization encoding and the real signal, thereby reducing the residual between the encoded signal and the real signal.

[0079] Taking the framing method of 10ms per frame as an example, each frame signal is encoded using 60 bits, and the audio bit rate can be compressed to 6kbps. This bit rate is very low compared to traditional methods, such as OPUS high-quality encoding requires 24kbps.

[0080] When it is determined that redundant packets need to be transmitted in a superimposed manner at the current moment, the redundant frames can be encoded using the encoding method assigned to each redundant frame in step S102. Taking the example of transmitting 8 redundant packets in a superimposed manner, the encoding method of each redundant frame is described in detail.

[0081] Figure 3 A schematic diagram of a redundant frame encoding method according to an embodiment of the present disclosure is shown. Assume that based on the current packet loss rate, it is determined that 8 redundant packets need to be sent in a superimposed manner when sending the current audio packet at the current moment. Assume that the current frame is the nth frame, the previous frame of the current frame is the n-1th frame (i.e., redundant frame No. 1), the previous frame of the n-1th frame is the n-2th frame (i.e., redundant frame No. 2), and so on, the previous frame of the n-7th frame is the n-8th frame (i.e., redundant frame No. 8).

[0082] Reference Figure 3 , for the audio feature feature(n) of the current nth frame, a multi-level vector quantization (VQ) encoding method can be used for compression, and 60 bits of data are required to encode the audio features of the current audio frame. If the current audio frame is the first frame of the input speech signal, the redundant frame can be set to an empty frame.

[0083] For the audio feature feature(n-1) of the n-1th frame adjacent to the current nth audio frame, that is, the previous frame of the nth audio frame, the similarity of the two adjacent frames can be used for encoding. For example, the similarity between the audio feature feature(n-1) of this frame and the four audio feature vectors of feature(n), feature(n-2), 0.25×feature(n)+0.75×feature(n-2) and 0.75×feature(n)+0.25×feature(n-2) can be referred to. The similarity can be measured by the Euclidean distance between the feature vectors, and the feature vector most similar to feature(n-1) is selected to represent the audio feature feature(n-1) of the n-1th frame. Since the feature vectors of the adjacent frames are known, only 2 bits of data are needed to mark which of the above four feature vectors feature(n-1) is most similar to. Here, feature(n-2) refers to the audio feature of the previous frame of the n-1th frame.

[0084] For the audio feature feature(n-2) of the n-2th frame, the feature vector of the difference between it and the feature(n) feature vector can be used for encoding, that is, the feature vector (feature(n)-feature(n-2)) can be encoded using the differential VQ encoding method. Since feature(n) is known, only 10 bits of data can be used to encode the feature(n-2) vector, which can achieve higher recovery accuracy.

[0085] For the redundant frames No. 3, No. 5, and No. 7, the same encoding method as that of the redundant frame No. 1 can be used, that is, the similarity encoding method of adjacent frames is used.

[0086] For redundant frames No. 4 and No. 8, the same encoding method as the current audio frame can be used, that is, a multi-level VQ encoding method is used.

[0087] For redundant frame No. 6, the same encoding method as redundant frame No. 2 may be used, that is, both use a differential vector quantization encoding method.

[0088] After using the above low-bitrate redundant packet strategy, even if 8 redundant packets are superimposed, only 148 bits of data will be added to the complete packet sent at the current moment, and the increased bit rate is only 14.8 kbps. If the 24 kbps bit rate of the traditional method is used, the bit rate will increase by 192 kbps after superimposing 8 redundant packets. Even if the low-quality 6 kbps bit rate is used, the redundant packets will increase by 48 kbps. It can be seen that the low-bitrate redundant packet solution disclosed in the present invention greatly alleviates the burden of superimposing redundant packets on network bandwidth.

[0089] According to another embodiment of the present disclosure, both the current audio frame and the redundant frame may be encoded using a multi-level VQ encoding method. Since the multi-level VQ encoding method is a deep learning-based encoding method, the method itself has a low bit rate, so even if the redundant frames above all use the same encoding method as the current frame (i.e., 60-bit multi-level VQ encoding), it will not significantly increase the bandwidth.

[0090] The present invention improves the audio transmission quality in a weak network environment and ensures a good audio real-time communication experience by using a low bit rate redundant packet strategy.

[0091] Figure 4 is a flowchart of an audio processing method for speech decoding according to an embodiment of the present disclosure. Figure 4 The audio processing method can be implemented in any electronic device with an audio decoding function. The electronic device can be at least one of the following, for example, a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), a Moving Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) player, a camera, and a wearable device, etc.

[0092] Reference Figure 4 In step S201, the current network status information of the voice transmission network is obtained. For example, the bandwidth, packet loss rate and other information of the transmission network from the sending end to the receiving end can be monitored in real time. Taking the detection of the packet loss rate as an example, the detection is generally performed based on the packet loss sequence counted by the receiving end. Each sent audio packet has a continuously increasing sequence number. The receiving end can determine which audio packets are lost based on the sequence number of the received audio packet, thereby counting the packet loss rate and then feeding it back to the sending end. The above examples are only exemplary, and the present disclosure is not limited thereto.

[0093] In step S202, a decoding method and a number of redundant packets to be decoded when decoding the current audio packet are determined based on the current network state information. Since the current audio frame and a determined number of redundant frames have been encoded based on the current network state at the encoding end, the current audio packet and the redundant packets are received at the decoding end, and a decoding operation can be performed using a decoding method corresponding to the encoding method.

[0094] As an example, when the number of redundant packets to be decoded is not zero and the current audio frame and the redundant frame are encoded using a multi-level vector quantization encoding method at the encoding end, the redundant packets can be decoded using the inverse process of the multi-level vector quantization encoding method (i.e., a multi-level vector quantization decoding method) to obtain feature information of the redundant frames corresponding to the redundant packets. The multi-level vector quantization decoding method may refer to cascading multiple vector quantization decodings to decode the signal, where there is a residual between the signal after vector quantization decoding and the real signal, and after cascading multiple vector quantization decodings, each level of vector quantization decoding can decode the residual between the signal after the previous several levels of vector quantization decoding and the real signal, thereby reducing the residual between the decoded signal and the real signal.

[0095] In the case where the number of redundant packets to be decoded is not zero and at least part of the redundant packets are encoded using a similarity encoding method or a differential vector quantization encoding method, at least part of the redundant packets may be decoded using a similarity decoding method or a differential vector quantization decoding method. For example, when a similarity encoding method is used for an n-1th redundant frame at the encoding side, a similarity decoding method may be used to decode an n-1th redundant packet at the decoding side.

[0096] The similarity decoding method may be used to decode a flag indicating a similar audio frame corresponding to a redundant packet and obtain feature information of the similar audio frame based on the flag.

[0097] The differential vector quantization decoding method can be used to decode the differential features between the audio frame corresponding to the redundant packet and the current audio frame and obtain the feature information of the audio frame corresponding to the redundant packet based on the differential features. Since the features of the current audio frame are known, the audio frame corresponding to the redundant packet can be obtained based on the differential features and the features of the current audio frame.

[0098] According to an embodiment of the present disclosure, redundant packets to be decoded may be grouped and then decoded for each group of redundant packets. For example, four redundant packets may be grouped and then decoded for each four redundant packets.

[0099] As an example, when the number of redundant packets is determined to be 4M, for every 4 redundant packets among the 4M redundant packets, the following operation for determining which redundant packet performs which decoding method may be performed. For the first redundant packet, a flag indicating a similar audio frame corresponding to the first redundant packet may be obtained by decoding the first redundant packet, and characteristic information of the similar audio frame may be obtained based on the flag as characteristic information of the audio frame corresponding to the first redundant packet; for the second redundant packet, a differential feature between an audio frame corresponding to the second redundant packet and a current audio frame may be obtained by decoding the second redundant packet, and characteristic information of the audio frame corresponding to the second redundant packet may be obtained based on the differential feature; for the third redundant packet, a flag indicating a similar audio frame corresponding to the third redundant packet may be obtained by decoding the third redundant packet, and characteristic information of the similar audio frame may be obtained based on the flag as characteristic information of the audio frame corresponding to the third redundant packet; for the fourth redundant packet, a multi-level vector quantization decoding method may be used to decode the fourth redundant packet to obtain characteristic information of the audio frame corresponding to the fourth redundant packet.

[0100] In step S203, the current audio packet is decoded using a multi-stage vector quantization decoding method, and the determined redundant packet is decoded based on the determined decoding method. The decoding operation process is the reverse process of the encoding operation described above.

[0101] In addition, after decoding the current audio packet and the determined redundant packet and obtaining feature information corresponding to the current audio packet and feature information corresponding to the redundant packet, feature information of the audio frame lost before the current audio frame can be restored based on the feature information corresponding to the redundant packet.

[0102] After the speech features are decoded and the packet loss is recovered, there may still be lost voice packets, so it is necessary to further recover them through the packet loss concealment function. At this time, the loss feature prediction method based on the neural network can be used for processing. For example, after the feature information of the lost audio frame is restored by using the redundant packet, it can be further detected whether there are still lost audio frames. If it is detected that there are still lost audio frames, the packet loss concealment model based on the neural network can be used to obtain the feature information of the audio frame. The packet loss concealment model can be any deep neural network structure, and can be trained based on the following method: obtain the feature information of continuous audio frames and determine the target audio frame in the continuous audio frames, based on the feature information of the first number (such as X) of audio frames before the target audio frame and the feature information of the second number (such as Y) of audio frames after the target audio frame, use the packet loss concealment model to predict the feature information of the audio frame, where X and Y are positive integers greater than or equal to 1. Update the parameters of the packet loss concealment model based on the predicted feature information of the audio frame and the acquired real feature information.

[0103] For example, in the training stage of the model, the lost data is simulated loss. After the predicted output is obtained, it is compared with the real features (i.e. the simulated lost frame features) to evaluate the similarity and then iterate the network through back propagation. The loss function is an indicator for evaluating the predicted features and the real features. The mean square error MSE can be selected as the loss function, or other reasonable loss functions can be selected. Figure 5 Describe the packet loss concealment model in detail.

[0104] Figure 5 is a schematic diagram of a packet loss concealment model according to an embodiment of the present disclosure.

[0105] Reference Figure 5 , the packet loss hiding model can be composed of two convolutional neural networks (CNN) and two gated recurrent units (GRU). However, Figure 5 The model shown is only exemplary, and the packet loss concealment model may also include various structures such as a fully connected network and a long short-term memory network.

[0106] Assuming that after using redundant packets to restore the frame loss features, it is determined that the features of frame n still do not exist, the features of the m frames before frame n and the l features after frame n can be input into the packet loss concealment model. Here, the features of the m frames before frame n are represented as HistoryFeat(n)={feature(n-1), feature(n-2, ..., featuren-m, and the l feature after frame n is represented as FutureFeat(n)={feature(n+1), ..., feature(n+l)}. Among them, the number of previous frames m is generally set to 10-20, and in order to ensure the high real-time performance of the processing process, the number of subsequent frames l is generally set to 1 or 2, and the output of the network is the predicted features of a frame of data that has actually been lost. After the previous frame and the subsequent frame pass through the packet loss concealment model, the predicted feature f'(n) of frame n can be output.

[0107] This paper uses a deep learning packet loss concealment method to improve the audio transmission quality in a weak network environment and ensure a good audio real-time communication experience.

[0108] After packet loss concealment processing, continuous feature information can be obtained, and then a high-quality speech signal can be reconstructed using the continuous feature information.

[0109] The deep learning-based packet loss concealment method proposed in the present invention is combined with deep learning encoding and decoding, and a method of predicting packet loss speech features through a neural network is used to predict the features required to reconstruct the speech signal, thereby achieving high-quality packet loss concealment, thereby improving the user's audio real-time communication experience under weak network conditions.

[0110] Figure 6 It is a flowchart for audio processing according to an embodiment of the present disclosure.

[0111] Reference Figure 6 , the feature extraction module can extract the feature information of the voice input from the sending end. The feature encoding module can encode the feature information to reduce its bit rate. The network status detection module can detect the current network status information of the transmission network. The low bit rate redundant packet module can add redundant packets under weak network conditions, while ensuring that the increase in redundant packets will not significantly increase the network burden. The network transmission module can transmit the encoded voice data packets from the sending end to the receiving end. The feature decoding and recovery module can decode the encoded feature information, and at the same time, in a weak network state, restore the voice feature information that cannot be transmitted to the receiving end through the network through redundant packets. The packet loss concealment module can further restore the lost voice feature information in the case of severe packet loss, that is, hide the empty voice frames caused by packet loss. The voice decoding module can decode the voice feature information, reconstruct the voice signal, and obtain the voice output by the final receiving end. Refer to the following Figure 6 The audio processing process of the present disclosure is described in detail.

[0112] The input speech signal may be framed. For example, the length of each frame of the speech signal may be set to 10 ms to 40 ms. The nth frame of the speech signal after framing may be represented as x(n).

[0113] By extracting features from x(n) through the feature extraction module, an abstract speech feature can be obtained. Various feature extraction methods can be used to extract features from the speech signal at time n. Here, the speech feature can be in various forms, for example, the speech feature can be a Mel frequency cepstral coefficient. The speech feature extraction process can be expressed as:

[0114] feature(n)=FEAT[x(n)]

[0115] Among them, feature(n) represents the speech feature of the speech signal at time n, and FEAT[*] represents the feature extraction processing operation.

[0116] The feature(n) is input into the feature coding module, and the multi-level vector quantization (VQ) coding method can be used to perform bit rate compression on the feature(n). For each speech frame at a moment, the multi-level VQ coding method can be used to perform bit rate compression on the speech features of the speech frame.

[0117] Taking the framing method of 10ms per frame as an example, each frame of speech signal is encoded with 60 bits, and the audio bit rate can be compressed to 6kbps, which is very low compared to traditional compression methods. For example, OPUS high-quality speech coding requires 24kbps. The multi-level vector quantization encoding process can be expressed as:

[0118] multiVQfeature(n)=multiVQ[feature(n)]

[0119] Wherein, multiVQfeature(n) represents the Mel-frequency cepstral coefficient feature after VQ encoding, and multiVQ[*] represents a multi-level VQ encoding operation.

[0120] The network status information of the voice transmission network from the current sending end to the receiving end can be obtained from the network status detection module. For example, the current bandwidth, packet loss rate and other information of the voice transmission network can be determined by real-time monitoring of the voice transmission network. Taking the detection of packet loss rate as an example, the detection is generally performed based on the packet loss sequence counted by the receiving end. Each sent audio packet has a continuously increasing sequence number. The receiving end can determine which audio packets are lost based on the sequence number of the received audio packet, thereby calculating the packet loss rate and feeding it back to the sending end.

[0121] The low bit rate redundant packet module can determine whether redundant packets need to be superimposed and the number of redundant packets to be superimposed when sending the current audio packet based on the current network status information. Whether redundant packets need to be superimposed and the number of redundant packets can be determined by comparing the detected current packet loss rate with a preset threshold. The preset threshold for the packet loss rate can be determined by sound quality curves such as audio packet loss, and can be adjusted according to the characteristics of the codec network.

[0122] As an example, when the network status is good, such as when the packet loss rate is less than 5%, there is no need to superimpose redundant packets, and the encoded current audio packet is sent directly. When the packet loss rate is greater than or equal to 5% and less than 10%, a preset number of redundant packets can be superimposed, such as 4 redundant packets. When the packet loss rate is greater than or equal to 10% and less than 20%, 2 times the preset number of redundant packets can be superimposed, such as 8 redundant packets. This ensures that when the continuous packet loss reaches 8, the receiving end can still recover all the voice feature data. If a higher packet loss rate occurs, such as greater than or equal to 20%, more redundant packets can be added as needed, such as three times or more of the preset number of redundant packets, such as 12 redundant packets.

[0123] The redundant packets may be increased according to the corresponding multiples according to the size of the preset range. For example, when the preset number is 6, the number of redundant packets increased is a multiple of 6, that is, every 6 redundant packets form a group, and the audio packet of the current frame and the corresponding number of redundant packets are sent at the current moment. The above examples are only exemplary, and the present disclosure is not limited thereto.

[0124] In order to implement forward error correction in a weak network environment without significantly increasing the network burden, a low-bitrate audio compression method can be used to encode the superimposed redundant packets. Taking the superposition of 8 redundant packets as an example, it means that at the current moment, in addition to sending the current audio packet, it is also necessary to send redundant packets of the 8 audio frames before the current audio frame. Assume that the current frame is the nth frame, the previous frame of the current frame is the n-1th frame (i.e., redundant frame No. 1), the previous frame of the n-1th frame is the n-2th frame (i.e., redundant frame No. 2), and so on. The previous frame of the n-7th frame is the n-8th frame (i.e., redundant frame No. 8).

[0125] For the audio feature feature(n) of the nth frame at the current moment, a multi-level VQ encoding method can be used for compression, and 60 bits of data are required to encode the audio feature of the current audio frame.

[0126] For the audio feature feature(n-1) of the n-1th frame (i.e., redundant frame No. 1) adjacent to the current nth frame, the similarity between the two adjacent frames can be used for encoding. For example, the similarity between the reference frame audio feature feature(n-1) and the four audio feature vectors feature(n), feature(n-2), 0.25×feature(n)+0.75×feature(n-2), and 0.75×feature(n)+0.25×feature(n-2) can be measured by the Euclidean distance between the feature vectors, and the feature vector most similar to feature(n-1) is selected to represent the audio feature feature(n-1) of the n-1th frame. Since the feature vectors of the adjacent frames are known, only 2 bits of data are needed to mark which of the above four feature vectors feature(n-1) is most similar to. Here, feature(n-2) refers to the feature vector of the previous frame of the n-1th audio frame.

[0127] For the audio feature feature(n-2) of the n-2th frame (i.e., redundant frame No. 2), the feature vector of the difference between it and the feature vector feature(n) can be used for encoding, that is, the feature vector (feature(n)-feature(n-2)) is encoded using the differential VQ encoding method. Since feature(n) is known, only 10-bit data can be used to encode the feature vector of feature(n-2), which can achieve higher recovery accuracy.

[0128] For the redundant frames No. 3, No. 5, and No. 7, the same encoding method as that of the redundant frame No. 1 can be used, that is, the similarity encoding method of adjacent frames is used.

[0129] For redundant frames No. 4 and No. 8, the same encoding method as the current audio frame can be used, that is, a multi-level VQ encoding method is used.

[0130] For redundant frame No. 6, the same encoding method as redundant frame No. 2 may be used, that is, both use a differential vector quantization encoding method.

[0131] After using the above low-bitrate redundant packet strategy, even if 8 redundant packets are superimposed, only 148 bits of data will be added to the complete packet sent at the current moment, and the increased bit rate is only 14.8 kbps. If the 24 kbps bit rate of the traditional method is used, the bit rate will increase by 192 kbps after superimposing 8 redundant packets. Even if the low-quality 6 kbps bit rate is used, the redundant packets will increase by 48 kbps. It can be seen that the low-bitrate redundant packet solution disclosed in the present invention greatly alleviates the burden of superimposing redundant packets on network bandwidth.

[0132] For another example, since the codec based on deep learning itself has a low bit rate, if the above redundant frames all use the same encoding method as the current frame (i.e., a 60-bit multi-level VQ encoding method), it will not cause a significant increase in bandwidth.

[0133] The above encoded data can be transmitted from the sending end to the receiving end through the network transmission module.

[0134] The feature decoding and recovery module can decode the received data packets and recover the lost data according to the network status information (such as packet loss rate) and redundant packets. The process of decoding redundant packets and recovering lost packets is the reverse process of encoding redundant frames and adding redundant packets.

[0135] After the above decoding and packet loss recovery processing, the speech features may still have lost speech packets, so they need to be further recovered through the packet loss concealment module. The packet loss concealment module can use a packet loss concealment model based on a neural network for processing. For example, the network structure of the packet loss concealment model can be a combination of CNN and GRU networks, or it can involve multiple structures such as fully connected networks and long short-term memory networks. Assuming that it is determined that the audio features of the nth frame are still missing after the packet loss is recovered through redundant packets, the features of the m frames before the nth frame HistoryFeat(n)={feature(n-1), feature(n-2,..., featuren-m and the features of the l frames after the nth frame FutureFeat(n)={feature(n+1),..., feature(n+l)} can be input into the packet loss concealment model, and the audio features of the nth frame can be obtained through model prediction.

[0136] Here, the number of previous frames m can be selected as 10-20, and in order to ensure the high real-time performance of the processing process, the number of subsequent frames l is generally selected as 1 or 2. The data lost in the training phase is simulated loss. After obtaining the predicted output, it is compared with the real features to evaluate the similarity and then iterate the network through back propagation. The loss function is an indicator for evaluating the predicted features and the real features. MSE can be selected as the loss function, or other reasonable loss functions can be selected.

[0137] The continuous feature information output by the packet loss concealment module can be sequentially fed into the speech decoding module to reconstruct a high-quality speech signal. The speech decoding module can use a vocoder scheme, such as LPCNet, WaveNet, MelGAN, WaveRNN, or related improved structures.

[0138] According to the embodiments of the present disclosure, in the case of a weak network, out-of-band forward error correction FEC can be performed without significantly increasing the network burden through the low-bitrate redundant packet method, ensuring good recovery capabilities in the case of continuous packet loss. In addition, after the packet loss characteristics are restored by adding redundant packets, if there are still lost packets, the loss characteristics can be further restored through a packet loss concealment module based on deep learning, and finally a high-quality voice signal in a weak network state is obtained through deep learning decoding, thereby improving the user's audio real-time communication experience in a weak network state.

[0139] Figure 7 5 is a schematic diagram of the structure of an audio processing device in a hardware operating environment of an embodiment of the present disclosure. Here, the audio processing device 500 can implement the above encoding function and / or decoding function.

[0140] like Figure 7 As shown, the audio processing device 500 may include: a processing component 501, a communication bus 502, a network interface 503, an input / output interface 504, a memory 505, and a power supply component 506. Among them, the communication bus 502 is used to realize the connection and communication signals between these components. The input / output interface 504 may include a video display (such as a liquid crystal display), a microphone and a speaker, and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.), and optionally, the input / output interface 504 may also include a standard wired interface and a wireless interface. The network interface 503 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity interface). The memory 505 may be a high-speed random access memory or a stable non-volatile memory. The memory 505 may also be a storage device independent of the aforementioned processing component 501.

[0141] Those skilled in the art will understand that Figure 7The structure shown in the figure does not constitute a limitation on the audio processing device 500, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0142] like Figure 7 As shown, the memory 505 as a storage medium may include an operating system (such as a MAC operating system), a data storage module, a network communication module, a user interface module, an audio processing program, and a database.

[0143] exist Figure 7 In the audio processing device 500 shown, the network interface 503 is mainly used for data communication with external devices / terminals; the input and output interface 504 is mainly used for data interaction with users; the processing component 501 and the memory 505 in the audio processing device 500 can be set in the audio processing device 500, and the audio processing device 500 calls the audio processing program stored in the memory 505 and various APIs provided by the operating system through the processing component 501 to execute the audio processing methods provided in the embodiments of the present disclosure, such as speech encoding methods and speech decoding methods.

[0144] The processing component 501 may include at least one processor, and a computer executable instruction set is stored in the memory 505. When the computer executable instruction set is executed by the at least one processor, the audio processing method according to the embodiment of the present disclosure is executed. In addition, the processing component 501 may perform the encoding operation and decoding operation as described above. However, the above examples are only exemplary, and the present disclosure is not limited thereto.

[0145] As an example, the audio processing device 500 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above-mentioned instruction set. Here, the audio processing device 500 is not necessarily a single electronic device, but can also be any device or circuit that can execute the above-mentioned instructions (or instruction sets) individually or in combination. The audio processing device 500 can also be part of an integrated control system or system manager, or can be configured to execute a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0146] In the audio processing device 500, the processing component 501 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processing component 501 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0147] The processing component 501 can run instructions or codes stored in the memory, wherein the memory 505 can also store data. Instructions and data can also be sent and received over the network via the network interface 503, wherein the network interface 503 can use any known transmission protocol.

[0148] The memory 505 may be integrated with the processor, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 505 may include a separate device, such as an external disk drive, a storage array, or any other storage device that may be used by a database system. The memory and processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.

[0149] Figure 8 is a block diagram of an audio processing apparatus for speech coding according to an embodiment of the present disclosure.

[0150] Reference Figure 8 , the audio processing device 600 may include a network status monitoring module 601, a determination module 602, and an encoding module 603. Each module in the audio processing device 600 may be implemented by one or more modules, and the name of the corresponding module may vary according to the type of the module. In various embodiments, some modules in the audio processing device 600 may be omitted, or additional modules may be included. In addition, the modules / elements according to various embodiments of the present disclosure may be combined to form a single entity, and therefore the functions of the corresponding modules / elements before the combination may be performed equivalently.

[0151] The network status monitoring module 601 can obtain the current network status information of the voice transmission network.

[0152] The determination module 602 may determine the encoding method and number of redundant frames to be encoded when encoding the current audio frame based on the current network status information.

[0153] The encoding module 603 may encode the current audio frame using a multi-level vector quantization encoding method, and encode a determined number of redundant frames based on a determined encoding method.

[0154] As an example, when the packet loss rate in the current network status information is less than a preset threshold, the determination module 602 may determine the number of redundant frames to be zero. When the current packet loss rate is greater than or equal to the preset threshold, the determination module 602 may determine the number of redundant frames based on a preset range of the packet loss rate, and determine the encoding method for each redundant frame. Here, several encoding methods for redundant frames may be pre-designed, and the determination module 602 may assign one encoding method to each redundant frame.

[0155] As an example, when the packet loss rate is within a first preset range, the determination module 602 may determine the number of redundant frames (redundant frames to be superimposed and sent) as a preset number. When the packet loss rate is within a second preset range, the determination module 602 may determine the number of redundant frames as twice the preset number, wherein the minimum value of the second preset range is greater than the maximum value of the first preset range. When the packet loss rate is within a third preset range, the determination module 602 may determine the number of redundant frames as N times the preset number, wherein the minimum value of the third preset range is greater than the maximum value of the second preset range, wherein N is an integer greater than 2.

[0156] As an example, when the number of redundant frames to be encoded is not zero, the determination module 602 may determine the encoding method of all redundant frames as a multi-level vector quantization encoding method, that is, determine to use the multi-level VQ encoding method to perform encoding operations on each redundant frame.

[0157] As an example, when the number of redundant frames to be encoded is not zero, the determination module 602 may determine the encoding method of at least a portion of the redundant frames as a similarity encoding method or a differential vector quantization encoding method, wherein the similarity encoding method may be used to determine similar audio frames similar to the redundant frames and encode flags indicating the similar audio frames. The differential vector quantization encoding method may be used to encode differential features between the redundant frames and the current audio frame.

[0158] The determination module 602 may group the redundant frames to be encoded, and then perform encoding operations on each group of redundant frames. For example, four redundant frames may be grouped together, and then encoding operations may be performed on each group of four redundant frames.

[0159] As an example, when the number of redundant frames is 4M, M is a positive integer greater than or equal to 1, and for every 4 redundant frames in the 4M redundant frames, the determination module 602 may perform the following determination operations: for the first redundant frame, determine a similar audio frame similar to the first redundant frame based on an audio frame adjacent to the first redundant frame, and determine to encode a flag indicating the similar audio frame; for the second redundant frame, determine to encode a differential feature between the second redundant frame and the current audio frame; for the third redundant frame, determine a similar audio frame similar to the third redundant frame based on an audio frame adjacent to the third redundant frame, and determine to encode a flag indicating the similar audio frame; for the fourth redundant frame, determine to encode the fourth redundant frame using a multi-level vector quantization encoding method.

[0160] As an example, when determining a similar audio frame for the first redundant frame, the determination module 602 may construct a reference audio frame based on the current audio frame and the second redundant frame, calculate feature similarities between the first redundant frame and the reference audio frame, the current audio frame, and the second redundant frame, and select the similar audio frame from the reference audio frame, the current audio frame, and the second redundant frame based on the feature similarities. The number of constructed reference frames may be set differently according to actual conditions.

[0161] Referenced above Figure 2 and Figure 3 The encoding method for redundant frames has been described in detail and will not be repeated here.

[0162] Fig. 9 It is a block diagram of an audio processing device at a receiving end according to an embodiment of the present disclosure.

[0163] Reference Fig. 9 , the audio processing device 700 may include a network status monitoring module 701, a determination module 702, a decoding module 703, and a packet loss concealment module 704. Each module in the audio processing device 700 may be implemented by one or more modules, and the name of the corresponding module may vary according to the type of the module. In various embodiments, some modules in the audio processing device 700 may be omitted, or additional modules may be further included. In addition, the modules / elements according to various embodiments of the present disclosure may be combined to form a single entity, and thus the functions of the corresponding modules / elements before the combination may be equivalently performed.

[0164] The network status monitoring module 701 can obtain the current network status information of the voice transmission network.

[0165] The determination module 702 may determine a decoding method and a number of redundant packets to be decoded when decoding a current audio packet based on current network status information.

[0166] The decoding module 703 may decode the current audio packet using a multi-stage vector quantization decoding method, and decode a determined number of redundant packets based on a determined decoding method.

[0167] As an example, when the number of redundant packets to be decoded is not zero, the decoding module 703 may use a multi-stage vector quantization decoding method to decode all redundant packets to obtain feature information of redundant frames corresponding to all redundant packets.

[0168] As an example, when the number of redundant packets to be decoded is not zero, the decoding module 703 may use a similarity decoding method or a differential vector quantization decoding method to decode at least part of the redundant packets of the determined number. The similarity decoding method may be used to decode a flag indicating a similar audio frame corresponding to the redundant packet and obtain feature information of the similar audio frame based on the flag. The differential vector quantization decoding method may be used to decode a differential feature between an audio frame corresponding to the redundant packet and a current audio frame and obtain feature information of the audio frame corresponding to the redundant packet based on the differential feature.

[0169] The decoding module 703 may group the redundant packets to be decoded, and then perform a decoding operation on each group of redundant packets. For example, four redundant packets may be grouped as one group, and then a decoding operation may be performed on each four redundant packets.

[0170] As an example, when the number of redundant packets is 4M, M is a positive integer greater than or equal to 1, and for every 4 redundant packets among the 4M redundant packets, the decoding module 703 may perform the following operations: for the first redundant packet, a flag indicating a similar audio frame corresponding to the first redundant packet is obtained by decoding the first redundant packet, and characteristic information of the similar audio frame is obtained based on the flag as characteristic information of the audio frame corresponding to the first redundant packet; for the second redundant packet, a differential feature between the audio frame corresponding to the second redundant packet and the current audio frame is obtained by decoding the second redundant packet, and characteristic information of the audio frame corresponding to the second redundant packet is obtained based on the differential feature; for the third redundant packet, a flag indicating a similar audio frame corresponding to the third redundant packet is obtained by decoding the third redundant packet, and characteristic information of the similar audio frame is obtained based on the flag as characteristic information of the audio frame corresponding to the third redundant packet; for the fourth redundant packet, the fourth redundant packet is decoded using a multi-level vector quantization decoding method to obtain characteristic information of the audio frame corresponding to the fourth redundant packet.

[0171] Here, whichever encoding method is used to encode the redundant frame at the encoding side, the corresponding decoding method will be used to decode the redundant frame at the decoding side. That is, the process of decoding the audio packet at the decoding side is the reverse process of encoding the audio frame at the encoding side.

[0172] The packet loss concealment module 704 may restore the characteristic information of the audio frame lost before the current audio frame based on the characteristic information corresponding to the determined number of redundant packets, and after restoring the characteristic information of the lost audio frame using the redundant packets, detect whether there are still lost audio frames. If it is detected that there are still lost audio frames, the packet loss concealment module 704 may use a packet loss concealment model based on a neural network to obtain the characteristic information of the audio frame.

[0173] For example, if it is detected that there are still lost audio frames, the packet loss concealment module 704 can use a packet loss concealment model to process the feature information of at least one audio frame before the audio frame and the features of at least one audio frame after the audio frame to obtain the feature information of the audio frame.

[0174] As an example, the packet loss concealment model can be trained based on the following method: obtaining feature information of continuous audio frames; predicting feature information of an audio frame using the packet loss concealment model based on feature information of X audio frames before an audio frame in the continuous audio frames and feature information of Y audio frames after the audio frame, where X and Y are positive integers greater than or equal to 1; updating parameters of the packet loss concealment model based on the predicted feature information of the audio frame and the obtained feature information. Here, the packet loss concealment model can be trained using any neural network.

[0175] Referenced above Figure 4 and Figure 5 The decoding method for redundant frames has been described in detail and will not be repeated here.

[0176] According to another embodiment of the present disclosure, an audio processing device is provided, which may include the network status monitoring module 701, the determination module 602, the determination module 702, the encoding module 603, the decoding module 703 and the packet loss concealment module 704 described above. In addition, it may also include a receiving module (not shown) and a sending module (not shown). The sending module can send the encoded voice packet to other devices, and the receiving module can receive the voice packet from other devices to implement the encoding operation of the input voice and the decoding operation of the received voice, thereby realizing voice communication.

[0177] According to an embodiment of the present disclosure, an electronic device may be provided. Fig.10 It is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 1000 may include at least one memory 1002 and at least one processor 1001. The at least one memory 1002 stores a set of computer-executable instructions. When the computer-executable instruction set is executed by the at least one processor 1001, the audio processing method (i.e., speech encoding method and / or speech decoding method) according to the embodiment of the present disclosure is executed.

[0178] Processor 1001 may include a central processing unit (CPU), an audio processor, a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, processor 1001 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0179] The memory 1002 as a storage medium may include an operating system (eg, a MAC operating system), a data storage module, a network communication module, a user interface module, an audio processing program, an audio codec program, and a database.

[0180] The memory 1002 may be integrated with the processor 1001, for example, a RAM or flash memory may be arranged within an integrated circuit microprocessor or the like. In addition, the memory 1002 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1002 and the processor 1001 may be operationally coupled, or may communicate with each other, for example, via an I / O port, a network connection, etc., so that the processor 1001 can read files stored in the memory 1002.

[0181] In addition, the electronic device 1000 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1000 may be connected to each other via a bus and / or a network.

[0182] As an example, the electronic device 1000 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1000 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 1000 may also be part of an integrated control system or a system manager, or may be configured to execute a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0183] Those skilled in the art will understand that Fig.10 The structures shown in the drawings do not constitute limitations of the drawings and may include more or less components than shown in the drawings, or combine certain components, or arrange the components differently.

[0184] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the audio processing method (speech encoding method and / or speech decoding method) according to the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to perform non-transitory storage of computer programs and any associated data, data files and data structures and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0185] According to an embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by a processor of a computer device to complete the above-mentioned audio processing method (speech encoding method and / or speech decoding method).

[0186] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0187] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that: The method comprises: Obtain current network status information of the voice transmission network; Determining, based on the current network status information, an encoding method and a number of redundant frames to be encoded when encoding the current audio frame; The current audio frame is encoded using a multi-level vector quantization encoding method, and the number of redundant frames is encoded based on a determined encoding method, Wherein, when the number of redundant frames is 4M, and M is a positive integer greater than or equal to 1, encoding the number of redundant frames based on the determined encoding method includes: performing grouping processing on the number of redundant frames; For each set of redundant frames, perform the following operations: For a first redundant frame, determining a similar audio frame similar to the first redundant frame based on audio frames adjacent to the first redundant frame, and encoding a flag for indicating the similar audio frame; For a second redundant frame, encoding a differential feature between the second redundant frame and the current audio frame; For a third redundant frame, determining a similar audio frame similar to the third redundant frame based on audio frames adjacent to the third redundant frame, and encoding a flag for indicating the similar audio frame; For the fourth redundant frame, a multi-level vector quantization encoding method is used to encode the fourth redundant frame.

2. The method according to claim 1, characterized in that Determining, based on the current network status information, an encoding method and a number of redundant frames to be encoded when encoding a current audio frame, comprising: When the packet loss rate in the current network status information is less than a preset threshold, determining the number to be zero; When the packet loss rate is greater than or equal to the preset threshold, the number is determined based on a preset range of the packet loss rate, and an encoding method for each redundant frame is determined.

3. The method according to claim 2, characterized in that Determining the number based on a preset range of the packet loss rate includes: When the packet loss rate is within a first preset range, determining the number as a preset number; When the packet loss rate is within a second preset range, determining the number to be twice the preset number, wherein the minimum value of the second preset range is greater than the maximum value of the first preset range; When the packet loss rate is within a third preset range, the number is determined to be N times the preset number, wherein a minimum value of the third preset range is greater than a maximum value of the second preset range, and wherein N is an integer greater than 2.

4. The method according to claim 1, characterized in that: Determining a similar audio frame similar to the first redundant frame based on an audio frame adjacent to the first redundant frame includes: constructing a reference feature based on the current audio frame and the second redundant frame; Calculating feature similarities between the feature of the first redundant frame and the reference feature, the feature of the current audio frame, and the feature of the second redundant frame; A feature with the highest similarity is selected from the reference feature, the feature of the current audio frame, and the feature of the second redundant frame based on the feature similarity, and a frame corresponding to the feature with the highest similarity is used as the similar audio frame.

5. An audio processing method, characterized in that: The method comprises: Obtain current network status information of the voice transmission network; determining, based on the current network status information, a decoding method and a number of redundant packets to be decoded when decoding a current audio packet; Decoding the current audio packet using a multi-stage vector quantization decoding method, and decoding the number of redundant packets based on a determined decoding method, Wherein, when the number of redundant packets is 4M, and M is a positive integer greater than or equal to 1, decoding the number of redundant packets based on the determined decoding method includes: performing grouping processing on the number of redundant packets; For each set of redundant packets, do the following: For a first redundant packet, obtain a flag indicating a similar audio frame corresponding to the first redundant packet by decoding the first redundant packet, and obtain characteristic information of the similar audio frame based on the flag as characteristic information of the audio frame corresponding to the first redundant packet; For the second redundant packet, obtain a differential feature between an audio frame corresponding to the second redundant packet and a current audio frame by decoding the second redundant packet, and obtain feature information of the audio frame corresponding to the second redundant packet based on the differential feature; For a third redundant packet, a flag indicating a similar audio frame corresponding to the third redundant packet is obtained by decoding the third redundant packet, and characteristic information of the similar audio frame is obtained based on the flag as characteristic information of the audio frame corresponding to the third redundant packet; For the fourth redundant packet, a multi-level vector quantization decoding method is used to decode the fourth redundant packet to obtain feature information of an audio frame corresponding to the fourth redundant packet.

6. The method according to claim 5, characterized in that Also includes: By decoding the current audio packet and the number of redundant packets, characteristic information corresponding to the current audio packet and characteristic information corresponding to the redundant packets are obtained; Restoring characteristic information of an audio frame lost before a current audio frame based on characteristic information corresponding to the redundant packet; After restoring the characteristic information of the lost audio frame by using the redundant packet, detecting whether there is still a lost audio frame; If it is detected that there are still lost audio frames, a packet loss concealment model based on a neural network is used to obtain feature information of the lost audio frames.

7. The method according to claim 6, characterized in that Using a packet loss concealment model based on a neural network to obtain feature information of the audio frame includes: The packet loss concealment model is adopted to process feature information of at least one audio frame before the audio frame and features of at least one audio frame after the audio frame to obtain feature information of the audio frame.

8. The method according to claim 6, characterized in that The packet loss concealment model is trained based on the following method: Acquire feature information of continuous audio frames and determine a target audio frame in the continuous audio frames; Predicting the feature information of the target audio frame using the packet loss concealment model based on feature information of a first number of audio frames before the target audio frame and feature information of a second number of audio frames after the target audio frame, wherein the first number and the second number are positive integers greater than or equal to 1; Parameters of the packet loss concealment model are updated based on the predicted feature information of the target audio frame and the acquired feature information of the target audio frame.

9. An audio processing device, characterized in that: The device comprises: A network status monitoring module is configured to obtain current network status information of a voice transmission network; a determination module configured to determine, based on the current network status information, an encoding method and a number of redundant frames to be encoded when encoding a current audio frame; an encoding module configured to perform encoding of the current audio frame using a multi-level vector quantization encoding method, and to encode the number of redundant frames based on a determined encoding method, Wherein, when the number of redundant frames satisfies 4M, and M is a positive integer greater than or equal to 1, the determination module is configured to execute: performing grouping processing on the number of redundant frames; For each set of redundant frames, the following determination operations are performed: For a first redundant frame, determine a similar audio frame similar to the first redundant frame based on an audio frame adjacent to the first redundant frame, and determine to encode a flag for indicating the similar audio frame; For the second redundant frame, determine to encode the differential features between the second redundant frame and the current audio frame; For a third redundant frame, determining a similar audio frame similar to the third redundant frame based on an audio frame adjacent to the third redundant frame, and determining to encode a flag for indicating the similar audio frame; For the fourth redundant frame, it is determined to use a multi-level vector quantization encoding method to encode the fourth redundant frame.

10. The device according to claim 9, characterized in that Make sure the module is configured to execute: When the packet loss rate in the current network status information is less than a preset threshold, determining the number to be zero; When the packet loss rate is greater than or equal to the preset threshold, the number is determined based on a preset range of the packet loss rate, and an encoding method for each redundant frame is determined.

11. The device according to claim 10, characterized in that Make sure the module is configured to execute: When the packet loss rate is within a first preset range, determining the number as a preset number; When the packet loss rate is within a second preset range, determining the number to be twice the preset number, wherein the minimum value of the second preset range is greater than the maximum value of the first preset range; When the packet loss rate is within a third preset range, the number is determined to be N times the preset number, wherein a minimum value of the third preset range is greater than a maximum value of the second preset range, and wherein N is an integer greater than 2.

12. The device according to claim 9, characterized in that Make sure the module is configured to execute: constructing a reference feature based on the current audio frame and the second redundant frame; Calculating feature similarities between the feature of the first redundant frame and the reference feature, the feature of the current audio frame, and the feature of the second redundant frame; A feature with the highest similarity is selected from the reference feature, the feature of the current audio frame, and the feature of the second redundant frame based on the feature similarity, and a frame corresponding to the feature with the highest similarity is used as the similar audio frame.

13. An audio processing device, characterized in that: The device comprises: A network status monitoring module is configured to obtain current network status information of a voice transmission network; a determination module configured to perform a decoding method and a number of redundant packets to be decoded when decoding a current audio packet based on the current network status information; a decoding module configured to perform decoding of the current audio packet using a multi-level vector quantization decoding method, and to decode the number of redundant packets based on a determined decoding method, Wherein, when the number of redundant packets is 4M, and M is a positive integer greater than or equal to 1, the decoding module is configured to execute: performing grouping processing on the number of redundant packets; For each set of redundant packets, do the following: For a first redundant packet, obtain a flag indicating a similar audio frame corresponding to the first redundant packet by decoding the first redundant packet, and obtain characteristic information of the similar audio frame based on the flag as characteristic information of the audio frame corresponding to the first redundant packet; For the second redundant packet, obtain a differential feature between an audio frame corresponding to the second redundant packet and a current audio frame by decoding the second redundant packet, and obtain feature information of the audio frame corresponding to the second redundant packet based on the differential feature; For a third redundant packet, a flag indicating a similar audio frame corresponding to the third redundant packet is obtained by decoding the third redundant packet, and characteristic information of the similar audio frame is obtained based on the flag as characteristic information of the audio frame corresponding to the third redundant packet; For the fourth redundant packet, a multi-level vector quantization decoding method is used to decode the fourth redundant packet to obtain feature information of an audio frame corresponding to the fourth redundant packet.

14. The device according to claim 13, characterized in that Also included is a packet loss concealment module, configured to perform: restoring feature information of an audio frame lost before a current audio frame based on feature information corresponding to the number of redundant packets; After restoring the characteristic information of the lost audio frame by using the redundant packet, detecting whether there is still a lost audio frame; If it is detected that there are still lost audio frames, a packet loss concealment model based on a neural network is used to obtain feature information of the lost audio frames.

15. The device according to claim 14, characterized in that The packet loss concealment module is configured to perform: The packet loss concealment model is adopted to process feature information of at least one audio frame before the audio frame and features of at least one audio frame after the audio frame to obtain feature information of the audio frame.

16. The device according to claim 14, characterized in that The packet loss concealment model is trained based on the following method: Acquire feature information of continuous audio frames and determine a target audio frame in the continuous audio frames; Predicting the feature information of the one audio frame using the packet loss concealment model based on feature information of a first number of audio frames before the target audio frame and feature information of a second number of audio frames after the target audio frame, wherein the first number and the second number are positive integers greater than or equal to 1; Parameters of the packet loss concealment model are updated based on the predicted feature information of the target audio frame and the acquired feature information of the target audio frame.

17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor, The processor is configured to execute the instructions to implement the audio processing method as described in any one of claims 1-8.

18. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the audio processing method according to any one of claims 1 to 8.

19. A computer program product, wherein instructions in the computer program product are executed by at least one processor in an electronic device to perform the audio processing method according to any one of claims 1 to 8.

Images