Method, device and related equipment for compensating audio data packet loss

Abstract

The present disclosure provides a method, apparatus, and related equipment for compensating for packet loss of audio data, relating to the technical field of audio encoding and decoding, wherein the method comprises: upon detecting that a target audio data packet is missing from a cache, obtaining N reference audio data packets from the cache, wherein the audio playback time of any of the reference audio data packets is earlier than the audio playback time of the target audio data packet, and the N reference audio data packets are continuous and adjacent to the target audio data packet; performing sub-band decomposition on the reference audio data packets to obtain M reference sub-band data sets, where M is an integer greater than 1; predicting the target audio data packet based on the M reference sub-band data sets to obtain compensated audio data packets on the M sub-bands, synthesizing the M sub-bands to obtain compensated audio data packets, and storing the compensated audio data packets in the cache. The present disclosure can make the predicted compensated audio data packets more accurate.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio coding and decoding, and in particular to a method and apparatus for compensating for packet loss of audio data, and related equipment. Background Art

[0002] During audio data transmission, audio data packets may be lost due to signal attenuation, interference, network congestion or other factors. This loss of audio data packets may cause noticeable interruption of the audio signal or loss of sound quality, seriously affecting the user's listening experience.

[0003] At present, relevant technologies use linear prediction algorithms to compensate for lost audio data packets, that is, by analyzing the received audio data packets and extracting characteristic values ​​such as the resonance peaks of the audio signals therein to estimate the lost audio data packets. In applications, it is found that when faced with diverse audio signals (such as multimedia resource transmission of audio and video) and different communication environments (such as situations where the network environment fluctuates greatly), the accuracy of the audio data packets estimated based on this method is low. Summary of the Invention

[0004] The present disclosure aims to provide a method, apparatus and related devices for compensating for packet loss of audio data, so as to solve the technical problem of low accuracy of audio data packets estimated in complex application environments in related technologies.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for compensating for packet loss of audio data, comprising:

[0006] When a target audio data packet is detected to be missing from the cache, obtaining N reference audio data packets from the cache, wherein the audio playback time of any of the reference audio data packets is earlier than the audio playback time of the target audio data packet, and the N reference audio data packets are continuous and adjacent to the target audio data packet, where N is an integer greater than 1;

[0007] Performing sub-band decomposition on the N reference audio data packets to obtain M reference sub-band data sets, where M is an integer greater than 1;

[0008] The target audio data packet is predicted according to the M reference sub-band data sets to obtain a compensated audio data packet, and the compensated audio data packet is stored in the buffer.

[0009] In one embodiment, predicting the target audio data packet based on the M reference sub-band data sets to obtain the compensated audio data packet includes:

[0010] Downsampling the M reference subband data sets to obtain M downsampled data sets, where a downsampling factor is greater than 1 and less than or equal to M;

[0011] The target audio data packet is predicted according to the M down-sampled data sets to obtain a compensated audio data packet.

[0012] In one embodiment, predicting the target audio data packet based on the M downsampled data sets to obtain the compensated audio data packet includes:

[0013] Predicting the target audio data packet according to the M down-sampled data sets to obtain M target sub-bands, wherein the M target sub-bands correspond one-to-one to the M down-sampled data sets;

[0014] Data synthesis is performed according to the M target sub-bands to obtain the compensated audio data packet.

[0015] In one embodiment, synthesizing data according to the M target sub-bands to obtain the compensated audio data packet includes:

[0016] performing upsampling processing on the M target subbands respectively to obtain M upsampled subbands, wherein the upsampling factor of each target subband is the same as the downsampling factor of the corresponding reference subband dataset;

[0017] Data synthesis is performed on the M up-sampled sub-bands to obtain the compensated audio data packet.

[0018] In one embodiment, performing sub-band decomposition on the N reference audio data packets to obtain M reference sub-band data sets includes:

[0019] Performing sub-band decomposition on each reference audio data packet to obtain M reference sub-bands corresponding to the M sub-band positions. The N reference sub-band data at the same sub-band position of the N reference audio data packets constitute a reference sub-band set. The reference sub-band data of the N reference audio data packets form M reference sub-band data sets.

[0020] The predicting the target audio data packet according to the M reference sub-band data sets to obtain the compensated audio data packet includes:

[0021] Downsampling the M reference subband data sets respectively to obtain M downsampled data sets corresponding to the M subband positions;

[0022] Predicting the audio subbands at the same subband position of the target audio data packet according to each downsampled data set to obtain corresponding target subbands, wherein the M subband positions correspond one-to-one to the M audio subbands of the target audio data packet;

[0023] The compensated audio data packet is obtained according to the M target sub-bands.

[0024] In one embodiment, the buffer stores a plurality of audio data packets, each of the audio data packets corresponds to a sequence number, and the sequence numbers corresponding to different audio data packets in the plurality of audio data packets are different;

[0025] The method further comprises:

[0026] When it is detected that multiple serial numbers corresponding to multiple audio data packets stored in the cache are discontinuous, it is determined that a target audio data packet is missing from the cache, wherein the target audio data packet is the audio data packet corresponding to the missing serial number among the multiple serial numbers.

[0027] In one embodiment, after storing the compensated audio data packet in the buffer, the method further includes:

[0028] Acquire output configuration information, where the output configuration information is used to indicate a time delay and / or audio duration of target audio data;

[0029] extracting the target audio data from the buffer according to the output configuration information;

[0030] The target audio data is output.

[0031] In a second aspect, an embodiment of the present disclosure provides a device for compensating for packet loss of audio data, the device comprising:

[0032] an acquisition module, configured to acquire N reference audio data packets from the cache when detecting that a target audio data packet is missing from the cache, wherein the audio playback time of any of the reference audio data packets is earlier than the audio playback time of the target audio data packet, and the N reference audio data packets are continuous and adjacent to the target audio data packet, where N is an integer greater than 1;

[0033] a decomposition module, configured to perform sub-band decomposition on the N reference audio data packets to obtain M reference sub-band data sets, where M is an integer greater than 1;

[0034] The compensation module is configured to predict the target audio data packet according to the M reference sub-band data sets to obtain a compensated audio data packet, and store the compensated audio data packet in the buffer.

[0035] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned audio data packet loss compensation method when executed by the processor.

[0036] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned audio data packet loss compensation method are implemented.

[0037] In an embodiment of the present disclosure, when audio data packet loss is detected, multiple reference audio data packets adjacent to the lost audio data packet are obtained from a cache, and the multiple reference audio data packets are converted into multiple reference sub-band data sets through sub-band decomposition. The lost audio data packet is then predicted based on the multiple reference sub-band data sets. The sub-band decomposition measure can simplify the spectral representation of the audio signal corresponding to the audio data packet, thereby reducing the data error of subsequent predictions and making the predicted audio data packet more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of a method for compensating for packet loss of audio data provided by an embodiment of the present disclosure;

[0039] Figure 2 This is a flow chart of a method for compensating for audio signal packet loss provided by an embodiment of the present disclosure;

[0040] Figure 3 1 is a schematic structural diagram of a packet loss compensation device for audio data provided by an embodiment of the present disclosure;

[0041] Figure 4 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0043] The present disclosure provides a method for compensating for packet loss of audio data. Figure 1 As shown, the method for compensating for packet loss of audio data includes:

[0044] Step 101: When it is detected that a target audio data packet is missing from a cache, N reference audio data packets are obtained from the cache.

[0045] The audio playing time of any of the reference audio data packets is earlier than the audio playing time of the target audio data packet, and the N reference audio data packets are continuous and adjacent to the target audio data packet.

[0046] The method disclosed herein is applied to audio transmission scenarios, specifically to the transmission of audio streams, such as a first audio stream for music playback, a second audio stream for forming a video soundtrack, and a third audio stream for audio and video calls.

[0047] In the present disclosure, an audio stream is transmitted in the form of a plurality of audio data packets, and the plurality of audio data packets are transmitted in an orderly manner based on the plurality of audio playback moments corresponding thereto in the audio stream. For example, when transmitting audio data packet A1 corresponding to the first second of the audio playback moment, audio data packet A2 corresponding to the second second of the audio playback moment, and audio data packet A3 corresponding to the third second of the audio playback moment, audio data packet A1 is transmitted first, followed by audio data packet A2, and finally audio data packet A3.

[0048] The above-mentioned cache is set up to facilitate the management of several audio data packets that form the corresponding audio stream, that is, to timely monitor whether several audio data packets of the corresponding audio stream are missing. If missing, the missing audio data packets are compensated by executing subsequent steps, thereby ensuring the integrity of the multiple audio data packets stored in the cache, and avoiding playback interruptions or sound quality loss in the audio signals extracted and output from the cache.

[0049] If the audio data packets corresponding to the audio stream are not missing, the subsequent steps are not performed.

[0050] For example, if the audio data packets expected to be transmitted to the cache are: audio data packet A1, audio data packet A2, audio data packet A3 and audio data packet A4 whose corresponding audio playback time is the fourth second, if the target audio data packet missing from the cache is audio data packet A4, then audio data packet A1, audio data packet A2 and audio data packet A3 can all be understood as the aforementioned reference audio data packets.

[0051] Step 102: Perform sub-band decomposition on the N reference audio data packets to obtain M reference sub-band data sets.

[0052] It should be noted that the number of sub-bands obtained by sub-band decomposition of N reference audio data packets is the same. For example, each of the N reference audio data packets is decomposed into 4 or 8 sub-bands (also called frequency bands). The N data corresponding to the same sub-band obtained by sub-band decomposition of the N reference audio data packets form the aforementioned reference sub-band data set. In this case, M is 4 or 8.

[0053] In one example, a sub-band decomposition process may be performed using a cosine modulation scheme of sub-band coding (SBC).

[0054] Exemplarily, the number of reference subbands included in each reference subband data set is greater than 2, such as 4, 5, 8, etc. The present disclosure does not limit the specific number of reference subbands included in each reference subband data set. Users can make adaptive selections based on the actual spectral characteristics and transmission requirements of the audio stream.

[0055] Step 103: Predict the target audio data packet according to the M reference sub-band data sets to obtain a compensated audio data packet, and store the compensated audio data packet in the cache.

[0056] In an embodiment of the present disclosure, when audio data packet loss is detected, multiple reference audio data packets adjacent to the lost audio data packet are obtained from a cache, and the multiple reference audio data packets are converted into multiple reference sub-band data sets through sub-band decomposition. The lost audio data packet is then predicted based on the multiple reference sub-band data sets. The sub-band decomposition measure can simplify the spectral representation of the audio signal corresponding to the audio data packet, thereby reducing the data error of subsequent predictions and making the predicted audio data packet more accurate.

[0057] In one example, the M reference subband data sets can be processed based on a Burg algorithm to obtain the compensated audio data packet. It should be noted that in this example, although the execution of the subband decomposition action will result in a certain loss in processing efficiency (compared to the solution of directly inputting the M reference audio data packets into the Burg algorithm), since the input to the Burg algorithm is the M reference subband data sets that have undergone subband decomposition, and the multiple subbands after decomposition correspond to different frequency band components, the above arrangement can simplify the spectral representation of the audio signal corresponding to the reference audio data packet, thereby reducing the interference encountered by the Burg algorithm when predicting the target audio data packet based on the audio signal corresponding to the reference audio data packet. This allows the Burg algorithm to more finely analyze and process audio signal portions within different frequency ranges, thereby outputting a more accurate compensated audio data packet.

[0058] In one embodiment, performing sub-band decomposition on the N reference audio data packets to obtain M reference sub-band data sets includes:

[0059] Performing sub-band decomposition on each reference audio data packet to obtain M reference sub-bands corresponding to the M sub-band positions. The N reference sub-band data at the same sub-band position of the N reference audio data packets constitute a reference sub-band set. The reference sub-band data of the N reference audio data packets form M reference sub-band data sets.

[0060] The predicting the target audio data packet according to the M reference sub-band data sets to obtain the compensated audio data packet includes:

[0061] Downsampling the M reference subband data sets respectively to obtain M downsampled data sets corresponding to the M subband positions;

[0062] Predicting the audio subbands at the same subband position of the target audio data packet according to each downsampled data set to obtain corresponding target subbands, wherein the M subband positions correspond one-to-one to the M audio subbands of the target audio data packet;

[0063] The compensated audio data packet is obtained according to the M target sub-bands.

[0064] For example, if the subband decomposition is set to correspond to 3 subband positions (i.e., M=3), and there are 4 reference audio data packets (i.e., N=4), subband decomposition is performed on the 4 reference audio data packets respectively to obtain 4 reference subband sets A1[a11, a12, a13], A2[a21, a22, a23], A3[a31, a32, a33], and A4[a41, a42, a43], where a11, a21, a31, and a41 all correspond to the first subband position, a12, a22, a32, and a42 all correspond to the second subband position, and a13, a23, a33, and a43 all correspond to the third subband position.

[0065] In this example, three reference subband data sets can be obtained based on the four reference subband sets, namely, the reference subband data set corresponding to the first subband position [a11, a21, a31, a41], the reference subband data set corresponding to the second subband position [a12, a22, a32, a42], and the reference subband data set corresponding to the third subband position [a13, a23, a33, a43].

[0066] Based on the reference subband data set [a11, a21, a31, a41], the target subband b1 corresponding to the first subband position can be obtained;

[0067] Based on the reference subband data set [a12, a22, a32, a42], the target subband b2 corresponding to the second subband position can be obtained;

[0068] Based on the reference subband data set [13, a23, a33, a43], the target subband b3 corresponding to the third subband position can be obtained;

[0069] According to the target sub-band b1, the target sub-band b2 and the target sub-band b3, the aforementioned compensated audio data packet can be obtained.

[0070] In one embodiment, predicting the target audio data packet based on the M reference sub-band data sets to obtain the compensated audio data packet includes:

[0071] Downsampling the M reference subband data sets to obtain M downsampled data sets, where a downsampling factor is greater than 1 and less than or equal to M;

[0072] The target audio data packet is predicted according to the M down-sampled data sets to obtain a compensated audio data packet.

[0073] In this embodiment, a downsampling processing step is set to reduce the amount of computation in the prediction process (the effective information contained in the downsampled data is also reduced accordingly). This not only improves the prediction efficiency, but also reduces the error accumulation caused by excessive computation, making the output compensated audio data packet more accurate.

[0074] Among them, by limiting the downsampling multiple to no more than the number of subbands corresponding to the subband decomposition step, the problem of information loss caused by excessive downsampling can be avoided, which can ensure the accuracy of the obtained prediction results.

[0075] In one example, if the Berg algorithm is applied to perform the aforementioned prediction processing, although the Berg algorithm has a very good prediction effect for short-time stationary signals, the more prediction points are input into the Berg algorithm, the more errors are accumulated, which will cause the accuracy of the prediction results finally output by the algorithm to decrease. After adopting the above-mentioned downsampling processing, by reducing the number of prediction points input into the Berg algorithm, the accumulation of errors can be suppressed, thereby improving the accuracy of the prediction results finally output by the algorithm.

[0076] It should be noted that, in some embodiments, if the aforementioned prediction processing is performed based on the Berg algorithm, the sub-band decomposition step can be skipped, and the M reference audio data packets can be directly downsampled separately, and then prediction can be performed based on the M reference audio data packets after downsampling. This can also improve the accuracy of the prediction result finally output by the algorithm.

[0077] In the application, for any two different reference subband data sets among the M reference subband data sets, the downsampling multiples of the two can be the same or different, which makes the processing of the method described in the present disclosure more flexible in the downsampling stage.

[0078] In one embodiment, predicting the target audio data packet based on the M downsampled data sets to obtain the compensated audio data packet includes:

[0079] Predicting the target audio data packet according to the M down-sampled data sets to obtain M target sub-bands, wherein the M target sub-bands correspond one-to-one to the M down-sampled data sets;

[0080] Data synthesis is performed according to the M target sub-bands to obtain the compensated audio data packet.

[0081] For example, if the number of subbands M is set to 4 and the downsampling factor L is 4, the N reference audio data packets are decomposed into reference subband data sets C1, C2, C3, and C4. After L-fold downsampling, the downsampled data sets c1, c2, c3, and c4 are obtained. The downsampled data sets c1, c2, c3, and c4 are predicted respectively to obtain the corresponding target subbands t1, t2, t3, and t4.

[0082] Exemplarily, data synthesis may be performed on the M target sub-bands through a synthesis filter to obtain the aforementioned compensated audio data packet.

[0083] In one embodiment, synthesizing data according to the M target sub-bands to obtain the compensated audio data packet includes:

[0084] performing upsampling processing on the M target subbands respectively to obtain M upsampled subbands, wherein the upsampling factor of each target subband is the same as the downsampling factor of the corresponding reference subband dataset;

[0085] Data synthesis is performed on the M up-sampled sub-bands to obtain the compensated audio data packet.

[0086] In this embodiment, after downsampling the subbands in the reference subband data set, the predicted target subband is adaptively upsampled to restore the data volume corresponding to the target subband, thereby ensuring data consistency between the subsequently synthesized compensated audio data packet and the reference audio data packet.

[0087] In one example, data synthesis may be performed on the M target sub-bands to obtain an initial audio data packet, and then the initial audio data packet may be upsampled to obtain the compensated audio data packet. This can also ensure data consistency between the compensated audio data packet and the reference audio data packet.

[0088] In one embodiment, the buffer stores a plurality of audio data packets, each of the audio data packets corresponds to a sequence number, and the sequence numbers corresponding to different audio data packets in the plurality of audio data packets are different;

[0089] The method further comprises:

[0090] When it is detected that multiple serial numbers corresponding to multiple audio data packets stored in the cache are discontinuous, it is determined that a target audio data packet is missing from the cache, wherein the target audio data packet is the audio data packet corresponding to the missing serial number among the multiple serial numbers.

[0091] In this embodiment, different serial numbers are set for multiple audio data packets corresponding to the same audio stream, so that by monitoring the serial numbers corresponding to the audio data packets stored in the cache, it is possible to quickly identify whether there are any missing audio data packets in the cache, and then promptly complete the data compensation of the missing audio data packets, thereby ensuring the continuity of the audio data packets stored in the cache, so that the audio data extracted and output from the cache has better quality and more reliable data continuity, so that users can obtain a better listening experience.

[0092] For example, the serial numbers of several audio data packets stored in the cache can be set to show a continuous increasing or decreasing trend. In this case, if several serial numbers corresponding to several audio data packets stored in the cache are discontinuous or missing, it can be determined that a packet loss problem occurs during the audio stream transmission, that is, it is determined that the target audio data packet is missing in the cache.

[0093] In one embodiment, after storing the compensated audio data packet in the buffer, the method further includes:

[0094] Acquire output configuration information, where the output configuration information is used to indicate a time delay and / or audio duration of target audio data;

[0095] extracting the target audio data from the buffer according to the output configuration information;

[0096] The target audio data is output.

[0097] As mentioned above, based on the aforementioned packet loss compensation measures, the continuity of the audio data packets stored in the cache is guaranteed. In this case, the user can specify the time delay and / or playback duration (i.e., audio duration) of the required target audio data based on actual needs, and extract the corresponding target audio data from the cache accordingly to meet the audio playback requirements or audio processing requirements in different application scenarios.

[0098] For example, the formula of the prediction process corresponding to the Berg algorithm is as follows:

[0099]

[0100] Where, X t is the prediction signal (i.e., a target subband forming the compensated audio data packet), X t-p is a reference subband of the historical signal (i.e., a reference audio data packet, X t-1 …X t-p corresponding to the aforementioned M reference audio data packets), p is the order of the Berg algorithm, t indicates the sequence number of the prediction signal in the signal stream (that is, the sequence number corresponding to the target audio data packet in the audio stream), a pis the recursive prediction coefficient of the corresponding historical signal, ε t is the random disturbance term (i.e. noise) corresponding to the predicted signal.

[0101] By applying the method described in the present disclosure, it is possible to effectively compensate for packet loss while preserving audio quality, thereby providing users with a more stable and coherent audio transmission experience, and having the advantages of applicability, adaptability, and efficiency in multimedia data transmission scenarios.

[0102] For easier understanding, the following examples are provided:

[0103] See also Figure 2 , Figure 2 A method for compensating for audio signal packet loss is shown, specifically:

[0104] Step 1: The audio data packet corresponding to the input audio is input into the cache; the cache is used to store historical data for packet loss compensation. If there is no packet loss, the audio data is directly output from the cache; if there is packet loss, the predicted audio data packet for packet loss compensation is first stored in the cache, and then the corresponding audio data is taken out from the cache for output.

[0105] Step 2: During audio streaming, each audio data packet is assigned a unique sequence number for reassembly and recovery at the audio receiver. The receiver detects packet loss by monitoring the sequence numbers of received audio data packets.

[0106] In the absence of packet loss, the sequence number of the audio data should be continuously increasing; if the receiving end detects that the sequence number is discontinuous or missing, it is determined that packet loss has occurred.

[0107] When packet loss is detected, the receiving end will mark the audio data packet with the corresponding sequence number as lost, and obtain the corresponding predicted audio data packet based on the packet loss compensation step.

[0108] The steps for packet loss compensation are as follows:

[0109] First, multiple audio data packets in the buffer before the loss state are sequentially decomposed into different sub-bands or frequency bands, each sub-band representing a portion of spectral information in the audio data packet;

[0110] Secondly, the multiple sub-bands obtained by decomposition are sequentially downsampled to reduce the amount of calculation and improve the prediction accuracy;

[0111] Then, the burg algorithm is used to perform adaptive prediction compensation on the downsampled output, combined with the filtering process to generate subband estimates of the lost audio data packets, and then the subband estimates are upsampled;

[0112] The above packet loss compensation step is applied to each sub-band to obtain a sub-band estimation value of each sub-band. Finally, a synthesis filter is used to synthesize multiple sub-band estimation values ​​to obtain the final predicted audio data packet.

[0113] Finally, the delay and length of extracting audio data from the cache are determined based on user needs, and the extracted audio data is sent to other modules for further processing.

[0114] See also Figure 3 , Figure 3 The present disclosure provides an audio data packet loss compensation device 300, such as Figure 3 As shown, the audio data packet loss compensation device 300 includes:

[0115] an acquisition module 301 for acquiring N reference audio data packets from the cache when a target audio data packet is detected to be missing from the cache, wherein the audio playback time of any of the reference audio data packets is earlier than the audio playback time of the target audio data packet, and the N reference audio data packets are continuous and adjacent to the target audio data packet, where N is an integer greater than 1;

[0116] a decomposition module 302 configured to perform sub-band decomposition on the N reference audio data packets to obtain M reference sub-band data sets, where M is an integer greater than 1;

[0117] The compensation module 303 is configured to predict the target audio data packet according to the M reference sub-band data sets to obtain a compensated audio data packet, and store the compensated audio data packet in the buffer.

[0118] In one embodiment, the decomposition module 302 is specifically configured to:

[0119] Performing sub-band decomposition on each reference audio data packet to obtain M reference sub-bands corresponding to the M sub-band positions. The N reference sub-band data at the same sub-band position of the N reference audio data packets constitute a reference sub-band set. The reference sub-band data of the N reference audio data packets form M reference sub-band data sets.

[0120] The compensation module 303 is specifically used for:

[0121] Downsampling the M reference subband data sets respectively to obtain M downsampled data sets corresponding to the M subband positions;

[0122] Predicting the audio subbands at the same subband position of the target audio data packet according to each downsampled data set to obtain corresponding target subbands, wherein the M subband positions correspond one-to-one to the M audio subbands of the target audio data packet;

[0123] The compensated audio data packet is obtained according to the M target sub-bands.

[0124] In one embodiment, the compensation module 303 includes:

[0125] a downsampling unit, configured to perform downsampling processing on the M reference subband data sets respectively to obtain M downsampled data sets, where a multiple of the downsampling processing is greater than 1 and less than or equal to M;

[0126] The prediction unit is configured to predict the target audio data packet according to the M down-sampled data sets to obtain a compensated audio data packet.

[0127] In one embodiment, the prediction unit is specifically configured to:

[0128] Predicting the target audio data packet according to the M down-sampled data sets to obtain M target sub-bands, wherein the M target sub-bands correspond one-to-one to the M down-sampled data sets;

[0129] Data synthesis is performed according to the M target sub-bands to obtain the compensated audio data packet.

[0130] In one embodiment, the prediction unit is further configured to:

[0131] performing upsampling processing on the M target subbands respectively to obtain M upsampled subbands, wherein the upsampling factor of each target subband is the same as the downsampling factor of the corresponding reference subband dataset;

[0132] Data synthesis is performed on the M up-sampled sub-bands to obtain the compensated audio data packet.

[0133] In one embodiment, the downsampling factor is greater than 1 and less than or equal to M.

[0134] In one embodiment, the buffer stores a plurality of audio data packets, each of the audio data packets corresponds to a sequence number, and the sequence numbers corresponding to different audio data packets in the plurality of audio data packets are different;

[0135] The apparatus 300 further includes:

[0136] The monitoring module is used to determine that a target audio data packet is missing from the cache when it is detected that multiple serial numbers corresponding to multiple audio data packets stored in the cache are discontinuous, wherein the target audio data packet is the audio data packet corresponding to the missing serial number among the multiple serial numbers.

[0137] In one embodiment, the apparatus 300 further includes an output module, wherein the output module is configured to:

[0138] Acquire output configuration information, where the output configuration information is used to indicate a time delay and / or audio duration of target audio data;

[0139] extracting the target audio data from the buffer according to the output configuration information;

[0140] The target audio data is output.

[0141] The audio data packet loss compensation device 300 provided in the embodiment of the present disclosure can implement each process in the above-mentioned audio data packet loss compensation method embodiment, and will not be described again here to avoid repetition.

[0142] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device and a readable storage medium.

[0143] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0144] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of device 400 can also be stored in RAM 403. Computing unit 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0145] Various components in device 400 are connected to I / O interface 405, including an input unit 406, such as a keyboard, mouse, etc.; an output unit 404, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, optical disk, etc.; and a communication unit 409, such as a network card, modem, wireless communication transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0146] The computing unit 401 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the method for compensating for packet loss of audio data. For example, in some embodiments, the method for compensating for packet loss of audio data can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the method for compensating for packet loss of audio data described above can be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the audio data packet loss compensation method in any other appropriate manner (for example, by means of firmware).

[0147] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0148] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0149] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0150] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0151] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0152] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0153] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0154] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for compensating for packet loss of audio data, characterized in that: The method comprises: When a target audio data packet is detected to be missing from the cache, obtaining N reference audio data packets from the cache, wherein the audio playback time of any of the reference audio data packets is earlier than the audio playback time of the target audio data packet, and the N reference audio data packets are continuous and adjacent to the target audio data packet, where N is an integer greater than 1; Performing sub-band decomposition on the N reference audio data packets based on a sub-band coding (SBC) algorithm to obtain M reference sub-band data sets, where M is an integer greater than 1; Predicting the target audio data packet according to the M reference sub-band data sets based on the Berg algorithm to obtain a compensated audio data packet, and storing the compensated audio data packet in the buffer; The step of predicting the target audio data packet according to the M reference sub-band data sets to obtain the compensated audio data packet includes: Downsampling the M reference subband data sets to obtain M downsampled data sets, where a downsampling factor is greater than 1 and less than or equal to M; Predicting the target audio data packet according to the M downsampled data sets based on the Berg algorithm to obtain M target subbands, where the M target subbands correspond one-to-one to the M downsampled data sets; performing upsampling processing on the M target subbands respectively to obtain M upsampled subbands, wherein the upsampling factor of each target subband is the same as the downsampling factor of the corresponding reference subband dataset; Data synthesis is performed on the M up-sampled sub-bands to obtain the compensated audio data packet.

2. The method according to claim 1, characterized in that The performing sub-band decomposition on the N reference audio data packets to obtain M reference sub-band data sets includes: Performing sub-band decomposition on each reference audio data packet to obtain M reference sub-bands corresponding to the M sub-band positions. The N reference sub-band data at the same sub-band position of the N reference audio data packets constitute a reference sub-band set. The reference sub-band data of the N reference audio data packets form M reference sub-band data sets. The downsampling the M reference sub-band data sets to obtain M down-sampled data sets includes: downsampling the M reference sub-band data sets to obtain M down-sampled data sets corresponding to the M sub-band positions; The method of predicting the target audio data packet based on the Berg algorithm according to the M downsampled data sets to obtain M target subbands includes: predicting the audio subband at the same subband position of the target audio data packet according to each downsampled data set to obtain the corresponding target subband, and the M subband positions correspond one-to-one to the M audio subbands of the target audio data packet.

3. The method according to claim 1, characterized in that The buffer stores a plurality of audio data packets, each of the audio data packets corresponds to a sequence number, and the sequence numbers corresponding to different audio data packets in the plurality of audio data packets are different; The method further comprises: When it is detected that multiple serial numbers corresponding to multiple audio data packets stored in the cache are discontinuous, it is determined that a target audio data packet is missing from the cache, wherein the target audio data packet is the audio data packet corresponding to the missing serial number among the multiple serial numbers.

4. The method according to claim 1, wherein After storing the compensated audio data packet in the buffer, the method further includes: Acquire output configuration information, where the output configuration information is used to indicate a time delay and / or audio duration of target audio data; extracting the target audio data from the buffer according to the output configuration information; The target audio data is output.

5. A packet loss compensation device for audio data, characterized in that: The device comprises: an acquisition module, configured to acquire N reference audio data packets from the cache when detecting that a target audio data packet is missing from the cache, wherein the audio playback time of any of the reference audio data packets is earlier than the audio playback time of the target audio data packet, and the N reference audio data packets are continuous and adjacent to the target audio data packet, where N is an integer greater than 1; a decomposition module, configured to perform sub-band decomposition on the N reference audio data packets based on a sub-band coding (SBC) algorithm to obtain M reference sub-band data sets, where M is an integer greater than 1; a compensation module, configured to predict the target audio data packet according to the M reference sub-band data sets based on a Berg algorithm, obtain a compensated audio data packet, and store the compensated audio data packet in the buffer; Wherein, the compensation module includes: a downsampling unit, configured to perform downsampling processing on the M reference subband data sets respectively to obtain M downsampled data sets, where a multiple of the downsampling processing is greater than 1 and less than or equal to M; A prediction unit is configured to predict the target audio data packet based on the Berg algorithm and the M downsampled data sets to obtain M target subbands, wherein the M target subbands correspond one-to-one to the M downsampled data sets; upsample the M target subbands to obtain M upsampled subbands; and perform data synthesis on the M upsampled subbands to obtain the compensated audio data packet; wherein the upsampling multiple of each target subband is the same as the downsampling multiple of the corresponding reference subband data set.

6. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 4 when executed by the processor.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which implements the steps of the method according to any one of claims 1 to 4 when executed by a processor.

Images