Encoding method, encoding device, decoding method, decoding device, and transmission system
By introducing in-band FEC technology in CELT audio encoding, combining historical frame data and main frame data to generate the current frame code stream, the problem of weak packet loss resistance in RTC music scenes is solved, and the reliability and quality of audio transmission is significantly improved.
Patent Information
- Application Number
- CN202311861467.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2025-07-01
AI Technical Summary
The prior art has weak packet loss resistance in RTC music scenes, resulting in poor audio transmission effect.
By implementing in-band FEC technology in CELT-based audio encoding, historical frame data is generated and combined with main frame data to generate the current frame code stream, thereby improving the anti-packet loss capability.
Effectively combat packet loss problem in historical frame audio information, improve the reliability and quality of audio transmission, and improve the anti- packet loss capability.
Smart Images
Figure CN120236596A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of audio processing, and particularly relates to an encoding method, an encoding device, a decoding method, a decoding device, a transmission system, and a computer-readable storage medium. Background Art
[0002] There are many music scenarios in interactive entertainment applications, such as duet singing, competitions, or KTV (Karaoke TV), etc. In music scenarios, users generally have relatively high requirements for audio quality. Even in weak network scenarios, users also hope to obtain a smooth music experience. This requires the adopted audio encoding technology to have high packet loss resistance.
[0003] In the related art, in the RTC (Real-Time Communication) music scenario, the basic PLC (Packet Loss Concealment) algorithm in the Opus music encoding mode is used as the packet loss resistance technology. Summary of the Invention
[0004] The inventors of the present disclosure found the following problems in the above-mentioned related art: The packet loss resistance is weak, resulting in poor audio transmission effects.
[0005] In view of this, the present disclosure proposes an encoding technical solution, which can improve the packet loss resistance and thus improve the audio transmission effect.
[0006] According to some embodiments of the present disclosure, an encoding method is provided, including: encoding the current frame audio information to generate main frame data; generating historical frame data, where the historical frame data includes historical frame audio information, and the encoding bit rate of the historical frame data is lower than that of the main frame data; generating the current frame code stream according to the main frame data and the historical frame data.
[0007] In some embodiments, the historical frame audio information includes high-frequency band information and low-frequency band information, and the frequency of the high-frequency band information is higher than that of the low-frequency band information; the number of bits encoding the spectral shape of the high-frequency band information in the historical frame data of the current frame code stream is less than the number of bits encoding the spectral shape of the high-frequency band information in the main frame data of the historical frame code stream.
[0008] In some embodiments, the number of bits encoding the spectral shape of the low-frequency band information in the historical frame data of the current frame code stream is less than or equal to the number of bits encoding the spectral shape of the low-frequency band information in the main frame data of the historical frame code stream.
[0009] In some embodiments, the encoding of the spectral shape of the high-frequency band information is disabled in the historical frame data.
[0010] In some embodiments, when the remaining bits of the historical frame data are lower than a threshold, the historical frame data disables the spectral shape encoding of the high-band information, and the remaining bits are the difference between the available encoding bits and the used encoding bits.
[0011] In some embodiments, the historical frame data includes skip information for indicating the disabling of the spectral shape encoding of the high-band information.
[0012] In some embodiments, the historical frame data includes the spectral energy information of the historical frame audio information and the spectral shape information of the low-band information.
[0013] In some embodiments, the historical frame data includes a bitstream header byte for indicating the time offset information of the historical frame audio information relative to the current frame audio information.
[0014] In some embodiments, at least a part of the high-band information of the historical frame audio information in the historical frame data is encoded using the bandwidth expansion BWE algorithm.
[0015] In some embodiments, the historical frame data includes BWE bitstream information for encoding the spectral shape of the high-band information.
[0016] In some embodiments, the BWE bitstream information is used to encode at least one of the spectral energy of the high-band information or the bit allocation information of the high-band information.
[0017] In some embodiments, the historical frame data includes BWE status information for indicating whether BWE bitstream information is written in the historical frame data.
[0018] In some embodiments, the historical frame data is encoded using the Constrained Energy Lapped Transform CELT.
[0019] In some embodiments, the historical frame data includes an indication flag for indicating whether spectral shape encoding is performed on the historical frame audio information.
[0020] According to some other embodiments of the present disclosure, a decoding method is provided, including: receiving a current frame bitstream, where the historical frame data of the current frame bitstream is generated by encoding historical frame audio information, and the main frame data of the current frame bitstream is generated by encoding current frame audio information; decoding the historical frame data to recover the historical frame audio information when packet loss occurs in the historical frame audio information.
[0021] In some embodiments, the historical frame audio information includes high-band information and low-band information, and the frequency of the high-band information is higher than that of the low-band information; the number of bits of the spectral shape encoding of the high-band information in the historical frame data of the current frame bitstream is less than the number of bits of the spectral shape encoding of the high-band information in the main frame data of the historical frame bitstream.
[0022] In some embodiments, the number of bits for encoding the spectral shape of the low-band information in the historical frame data of the current frame stream is less than or equal to the number of bits for encoding the spectral shape of the low-band information in the main frame data of the historical frame stream.
[0023] In some embodiments, the historical frame data disables the encoding of the spectral shape of the high-band information.
[0024] In some embodiments, when the remaining bits of the historical frame data are lower than a threshold, the historical frame data disables the encoding of the spectral shape of the high-band information, and the remaining bits are the difference between the available encoding bits and the used encoding bits.
[0025] In some embodiments, the historical frame data includes skip information for indicating the disabling of the encoding of the spectral shape of the high-band information.
[0026] In some embodiments, the historical frame data includes the spectral energy information of the historical frame audio information and the spectral shape information of the low-band information.
[0027] In some embodiments, the historical frame data includes a bitstream header byte for indicating the time offset information of the historical frame audio information relative to the current frame audio information.
[0028] In some embodiments, at least a part of the high-band information of the historical frame audio information in the historical frame data is encoded using the bandwidth expansion (BWE) algorithm.
[0029] In some embodiments, the historical frame data includes BWE bitstream information for encoding the spectral shape of the high-band information.
[0030] In some embodiments, the BWE bitstream information is used to encode at least one of the spectral energy of the high-band information or the bit allocation information of the high-band information.
[0031] In some embodiments, the historical frame data includes BWE status information for indicating whether BWE bitstream information is written in the historical frame data.
[0032] In some embodiments, the historical frame data is encoded using the Constrained Energy Lapped Transform (CELT).
[0033] In some embodiments, the historical frame data includes an indication flag for indicating whether to perform spectral shape encoding on the historical frame audio information.
[0034] According to some further embodiments of the present disclosure, there is provided an encoding device, including: a first encoding unit for encoding the audio information of the current frame to generate main frame data; a second encoding unit for generating historical frame data, where the historical frame data includes historical frame audio information and the encoding bit rate of the historical frame data is lower than that of the main frame data; and a generating unit for generating a current frame bitstream according to the main frame data and the historical frame data.
[0035] According to some further embodiments of the present disclosure, there is provided a decoding device, including: a receiving unit for receiving a current frame bitstream, where the historical frame data of the current frame bitstream is generated by encoding historical frame audio information and the main frame data of the current frame bitstream is generated by encoding the current frame audio information; and a decoding unit for decoding the historical frame data to recover the historical frame audio information in the case of packet loss of the historical frame audio information.
[0036] According to some further embodiments of the present disclosure, there is provided a transmission system, including: an encoding device for performing the encoding method in any one of the above embodiments; and a decoding device for performing the decoding method in any one of the above embodiments.
[0037] According to some further embodiments of the present disclosure, there is provided an electronic device, including: a memory; and a processor coupled to the memory, where the processor is configured to execute the encoding method or the decoding method in any one of the above embodiments based on instructions stored in the memory device.
[0038] According to some further embodiments of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, it implements the encoding method or the decoding method in any one of the above embodiments.
[0039] In the above embodiments, historical frame data is added to the current frame bitstream, so that the current frame bitstream can carry historical frame audio information. In this way, the problem of packet loss of historical frame audio information can be effectively countered, the packet loss resistance ability can be improved, and thus the audio transmission effect can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The drawings forming a part of the specification depict embodiments of the present disclosure and, together with the description, are used to explain the principles of the present disclosure.
[0041] Referring to the drawings, the present disclosure can be more clearly understood from the following detailed description:
[0042] Figure 1 A flowchart showing some embodiments of the encoding method of the present disclosure;
[0043] Figures 2a - 2c A schematic diagram showing some embodiments of the audio data packet of the present disclosure;
[0044] Figure 3a Schematic diagrams showing some embodiments of the encoding method of the present disclosure;
[0045] Figure 3b Schematic diagrams showing some embodiments of the BWE encoding method of the present disclosure;
[0046] Figure 4 Flowcharts showing some embodiments of the decoding method of the present disclosure;
[0047] Figure 5 Block diagrams showing some embodiments of the encoding apparatus of the present disclosure;
[0048] Figure 6 Block diagrams showing some embodiments of the decoding apparatus of the present disclosure;
[0049] Figure 7 Block diagrams showing some embodiments of the electronic device of the present disclosure;
[0050] Figure 8 Block diagrams showing some other embodiments of the electronic device of the present disclosure;
[0051] Figure 9 Block diagrams showing some embodiments of the transmission system of the present disclosure. Detailed implementation manners
[0052] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure.
[0053] Meanwhile, it should be understood that, for the sake of description, the dimensions of the various parts shown in the drawings are not drawn in actual proportional relationship.
[0054] The following description of at least one exemplary embodiment is merely illustrative in nature and in no way serves as a limitation to the present disclosure, its application, or its use.
[0055] Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods, and devices should be regarded as part of the specification.
[0056] In all the examples shown and discussed herein, any specific value should be construed as merely exemplary and not as a limitation. Therefore, other examples of the exemplary embodiments may have different values.
[0057] It should be noted that like reference numerals and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0058] As mentioned above, to address the above technical problems, it is necessary to improve the packet loss resistance technology of the encoder. The present disclosure can increase the packet loss resistance ability of the encoder by implementing in-band FEC (Frame Error Concealment) technology in CELT-based audio coding. In this way, the characteristics of music signals can be utilized to efficiently encode music redundancy and achieve a better packet loss resistance effect under the premise of the same bandwidth.
[0059] For example, the technical solution of the present disclosure can be implemented through the following embodiments.
[0060] Figure 1 The flowchart showing some embodiments of the encoding method of the present disclosure.
[0061] As Figure 1 shown, in step 110, the audio information of the current frame is encoded to generate main frame data. For example, the audio information of the current frame can be encoded using the CELT encoding framework to generate main frame data.
[0062] For example, through CELT encoding, the spectrum of the audio to be encoded can be divided into multiple subbands (such as 21) using psychoacoustic principles; the energy of each subband and the normalized subband spectrum shape are quantized. The energy of each subband can be encoded preferentially, and then the shape of each subband can be encoded (such as using the PVQ pyramid vector quantization coding algorithm); in the case of encoding the shape of the subband, the available coding bits (Bit-alloc) are preferentially allocated to the low-frequency band.
[0063] In some embodiments, an in-band FEC algorithm can be implemented in CELT audio coding to improve the packet loss resistance ability of CELT audio coding. For example, the above technical solution can be implemented through the related embodiments of steps 120 and 130.
[0064] In step 120, historical frame data is generated, and the historical frame data includes historical frame audio information. The coding bit rate of the historical frame data is lower than that of the main frame data. For example, the coding bit rate of the historical frame data (which can be called the in-band frame) can not exceed a preset threshold (such as 24 kbps, etc.).
[0065] In some embodiments, the main frame data and the historical frame data adopt the same coding framework. For example, the same coding framework includes the CELT coding framework. For example, the CELT main frame data is encoded at a high bit rate; the FEC historical frame data is encoded at a low bit rate and low complexity for higher quality.
[0066] For example, the FEC historical frame data and the primary frame data use the same algorithm framework, and the encoding logic in Table 1 can be adopted.
[0067] Table 1
[0068]
[0069]
[0070] In step 130, a current frame bitstream is generated according to the primary frame data and the historical frame data. For example, the current frame bitstream can be generated through the embodiments in Figures 2a - 2c to implement the above technical solution.
[0071] Figures 2a - 2c Schematic diagrams showing some embodiments of the audio data packets of the present disclosure.
[0072] As Figure 2a shown, in the data packet of the current frame (such as the nth frame) for audio encoding using CELT, in addition to carrying the audio information of the current frame, the audio information of the historical frame (such as the (n-k)th frame) is also carried to implement the in-band FEC algorithm in the audio encoding of CELT.
[0073] For example, the encoding bit rate of the historical frame data is lower than that of the primary frame data. Implementing the in-band FEC algorithm in the audio encoding of CELT can meet the following requirements: the historical frame data with a low bit rate, such as the encoding bit rate of FEC does not exceed 24 kbps, etc.; the encoding with a low complexity, such as after adding the historical frame data, the increase in encoding complexity does not exceed a preset threshold, such as 25%, etc.; high-quality audio, such as in the packet loss scenario, the improvement of MOS (Mean Opinion Score) exceeds a preset threshold, such as 1.0, etc.
[0074] In some embodiments, the historical frame data includes a bitstream header byte for indicating the time offset information of the historical frame audio information relative to the current frame audio information.
[0075] As Figure 2b shown, the current frame bitstream can include a toc field, a CELT payload format field, and a padding payload field. The toc field (belonging to the bitstream header byte) and the CELT payload format field constitute the primary frame data for storing the encoding result of the current frame audio information.
[0076] The extended payload field includes a toh field and a (n-k)-th frame CELT FEC payload field, which are used to store the encoding results of historical frame audio information. The (n-k)-th frame CELT FEC payload field is the in-band FEC field of the (n-k)-th frame CELT. The toh field (which can occupy 1 byte, for example) can be used to store the offset information between the historical frame and the current frame, such as k frames; the (n-k)-th frame CELT FEC payload field can be used to store the encoding results of the audio information of the (n-k)-th frame.
[0077] In some embodiments, the historical frame data is used to recover the historical frame audio information in the case where there is a packet loss in the historical frame audio information.
[0078] For example, in the case where the n-th frame data packet is received at the decoding end, in addition to obtaining the audio information of the n-th frame through decoding, the audio information of the (n-k)-th frame can also be obtained through decoding.
[0079] In this way, in the case where there is a packet loss in the audio signal of the (n-k)-th frame, the historical frame data of the n-th frame data packet can be used to recover the audio signal of the (n-k)-th frame, so as to improve the packet loss resistance during audio transmission, thereby improving the reliability of audio transmission.
[0080] As Figure 2c shown, the 4 data packets in the first row are the data packets of the n-th frame to the (n+3)-th frame, respectively including historical frame data for storing the audio information of the historical frame with an offset of 3 frames from the current frame. During the process of transmitting the data packets to the decoder, the data packet of the n-th frame is lost, and the data packets of the other 4 frames are not lost.
[0081] After the decoder side finishes playing the audio information of the (n-1)-th frame, in the case where it is necessary to decode the data packet of the n-th frame (the data packets of the (n+1), (n+2) and (n+3) frames are to be decoded), it is detected that the data packet of the n-th frame is lost; in the above case, the historical frame data of the received data packet of the (n+3)-th frame can be decoded to recover the audio information of the n-th frame.
[0082] In the above embodiments, by encoding the historical frame audio information to generate historical frame data, high-quality recovery of audio in the case of packet loss can be achieved, thereby enhancing the transmission reliability in audio scenarios such as RTC.
[0083] In some embodiments, the historical frame audio information includes high-band information and low-band information, and the frequency of the high-band information is higher than that of the low-band information. For example, the frequency of the high-band information is higher than the frequency threshold, and the frequency of the low-band information is lower than the frequency threshold. For example, the frequency threshold can be 8 kHz.
[0084] In some embodiments, the historical frame data disables the spectral shape encoding of high-band information. For example, the spectral shape information of high-band information is not encoded within the historical frame data.
[0085] In some embodiments, when the remaining bits of the historical frame data are lower than a threshold, the historical frame data disables the spectral shape encoding of high-band information, where the remaining bits are the difference between the available encoding bits and the used encoding bits.
[0086] For example, the remaining bits are the bits remaining in the historical frame data after spectral energy encoding of the historical frame audio information and spectral shape encoding of the low-band information. For example, the historical frame data includes the spectral energy information of the historical frame audio information and the spectral shape information of the low-band information.
[0087] For example, the FEC historical frame data uses a low-bitrate encoding method and the CELT encoding framework. The historical frame data preferentially guarantees the energy encoding quality of each sub-band and the spectral shape encoding quality of the low-band, reduces the encoding bitrate, and thus improves the encoding effect. For example, the allocation logic of the PVQ (Pyramid Vector Quantization) bits of the FEC historical frame data can be changed to disable the PVQ bit allocation of high-band information.
[0088] In this way, the PVQ encoding bitrate can be reduced, thereby improving the encoding effect.
[0089] In some embodiments, the remaining bits are used for at least one of the normalized spectral shape vector quantization, sub-band energy fine quantization, anti-spectral collapse, or energy residue quantization of the historical frame data. For example, the sub-band energy fine quantization and energy residue quantization of the FEC historical frame data can be adapted.
[0090] For example, the remaining bits of the FEC historical frame data are mainly used for the normalized in-band PVQ of the FEC historical frame data, the energy fine quantization of the historical frame data, the anti-spectral collapse of the historical frame data, and the energy residue quantization process of the historical frame data.
[0091] For example, the PVQ of the historical frame data preferentially guarantees the bit allocation of the low-band; when the bit allocation of the high-band is insufficient, the PVQ bit allocation of the high-band is disabled.
[0092] In some embodiments, the historical frame data includes skip information for indicating the disabling of the spectral shape encoding of high-band information. For example, for a high-band with insufficient bit allocation, the PVQ of this high-band is skipped by encoding the SKIP information.
[0093] In some embodiments, the available number of bits of the historical frame data is determined according to the coding rate of the historical frame data and the number of bits required for the spectral energy information of the low-frequency band information, and the available number of bits of the historical frame data is less than the available number of bits of the main frame data.
[0094] For example, the main frame data is encoded using a high coding rate, and the available number of bits (nbCompressedBytes) is a relatively large value; the historical frame data is encoded using a low coding rate, and the available number of bits of the FEC historical frame data (nbCompressedBytes_fec) calculated using the rate control logic of in-band FEC is a relatively small value. The calculation of the available number of bits of the historical frame data needs to consider the coding rate of the FEC historical frame data and the number of PVQ bits required for CELT low-frequency band coding.
[0095] In some embodiments, the spectral shape of the high-frequency band information of the historical frame audio information is restored by spectral folding or noise filling.
[0096] For example, in the case where the spectral shape coding of the high-frequency band of the historical frame audio information is skipped in the historical frame data through the SKIP information, if the decoding side wants to restore the spectral shape of the high-frequency band of the historical frame audio information, the spectral shape of the high-frequency band can be restored by spectral folding or noise filling.
[0097] Figure 3a A schematic diagram showing some embodiments of the encoding method of the present disclosure.
[0098] As Figure 3a shown, in the CELT encoding framework, after performing stereo analysis processing, historical frame data is generated through steps such as historical frame data rate calculation, historical frame data bit allocation, historical frame data energy fine quantization, normalized spectral in-band PVQ, and historical frame data countermeasure spectral collapse and energy residual quantization.
[0099] In the process of calculating the historical frame data rate, the main frame data is encoded using a high coding rate, and the available number of bits (nbCompressedBytes) is a relatively large value; the historical frame data is encoded using a low coding rate, and the available number of bits of the FEC historical frame data (nbCompressedBytes_fec) calculated using the rate control logic of in-band FEC is a relatively small value. The calculation of the available number of bits of the historical frame data needs to consider the coding rate of the FEC historical frame data and the number of PVQ bits required for CELT low-frequency band coding.
[0100] During the process of allocating bits for historical frame data, the remaining bits of the FEC historical frame data are mainly used for PVQ within the normalized frequency band of the FEC historical frame data, fine quantization of the historical frame data energy, countering spectral collapse of the historical frame data, and quantization of the remaining energy of the historical frame data. The PVQ of the historical frame data preferentially ensures bit allocation for the low-frequency band; in the case of insufficient bit allocation in the high-frequency band, the PVQ bit allocation for the high-frequency band is disabled.
[0101] During the PVQ process within the normalized frequency band, for the high-frequency band with insufficient bit allocation, the PVQ of this high-frequency band is skipped by encoding Skip information.
[0102] In some embodiments, each symbol in the historical frame data can be configured through the embodiments in Table 2.
[0103] Table 2
[0104]
[0105]
[0106] In some embodiments, at least a part of the high-frequency band information of the historical frame audio information in the historical frame data is encoded using the bandwidth extension BWE algorithm. For example, the BWE bitstream information is used to encode the spectral shape of the high-frequency band information.
[0107] In some embodiments, the BWE bitstream information is used to encode at least one of the spectral energy of the high-frequency band information or the bit allocation information of the high-frequency band information.
[0108] For example, in the case of encoding high-bitrate CELT main frame data, a low-bitrate, low-complexity, high-quality encoding method can be used to generate the FEC in-band bitstream; the FEC in-band bitstream contains the BWE bitstream, which is used to recover the audio information in the high-frequency band (such as 8 - 20 kHz).
[0109] In some embodiments, the historical frame data is encoded using CELT. For example, the high-frequency band information encoded using CELT in the historical frame data is deleted; the high-frequency band information is encoded using the BWE algorithm to generate the BWE bitstream information. For example, the generated BWE bitstream information can be used to replace the high-frequency band information encoded using CELT.
[0110] For example, in the case of introducing the BWE bitstream information into the FEC historical frame data, the encoding information of the high-frequency band in the FEC in-band code frame of the original CELT encoding framework can be removed.
[0111] For example, for audio information in a low frequency band (such as 0 - 8 kHz), CELT coding is used. At an appropriate position in the FEC historical frame data, BWE bitstream information is inserted, and the coding information of the high frequency band in the CELT bitstream is removed.
[0112] For example, for audio information in a high frequency band (such as 8 - 20 kHz), BWE coding is used. The coding method of the FEC historical frame data uses information such as the energy of the high frequency band, the spectral shape of the high frequency band, and the bit allocation of the high frequency band in the BWE algorithm.
[0113] In the above embodiments, a more efficient BWE technology is introduced, and BWE bitstream information is added to the FEC historical frame data, improving the recovery quality of the FEC historical frame data for high frequency band audio information, thereby enhancing the transmission reliability and processing effect of the audio information.
[0114] In some embodiments, the introduction of the BWE technology can be achieved through Figure 3b the embodiments in
[0115] Figure 3b A schematic diagram showing some embodiments of the BWE coding method of the present disclosure.
[0116] As Figure 3b shown, under the CELT coding framework, after performing time - frequency analysis processing, historical frame data is generated through steps such as historical frame data copying and BWE coding, rough quantization of historical frame data energy, in - band time - frequency coding, in - band stereo analysis, calculation of historical frame data bit rate, bit allocation of historical frame data, fine quantization of historical frame data energy, normalized in - band PVQ, and historical frame data anti - spectral collapse and energy residue quantization.
[0117] During the BWE coding process, the historical frame data may include BWE status information for indicating whether BWE bitstream information is written into the historical frame data. For example, the BWE_state (used to indicate whether the BWE status is enabled) field and the BWE bitstream information (information such as the energy of the high frequency band, the spectral shape of the high frequency band, and the high frequency band bit allocation) are encoded.
[0118] For example, when BWE_state = 0, the BWE status is disabled, and the coding method in Figure 3a can be adopted; when BWE_state = 1, the BWE status is enabled, BWE bitstream information is written into the historical frame data, and the bitstream information of the high frequency band in the CELT framework FEC in - band coding is removed.
[0119] In some embodiments, the symbols in the historical frame data can be configured through the embodiments in Table 3 to achieve the introduction of the BWE technology.
[0120] Table 3
[0121]
[0122]
[0123] In this way, on the decoder side, the audio information in the low-frequency band can be restored using the bitstream information of the CELT framework, and the audio information in the high-frequency band can be restored using the bitstream information of BWE. BWE can achieve guided high-frequency band spectrum restoration, and can improve the audio restoration effect compared with techniques such as spectral folding, thereby improving the reliability of audio transmission and the audio processing effect.
[0124] Figure 4 A flowchart showing some embodiments of the decoding method of the present disclosure.
[0125] As Figure 4 shown, in step 410, the current frame bitstream is received. The historical frame data of the current frame bitstream is generated by encoding the historical frame audio information, and the main frame data of the current frame bitstream is generated by encoding the current frame audio information.
[0126] In step 420, in the case where a packet loss occurs in the historical frame audio information, the historical frame data is decoded to restore the historical frame audio information.
[0127] In some embodiments, the historical frame audio information includes high-frequency band information and low-frequency band information, and the frequency of the high-frequency band information is higher than that of the low-frequency band information; the number of bits encoding the spectral shape of the high-frequency band information in the historical frame data of the current frame bitstream is less than the number of bits encoding the spectral shape of the high-frequency band information in the main frame data of the historical frame bitstream.
[0128] In some embodiments, the number of bits encoding the spectral shape of the low-frequency band information in the historical frame data of the current frame bitstream is less than or equal to the number of bits encoding the spectral shape of the low-frequency band information in the main frame data of the historical frame bitstream.
[0129] In some embodiments, the historical frame data disables encoding of the spectral shape of the high-frequency band information.
[0130] In some embodiments, in the case where the remaining bits of the historical frame data are lower than a threshold, the historical frame data disables encoding of the spectral shape of the high-frequency band information, and the remaining bits are the difference between the available encoding bits and the used encoding bits.
[0131] In some embodiments, the historical frame data includes skip information for indicating disabling encoding of the spectral shape of the high-frequency band information.
[0132] In some embodiments, the historical frame data includes the spectral energy information of the historical frame audio information and the spectral shape information of the low-frequency band information.
[0133] In some embodiments, the historical frame data includes a bitstream header byte for indicating time offset information of the historical frame audio information relative to the current frame audio information.
[0134] In some embodiments, at least a part of the high-frequency band information of the historical frame audio information in the historical frame data is encoded using a bandwidth expansion (BWE) algorithm.
[0135] In some embodiments, the historical frame data includes BWE bitstream information for encoding the spectral shape of the high-frequency band information.
[0136] In some embodiments, the BWE bitstream information is used to encode at least one of the spectral energy of the high-frequency band information or the bit allocation information of the high-frequency band information.
[0137] In some embodiments, the historical frame data includes BWE status information for indicating whether BWE bitstream information is written into the historical frame data.
[0138] In some embodiments, the historical frame data is encoded using a constrained energy-optimized transform (CELT).
[0139] In some embodiments, the historical frame data includes an indication flag for indicating whether spectral shape encoding is performed on the historical frame audio information.
[0140] Figure 5 A block diagram showing some embodiments of the encoding device of the present disclosure.
[0141] As Figure 5 shown, the encoding device 5 includes: a first encoding unit 51 for encoding the current frame audio information to generate main frame data; a second encoding unit 52 for generating historical frame data, where the historical frame data includes historical frame audio information, and the encoding bit rate of the historical frame data is lower than that of the main frame data; and a generating unit 53 for generating a current frame bitstream according to the main frame data and the historical frame data.
[0142] In some embodiments, the historical frame audio information includes high-frequency band information and low-frequency band information, and the frequency of the high-frequency band information is higher than that of the low-frequency band information; the number of bits for spectral shape encoding of the high-frequency band information in the historical frame data of the current frame bitstream is less than the number of bits for spectral shape encoding of the high-frequency band information in the main frame data of the historical frame bitstream.
[0143] In some embodiments, the number of bits for spectral shape encoding of the low-frequency band information in the historical frame data of the current frame bitstream is less than or equal to the number of bits for spectral shape encoding of the low-frequency band information in the main frame data of the historical frame bitstream.
[0144] In some embodiments, the historical frame data disables spectral shape encoding of the high-frequency band information.
[0145] In some embodiments, when the remaining bits of the historical frame data are lower than a threshold, the historical frame data disables the spectral shape encoding of the high-band information, and the remaining bits are the difference between the available coding bits and the used coding bits.
[0146] In some embodiments, the historical frame data includes skip information for indicating the disabling of the spectral shape encoding of the high-band information.
[0147] In some embodiments, the historical frame data includes the spectral energy information of the historical frame audio information and the spectral shape information of the low-band information.
[0148] In some embodiments, the historical frame data includes a bitstream header byte for indicating the time offset information of the historical frame audio information relative to the current frame audio information.
[0149] In some embodiments, at least a part of the high-band information of the historical frame audio information in the historical frame data is encoded using the bandwidth expansion BWE algorithm.
[0150] In some embodiments, the historical frame data includes BWE bitstream information for encoding the spectral shape of the high-band information.
[0151] In some embodiments, the BWE bitstream information is used to encode at least one of the spectral energy of the high-band information or the bit allocation information of the high-band information.
[0152] In some embodiments, the historical frame data includes BWE status information for indicating whether BWE bitstream information is written in the historical frame data.
[0153] In some embodiments, the historical frame data is encoded using the constrained energy overlap transform CELT.
[0154] In some embodiments, the historical frame data includes an indication identifier for indicating whether spectral shape encoding is performed on the historical frame audio information.
[0155] Figure 6 The block diagram showing some embodiments of the decoding device of the present disclosure.
[0156] As Figure 6 shown, the decoding device 6 includes: a receiving unit 61 for receiving a current frame bitstream, the historical frame data of the current frame bitstream is generated by encoding historical frame audio information, and the main frame data of the current frame bitstream is generated by encoding current frame audio information; a decoding unit 62 for decoding the historical frame data to recover the historical frame audio information when a packet loss occurs in the historical frame audio information.
[0157] In some embodiments, the historical frame audio information includes high-band information and low-band information, and the frequency of the high-band information is higher than that of the low-band information; the number of bits encoding the spectral shape of the high-band information in the historical frame data of the current frame bitstream is less than the number of bits encoding the spectral shape of the high-band information in the main frame data of the historical frame bitstream.
[0158] In some embodiments, the number of bits encoding the spectral shape of the low-band information in the historical frame data of the current frame bitstream is less than or equal to the number of bits encoding the spectral shape of the low-band information in the main frame data of the historical frame bitstream.
[0159] In some embodiments, the historical frame data disables encoding of the spectral shape of the high-band information.
[0160] In some embodiments, when the remaining bits in the historical frame data are lower than a threshold, the historical frame data disables encoding of the spectral shape of the high-band information, and the remaining bits are the difference between the available encoding bits and the used encoding bits.
[0161] In some embodiments, the historical frame data includes skip information for indicating disabling encoding of the spectral shape of the high-band information.
[0162] In some embodiments, the historical frame data includes the spectral energy information of the historical frame audio information and the spectral shape information of the low-band information.
[0163] In some embodiments, the historical frame data includes a bitstream header byte for indicating the time offset information of the historical frame audio information relative to the current frame audio information.
[0164] In some embodiments, at least a part of the high-band information of the historical frame audio information in the historical frame data is encoded using the bandwidth expansion (BWE) algorithm.
[0165] In some embodiments, the historical frame data includes BWE bitstream information for encoding the spectral shape of the high-band information.
[0166] In some embodiments, the BWE bitstream information is used to encode at least one of the spectral energy of the high-band information or the bit allocation information of the high-band information.
[0167] In some embodiments, the historical frame data includes BWE status information for indicating whether BWE bitstream information is written in the historical frame data.
[0168] In some embodiments, the historical frame data is encoded using the constrained energy-optimized transform (CELT).
[0169] In some embodiments, the historical frame data includes an indication flag for indicating whether to encode the spectral shape of the historical frame audio information.
[0170] Figure 7 Block diagram showing some embodiments of the electronic device of the present disclosure.
[0171] As Figure 7 shown, the electronic device 7 of this embodiment includes: a memory 71 and a processor 72 coupled to the memory 71. The processor 72 is configured to execute the encoding method or the decoding method in any one of the embodiments of the present disclosure based on the instructions stored in the memory 71.
[0172] Among them, the memory 71 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory stores, for example, an operating system, application programs, a boot loader, a database, and other programs.
[0173] Figure 8 Block diagram showing some other embodiments of the electronic device of the present disclosure.
[0174] As Figure 8 shown, the electronic device 8 of this embodiment includes: a memory 810 and a processor 820 coupled to the memory 810. The processor 820 is configured to execute the encoding method or the decoding method in any one of the foregoing embodiments based on the instructions stored in the memory 810.
[0175] The memory 810 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory stores, for example, an operating system, application programs, a boot loader, and other programs.
[0176] The electronic device 8 may further include an input / output interface 830, a network interface 840, a storage interface 850, etc. These interfaces 830, 840, 850 and the memory 810 and the processor 820 may be connected through a bus 860, for example. Among them, the input / output interface 830 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, a touch screen, a microphone, and a speaker. The network interface 840 provides a connection interface for various networking devices. The storage interface 850 provides a connection interface for external storage devices such as an SD card and a USB flash drive.
[0177] Figure 9 Block diagram showing some embodiments of the transmission system of the present disclosure.
[0178] As Figure 9 shown, the transmission system 9 includes: an encoding device 91 for executing the encoding method in any one of the above embodiments; a decoding device 92 for executing the decoding method in any one of the above embodiments.
[0179] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable non-transitory storage media including, but not limited to, disk storage, CD-ROM, optical storage, etc., which contain computer-usable program code.
[0180] So far, the encoding method, encoding device, decoding method, decoding device, transmission system, and computer-readable storage medium according to the present disclosure have been described in detail. To avoid obscuring the concept of the present disclosure, some details well known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.
[0181] The methods and systems of the present disclosure can be implemented in many ways. For example, the methods and systems of the present disclosure can be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present disclosure are not limited to the specific order described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure can also be implemented as a program recorded in a recording medium, and these programs include machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
[0182] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art should understand that the above examples are only for illustration and not for limiting the scope of the present disclosure. Those skilled in the art should understand that the above embodiments can be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.
Claims
1. A coding method, comprising: Encoding the audio information of the current frame to generate main frame data; Generating historical frame data, where the historical frame data includes historical frame audio information, and the coding bit rate of the historical frame data is lower than that of the main frame data; Generating the current frame bitstream according to the main frame data and the historical frame data.
2. The coding method according to claim 1, wherein: The historical frame audio information includes high-frequency band information and low-frequency band information, and the frequency of the high-frequency band information is higher than that of the low-frequency band information; The number of bits for encoding the spectral shape of the high-frequency band information in the historical frame data of the current frame bitstream is less than the number of bits for encoding the spectral shape of the high-frequency band information in the main frame data of the historical frame bitstream.
3. The encoding method according to claim 2, wherein, The number of bits for encoding the spectral shape of the low-frequency band information in the historical frame data of the current frame bitstream is less than or equal to the number of bits for encoding the spectral shape of the low-frequency band information in the main frame data of the historical frame bitstream.
4. The encoding method according to claim 2, wherein, The historical frame data disables the encoding of the spectral shape of the high-frequency band information.
5. The encoding method according to claim 4, wherein, In the case where the remaining bits of the historical frame data are lower than a threshold, the historical frame data disables the encoding of the spectral shape of the high-frequency band information, and the remaining bits are the difference between the available coding bits and the used coding bits.
6. The encoding method according to claim 4, wherein, The historical frame data includes skip information for indicating the disabling of the encoding of the spectral shape of the high-frequency band information.
7. The encoding method according to claim 2, wherein The historical frame data includes the spectral energy information of the historical frame audio information and the spectral shape information of the low-frequency band information.
8. The encoding method according to any one of claims 1-7, wherein, The historical frame data includes a bitstream header byte for indicating the time offset information of the historical frame audio information relative to the current frame audio information.
9. The encoding method according to any one of claims 1-7, wherein, At least a part of the high-frequency band information of the historical frame audio information in the historical frame data is encoded using the bandwidth expansion BWE algorithm.
10. The encoding method according to claim 9, wherein, The historical frame data includes BWE bitstream information for encoding the spectral shape of the high-frequency band information.
11. The encoding method according to claim 10, wherein, The BWE bitstream information is used to encode at least one of the spectral energy of the high-frequency band information or the bit allocation information of the high-frequency band information.
12. The encoding method according to claim 10, wherein, The historical frame data includes BWE status information for indicating whether the BWE bitstream information is written in the historical frame data.
13. The encoding method according to claims 1-7, wherein, The historical frame data is encoded using the constrained energy overlap transform CELT.
14. The encoding method according to any one of claims 1-7, wherein, The historical frame data includes an indication identifier for indicating whether to perform spectral shape encoding on the historical frame audio information.
15. A decoding method, comprising: Receiving a current frame bitstream, where the historical frame data of the current frame bitstream is generated by encoding historical frame audio information, and the main frame data of the current frame bitstream is generated by encoding current frame audio information; In the case where the historical frame audio information is lost, decoding the historical frame data to recover the historical frame audio information.
16. The decoding method according to claim 15, wherein: The historical frame audio information includes high-frequency band information and low-frequency band information, and the frequency of the high-frequency band information is higher than that of the low-frequency band information; The number of bits for encoding the spectral shape of the high-frequency band information in the historical frame data of the current frame stream is less than the number of bits for encoding the spectral shape of the high-frequency band information in the main frame data of the historical frame stream.
17. The decoding method according to claim 16, wherein, The number of bits for encoding the spectral shape of the low-frequency band information in the historical frame data of the current frame stream is less than or equal to the number of bits for encoding the spectral shape of the low-frequency band information in the main frame data of the historical frame stream.
18. The decoding method according to claim 16, wherein, Encoding of the spectral shape of the high-frequency band information is disabled in the historical frame data.
19. The decoding method according to claim 18, wherein, In the case where the remaining bits in the historical frame data are lower than a threshold, encoding of the spectral shape of the high-frequency band information is disabled in the historical frame data, and the remaining bits are the difference between the available encoding bits and the used encoding bits.
20. The decoding method according to claim 18, wherein, The historical frame data includes skip information for indicating that encoding of the spectral shape of the high-frequency band information is disabled.
21. The decoding method according to claim 16, wherein, The historical frame data includes the spectral energy information of the historical frame audio information and the spectral shape information of the low-frequency band information.
22. The decoding method according to any one of claims 15-21, wherein, The historical frame data includes a bitstream header byte for indicating the time offset information of the historical frame audio information relative to the current frame audio information.
23. The decoding method according to any one of claims 15-21, wherein, At least a part of the high-frequency band information of the historical frame audio information in the historical frame data is encoded using a bandwidth expansion (BWE) algorithm.
24. The decoding method according to claim 23, wherein, The historical frame data includes BWE bitstream information for encoding the spectral shape of the high-frequency band information.
25. The decoding method according to claim 24, wherein, The BWE bitstream information is used to encode at least one of the spectral energy of the high-frequency band information or the bit allocation information of the high-frequency band information.
26. The decoding method according to claim 24, wherein, The historical frame data includes BWE status information for indicating whether the BWE bitstream information is written in the historical frame data.
27. The decoding method according to claims 15-21, wherein, The historical frame data is encoded using a constrained energy overlap transform (CELT).
28. The decoding method according to any one of claims 15-21, wherein, The historical frame data includes an indication flag for indicating whether spectral shape encoding is performed on the historical frame audio information.
29. An encoding device, comprising: A first encoding unit for encoding current frame audio information to generate main frame data; A second encoding unit for generating historical frame data, where the historical frame data includes historical frame audio information, and the encoding bit rate of the historical frame data is lower than the encoding bit rate of the main frame data; A generating unit for generating a current frame stream according to the main frame data and the historical frame data.
30. A decoding device, comprising: A receiving unit for receiving a current frame stream, where the historical frame data of the current frame stream is generated by encoding historical frame audio information, and the main frame data of the current frame stream is generated by encoding current frame audio information; A decoding unit for decoding the historical frame data to recover the historical frame audio information in the case where packet loss occurs in the historical frame audio information.
31. A transmission system, comprising: An encoding device for performing the encoding method according to any one of claims 1 - 14; A decoding device for performing the decoding method according to any one of claims 15 - 28.
32. An electronic device, comprising: A memory; And A processor coupled to the memory, the processor being configured to execute the encoding method according to any one of claims 1-14 or the decoding method according to any one of claims 15-28 based on instructions stored in the memory.
33. A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the encoding method according to any one of claims 1-14 or the decoding method according to any one of claims 15-28.